100% CPU load caused by service "perfctl"

Question

I am running a dedicated High Performance AMD Server with Ubuntu 20.04. since a few months.

Suddenly tonight CPU shoot up to 100% until the point that I killed the service "perfctl" that appeared at 2am in the morning.

I am running Apparmor:

apparmor module is loaded.
8 profiles are loaded.
8 profiles are in enforce mode.
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/mysqld
   /{,usr/}sbin/dhclient
   lsb_release
   nvidia_modprobe
   nvidia_modprobe//kmod
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
   /usr/sbin/mysqld (1124) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

What could have caused this and how can this be prevented in the future?

Answer 1

The fact that the user is www is a bit suspect. Are you running AppArmor? Are you running a web service that is exposed to the public?

This looks to me like your server is compromised, and someone has used the www user to run a binary they've called perfctl to hide it's identity.

You may want to read How do I deal with a compromised server?

Answer 2

I have encountered the same malware.

You can try to check the all cronjob task whether is suspicious or not.

• check the cron job list with all user

for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done

I found the malware with above command then you should remove it.

www
11 * * * * /home/www/.config/cron/perfcc

Answer 3

I fixed the problem as follows:

1. I booted my VPS in Rescue mode from a special image provided by my hosting provider.
2. I deleted the perfcc, perfctl files on the VPS volume (find / -name perfcc, find / -name perfctl).
3. After that, I started the VPS in normal mode, installed the antivirus software "Dr.Web for Linux" and performed a full antivirus scan of the entire VPS. This allowed me to find 2 libraries for which the antivirus showed the status "Infected" with the Linux.Muldrop.137 virus.

Threat: Linux.MulDrop.137 ‌ Isolated: 9/30/24 10:35 PM Quarantine type: System Object name: libfsnldev.so Owner: root Modified: 9/29/24 6:08 PM Size: 8.87 MB Origin path: /usr/lib/libfsnldev.so

Threat: Linux.MulDrop.137 ‌ Isolated: 9/30/24 10:34 PM Quarantine type: System Object name: libpprocps.so Owner: root Modified: 9/29/24 6:08 PM Size: 8.87 MB Origin path: /usr/lib/libpprocps.so

But antivirus I couldn't immediately quarantine the problematic files because the immutable attribute was enabled on the files. I disabled the attribute with the commands

liv@singularity:~/kvrt$ sudo chattr -i /usr/lib/libpcap.so liv@singularity:~/kvrt$ sudo chattr -i /usr/lib/libfsnldev.so,

which allowed the antivirus to quarantine the files.

Answer 4

First, stop the process to prevent it from consuming CPU resources:

sudo killall perfctl

Locate and remove files associated with the perfctl process. Use the find command to scan for possible locations:

sudo find / -name perfctl

Delete all found files:

sudo rm -f /path/to/perfctl