I have apparently malicious php script somewhere in my webserver, Debian 8.3, Apache 2.4 Legal Apache process can be seen like this:
16169 www-data 20 0 482156 59180 41952 S 7.3 1.5 0:01.08 apache2
14791 www-data 20 0 483288 63456 45088 S 0.7 1.6 0:02.08 apache2
The illegal httpd process can be seen like this:
19214 www-data 20 0 142700 112156 2052 S 0.7 2.8 0:00.89 httpd
We can see user, which launched the process and image name, which is obviously faked, because when I do some tracking via PID:
root@VS1:~/skripty# lsof -p 19214
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 19214 www-data cwd DIR 8,1 4096 2 /
httpd 19214 www-data rtd DIR 8,1 4096 2 /
httpd 19214 www-data txt REG 8,1 10416 3278761 /usr/bin/perl
httpd 19214 www-data mem REG 8,1 27136 3281419 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/File/Glob/Glob.so
httpd 19214 www-data mem REG 8,1 89368 3281423 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/POSIX/POSIX.so
httpd 19214 www-data mem REG 8,1 18632 3281416 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Fcntl/Fcntl.so
httpd 19214 www-data mem REG 8,1 43408 3281417 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Socket/Socket.so
httpd 19214 www-data mem REG 8,1 18728 3281425 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/IO/IO.so
httpd 19214 www-data mem REG 8,1 35176 1313952 /lib/x86_64-linux-gnu/libcrypt-2.19.so
httpd 19214 www-data mem REG 8,1 1738176 1313950 /lib/x86_64-linux-gnu/libc-2.19.so
httpd 19214 www-data mem REG 8,1 137440 1313946 /lib/x86_64-linux-gnu/libpthread-2.19.so
httpd 19214 www-data mem REG 8,1 1051056 1313954 /lib/x86_64-linux-gnu/libm-2.19.so
httpd 19214 www-data mem REG 8,1 14664 1313953 /lib/x86_64-linux-gnu/libdl-2.19.so
httpd 19214 www-data mem REG 8,1 1827768 3281415 /usr/lib/x86_64-linux-gnu/libperl.so.5.20.2
httpd 19214 www-data mem REG 8,1 140928 1313947 /lib/x86_64-linux-gnu/ld-2.19.so
httpd 19214 www-data 0r CHR 1,3 0t0 5593 /dev/null
httpd 19214 www-data 1w CHR 1,3 0t0 5593 /dev/null
httpd 19214 www-data 2w CHR 1,3 0t0 5593 /dev/null
httpd 19214 www-data 3u IPv4 29921164 0t0 TCP *:27451 (LISTEN)
Please note, that this server is right now behind internal firewall, so there is no way to comunicate with outside world through SMTP on port 25. If I unblock this firewall rule, LSOF and NETSTAT will show me tens, or even hundreds of active SMTP connections. This malware is apparently designed for spamming. What I can not see from LSOF is the origin of perl script. I can kill the process but it wil reapear again. It is even shutting down itself.
So what I allready did:
- blocked any outside communication on port 25 to preserve me from banning
- searched for any suspitious code blocks like eval, base64_decode and handfully checked scripts for suspitious code blocks.
- In php.ini I blocked following: disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,exec,shell_exec,system,passthru,
- Hardened file/folder permissions to make the php applications (wordpresses, expression engine). Any php file has owner of root and permissions set to 444. Any folder owner is www-data with 500 permissions. For special directories, like uploads, there is 700 rule. To fix that, I set up cronjob for every folder which is running every 1 minute and cleans up any *.php file and puts harmless index.php file into every folder/subfolder. I needed to exclude som cache folders and I suspect that that might be my current attack vector.
Important note: When I stop Apache 2.4 server and manually kill fake httpd process, no new malicious httpd processes will spawn.
This leads me to a conclusion, to manualy enable/disable my virtualhosts (10 at this time) to se, which one is causing launching of malicious httpd process.
If you have better idea, I will gladly follow your advices.
Thanks
EDIT: This is allready a fresh new server but when I migrated on some old webapps, which I cleaned by automatic tools and eyeballing the source code, I apparently missed something. In the old server, there were a lot of other issues (like litecoin mining, image data for webshops which do not belonged to me, thousands of html files generated by robots). Right now I have no other known issue except this fake process. Server is hard (or at least) hardened enough then before. This is my last problem with him. I need to track perl script executing.
EDIT 2
root@VS1:~# pstree -p -u
systemd(1)─┬─acpid(461)
├─agetty(751)
├─apache2(28369)─┬─apache2(26542,www-data)
│ ├─apache2(27556,www-data)
│ ├─apache2(27895,www-data)
│ ├─apache2(28231,www-data)
│ ├─apache2(28233,www-data)
│ ├─apache2(30282,www-data)
│ ├─apache2(30283,www-data)
│ ├─apache2(30285,www-data)
│ ├─apache2(30286,www-data)
│ └─apache2(30288,www-data)
├─atd(447,daemon)
├─cron(450)
├─dbus-daemon(454,messagebus)
├─httpd(30628,www-data)
├─init(29268,www-data)
├─logger(15714)
├─mysqld_safe(15713)─┬─logger(15853)
│ └─mysqld(15852,mysql)─┬─{mysqld}(15854)
│ ├─{mysqld}(15855)
│ ├─{mysqld}(15856)
│ ├─{mysqld}(15857)
│ ├─{mysqld}(15858)
│ ├─{mysqld}(15859)
│ ├─{mysqld}(15860)
│ ├─{mysqld}(15861)
│ ├─{mysqld}(15862)
│ ├─{mysqld}(15863)
│ ├─{mysqld}(15864)
│ ├─{mysqld}(15866)
│ ├─{mysqld}(15867)
│ ├─{mysqld}(15868)
│ ├─{mysqld}(15869)
│ ├─{mysqld}(15870)
│ ├─{mysqld}(15871)
│ ├─{mysqld}(15872)
│ ├─{mysqld}(15873)
│ ├─{mysqld}(15874)
│ ├─{mysqld}(15875)
│ ├─{mysqld}(15876)
│ ├─{mysqld}(28177)
│ ├─{mysqld}(28253)
│ ├─{mysqld}(28259)
│ ├─{mysqld}(28275)
│ ├─{mysqld}(30162)
│ ├─{mysqld}(30166)
│ ├─{mysqld}(30172)
│ └─{mysqld}(30173)
├─named(2130,bind)─┬─{named}(2132)
│ ├─{named}(2133)
│ └─{named}(2134)
├─rpc.idmapd(446)
├─rpc.statd(432,statd)
├─rpcbind(423)
├─rsyslogd(460)─┬─{in:imklog}(498)
│ ├─{in:imuxsock}(497)
│ └─{rs:main Q:Reg}(499)
├─sshd(14432)───sshd(28238)───bash(28240)───pstree(30964)
├─systemd-journal(142)
├─systemd-logind(452)
├─systemd-udevd(145)
└─vmtoolsd(448)
