0

It has come to my attention that my CentOS 5.6 5.11 VPS has over 16,000 files with a timestamp during a 3-day period in September this year. I can't figure out why that is, as I did not log in via SSH during that time (nor were there any other logins, which would have been very worrisome) and there are no records of actions in Webmin during that period. Those are the only two ways I would have done anything to change system files. The server appears to run fine, but no one else is supposed to have access (other than the hoster, of course), so I want to know what's going on.

I captured the files with those dates in a file thus:

touch -t 201409160000 start.tmp
touch -t 201409190000 stop.tmp
find / -type f -newer start.tmp \! -newer stop.tmp -exec ls -la {} \; > sep-16-18.txt

But what do I do with this information? The volume is crazy!

[root](21:05:55)[~]$ grep "Sep 16" -c sep-16-18.txt
6079
[root](21:06:30)[~]$ grep "Sep 17" -c sep-16-18.txt
2580
[root](21:06:40)[~]$ grep "Sep 18" -c sep-16-18.txt
7653

The whole rest of the month of September was less than 500 files total, so something radical happened in the middle of the month.

Almost all the files were in the /usr/ directory:

/usr/:  16130
/lib/:     49
/sbin/:    49
/etc/:     45
/lib64/:   37
All others: 3

But the breakdown of subdirectories under /usr/ is spread across the board pretty widely:

/usr/lib/:     6514
/usr/include/: 5109
/usr/share/:   3438
/usr/lib64/:    853
/usr/bin/:      105
/usr/sbin/:      61
/usr/libexec/:   36
/usr/kerberos/:  14

Perhaps the file with the earliest timestamp might shed some light (perhaps it started the cascade), but I can't seem to sort the find results by time. I could repeatedly reset start.tmp and stop.tmp to smaller and smaller time periods, re-running find until I got a reasonable number of results to look through manually, but I would probably have no idea what I'm looking at - I'm really not a Linux expert anyway. Perhaps someone can give me some tips on what "questions" I should be asking my server. It looks as if Linux was reinstalled or something, but...

UPDATE: For those who are asking, here are the first several files with timestamps on the 16th. The first one seems unrelated, but then starting at 15:04, for the next couple minutes over 5000 files were updated (or created, I suppose):

1410846366 Tue Sep 16 14:46:06 2014 /etc/default/nss
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/features.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/gnu-versions.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/limits.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/include/values.h
1410847466 Tue Sep 16 15:04:26 2014 /usr/lib64/libc.so
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/bits/xopen_lim.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/libc-version.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/lib-names.h
1410847467 Tue Sep 16 15:04:27 2014 /usr/include/gnu/stubs.h
1410847468 Tue Sep 16 15:04:28 2014 /usr/include/gconv.h
1410847468 Tue Sep 16 15:04:28 2014 /usr/include/iconv.h
1410847473 Tue Sep 16 15:04:33 2014 /usr/lib64/gconv/gconv-modules
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/bits/locale.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/langinfo.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/locale.h
1410847475 Tue Sep 16 15:04:35 2014 /usr/include/xlocale.h
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ANSI_X3.110-1983.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ANSI_X3.4-1968.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ARMSCII-8.gz
1410847476 Tue Sep 16 15:04:36 2014 /usr/share/i18n/charmaps/ASMO_449.gz

After that, it goes on with character maps and locale-related files at some length, then more include files, then after about 5500 of that type of file it takes a break for a little while before more .so files and other things that look pretty normal. Does this tell anyone anything? None of I have seen in the list look suspicious; I'm no expert, but it looks more like an innocent update. But I don't know how how an update would happen besides a shell login or Webmin.

ewwhite
  • 201,205
OsakaWebbie
  • 221
  • 3
  • 11

2 Answers2

6

This question's on its way to getting closed as a duplicate of How do I deal with a compromised server, and I think that's a mistake, because the very correct advice in that question is to nuke the server and restore from known-good media. However, there's absolutely no evidence that this system has been compromised.

OsakaWebbie, you have admitted that you were wrong about the OS version; the system's at 5.11. That given, the files you list are all part of the glibc-headers package, with the exception of /usr/lib64/libc.so which is part of glibc-devel, a package that travels in lockstep with glibc-headers and is generally built at the same time.

I would have done this on a C5 system, but I don't have one to hand. On an up-to-date C6 system, the information about glibc-headers is

[me@lory 2012]$ rpm -qi glibc-headers
Name        : glibc-headers                Relocations: (not relocatable)
Version     : 2.12                              Vendor: CentOS
Release     : 1.149.el6                     Build Date: Wed 15 Oct 2014 03:00:58 BST
Install Date: Fri 24 Oct 2014 20:42:04 BST      Build Host: c6b9.bsys.dev.centos.org
[...]

Note the build and install dates. Now let's look at the files in question:

[me@lory 2012]$ for file in `rpm -ql glibc-headers`; do ls -la $file ; done
[...]
-rw-r--r--. 1 root root 2416 Oct 15 02:44 /usr/include/gnu-versions.h
-rw-r--r--. 1 root root 3057 Oct 15 02:44 /usr/include/gnu/lib-names.h
-rw-r--r--. 1 root root 1337 Oct 15 02:44 /usr/include/gnu/libc-version.h
-rw-r--r--. 1 root root 315 Oct 15 02:44 /usr/include/gnu/stubs.h
-rw-r--r--. 1 root root 6991 Oct 15 02:44 /usr/include/grp.h
-rw-r--r--. 1 root root 4611 Oct 15 02:44 /usr/include/gshadow.h
-rw-r--r--. 1 root root 1949 Oct 15 02:44 /usr/include/iconv.h
-rw-r--r--. 1 root root 4990 Oct 15 02:44 /usr/include/ieee754.h
[...]

Note how the files all have the same timestamp, and it's a few minutes before the Build Date baked into the RPM. From this we conclude that the modification time of an RPM-controlled system file is the modification time it had on the build system where the RPM was created; since a build happens as a monolithic process, it is completely reasonable that all the files in an RPM should have very similar mtimes, and indeed that those in associated packages should have them also, and that those mtimes will not correspond to the date on which the packages were installed.

If you can confirm that your system is indeed at C5.10/5.11, you can stop panicking; you have presented no evidence that anything bad has happened to your system.

Edit: OsakaWebbie, you having confirmed that the system is now at 5.11, I submit this is a non-issue. The modification times on the files you've shown are exactly as you'd expect them to be.

For reference - and it's quite important that you understand this - C5.11 isn't a major release of C5. C6 is a different major release to C5, but C5.11 is just a fully-patched C5.6 (or any other version of C5). You can read about this at some length in an earlier answer I wrote on C/RH versioning if you'd like, but in all probability you brought the system to 5.11 when you did that yum update in late Sep / early Oct. You can confirm that with grep centos-release /var/log/yum.log*, if you like.

Community
  • 1
MadHatter
  • 81,580
1

I'm going to largely C&P an answer of mine from an unrelated question. This is a better home for it I think. The instructions below are provided with the assumption that you have already followed the instructions for How do I deal with a compromised server? if applicable. Always disconnect the network cable when in doubt.

find /filesystem1 /filesystem2 -xdev -printf '%T@ %t %p\n' | sort -n

%T@ Modification time, epoch seconds (for sorting)

%t Modification time, human readable

%p Full file path

Season to taste. Run on one mounted filesystem at a time or as many as you think is relevant.

What this does is give you a timestamp sorted audit of the most recent modifications to files. This can occasionally give you insight into other things that were happening at the time of the unknown event. It's certainly not bulletproof, and may not reveal anything at all, but I occasionally find this extremely useful in identifying what work was being performed at the time that the unknown event happened. (i.e. someone performing rm -Rf /lib instead of rm -Rf lib)

Asides:

  • If the output of this command contains a wall of common libraries and executables that cannot be associated with an OS update, the most likely cause is a rootkit being overlaid on top of your operating system.

  • Another possibility (that you should be less inclined to consider) is a clueless admin running a greedy rsync from another system to this one. It's unlikely to be a restore from an archival copy (backup software, tar, etc.) as those are generally good about honoring the original mtime.

  • If you can't figure it out, you have no choice but to treat this as an intrusion.

Community
  • 1
Andrew B
  • 33,868