Our VPS is being used as a Warez mule

Question

The company I work for runs a series of ecommerce stores on a VPS. It's a WAMP stack, 50gb storage.

We use an archaic piece of ecommerce software which operates almost entirely client-side. When an order is taken, it writes it to disk and then we schedule a task to download the orders once every 10 minutes.

A few days ago, we ran out of disk space, which caused orders to fail to be written. I quickly hopped on to delete some old logs from the mailserver and freed up a couple of GB pretty quickly, but I wondered how we could fill up 50gb will nothing much more than logs.

Turns out, we didn't. Hidden deep within the c:\System Volume Information directory, we have a stack of pirated videos, which seem to have appeared (looking at the timestamps) over the past three weeks. Porn, American Sports, Australian cooking shows. A very odd collection. Doesn't look like an individual's personal tastes - more like the VPS is being used as a mule.

We have a 5-attempts and you're blocked policy on our FTP server (plus, there is no FTP account with access to that directory), and the windows user account has had it's password changed recently. The main avenues are sealed - and logs can verify that. I thought I'd watch and see if it happened again, and yes, another cooking show has appeared this morning.

I am the only one to know of this problem at my company, and only one of two with access to the VPS (the other being my boss, but no - it's not him).

So how is this happening?

Is there a vulnerability in some of the software on the VPS? Are the VPS owners peddling warez across our rented space? (can they do this?)

I don't want to delete the warez in case it is seen as a hostile action against this outside force, and they choose to retaliate.

What should I do? How do I troubleshoot this? Has this happened to anyone else before?

Answer 1

without auditing code (are you using custom store software?), you can't know if there's a bug that's being exploited (and even if you didn't find anything that doesn't mean it isn't there). Are you using custom SQL code, for example? With sanity checking and cleaning of input?

I'm assuming all your systems are fully up to date? Malware checks? Antivirus up to date?

Once someone cracks the system, they could have rootkitted it. No matter how you put in password changes or alterations, if something in the system has been changed to allow backdoor access, you're not going to keep it out. On top of that it could be logging your password changes and keystrokes, so you're just feeding more passwords to the attacker.

You can put in auditing to see where connections are coming from, but I doubt that'll help a heckuva lot.

In the end, you need to consider a wipe and reinstall from scratch. It's the only way you can trust the installation again and know that it's clean from trojan code, since the infection can mask itself once it's in control.

Scarier, if you're taking credit cards, this information could be getting swiped and you'll be liable for having customer identities swiped. If you're in the US this has ramifications where you need to notify customers of possible identity theft.

If this is a server that handles anything involving money you might need to consider calling in contractors to audit the system. Take images of the system for forensic use and wipe and reinstall. The longer you wait, the more liability you open yourself to.

To answer how it's happening, if the server is a dedicated server, it could be cracking something in your storefront (SQL injection, for example) vulnerability in Windows not patched anything web browse with that system? "Drive by" downloaders from a website. Run software on it that's not from the system? Could have been infected with something. Weak passwords. Audit them ever? And there's the possibility that you won't easily know how they did it. My bets are on the storefront software, especially if it's niche, as it's easy for developers to not clean input from the URL and open it to injection attacks. Or if it's using PHP open to an outside interface; you do keep that up to date? You don't mention if it uses something like a php administrative interface, but sloppy php coding can add an easy attack vector as well.

If you're at a total loss of how to fight this, seriously, hire outside help. No shame in getting help, and the rule of thumb is that once you're hacked, you CANNOT be sure it's fixed, and if customer data is flowing on that system you're opening yourself to liability and harming innocent customers. Plus if that system isn't partitioned off from other systems on the network, it could try to intercept other system's data.

Answer 2

First off, I think you'll find there are at least three people with access to the VPS, not just the two you know about. I believe the system has been hacked and taken over. I also suspect that it is now hosting files for a P2P network, most likely torrents. You could have a search for *.torrent files but they're probably hidden from your view anyway.

You are no longer in control of the system. They have simply left you with that illusion. Had they not been so careless as to use up all that disk space they may well have got away with it for a lot longer. Incidentally, even a rudimentary monitoring system should have alerted you to diminishing disk space.

At this point you need to take Bart's advice and gather forensic evidence in the form of a system snapshot. Then wipe it completely and reinstall from scratch. Your backups at this point are probably worthless, as you have no real way of knowing when the hacking took place.

You need to look at what can be done to better secure your newly rebuilt system. For that matter, I'd consider rebuilding it on a new host and continue to use the existing one till it's ready, then do a cut-over.