Remote Linux Admin Consultant - Best Practice

Question

We are engaging a consultant in India as our Linux Administrator. We don't know him well and he requires Root access to all our servers to do his job (including a security audit).

What is the best practice for enabling a remote consultant for such work such that we are protected against any malignant activities?

Thanks in advance.

Answer 1

Don't. Also, you're in as much danger of ineptitude as malice from what I've seen of the typical way companies handle this.

I'd like to say, there's probably great system administrators out there in India, but the way many companies do things is terrible.

If you're going through a body shop, you're also likely seeing a pretty big cut go to them, and many of them are unlikely to have properly vetted their employees. I've talked to three, one of whom I worked for and none of them have done any technical interviews.

So, if you must hire someone remotely, for god's sake, interview him yourself and make sure he knows his work. System administration is far too important to hand over to someone blindly

Now that I've handled the "ineptitude" part of it,

Administration is a pretty broad phrase. And someone with root access can do anything. Now, personally I think creating an account for the admin, and giving him the ability to elevate himself through sudo is a better idea (which your config management system should handle if you have many servers). That said, even that relies on a certain amount of trust. There's so many stories out there of the sheer damage a disgruntled sysadmin can do. Change all your passwords? Sure you could get in eventually, but its not trivial, and it would probably cost more than you're saving.

So, consider a local. If not, consider someone you have vetted yourself and have directly hired.

Answer 2

As has been mentioned, don't do this.

The only way you'll be able to protect yourself is by doing something like this:

1. Insist that the consultant use a configuration management system of your choosing.
2. The consultant will write configuration management manifests for the actions you need completed.
3. The consultant will test the manifests on a test system.
4. When ready, the consultant will commit the configuration to a code repository.
5. All changes are reviewed by either a member of your staff or another consultant that has absolutely no relation to the first, and has no way of contacting them.
6. Once the changes are signed off, they are applied to the server by you or a member of your staff. The original consultant should not have access to any of your systems.

As should be clear, this is a very clumsy and inefficient process, but if you insist on accepting work from a non-trusted individual, this is one way to handle things.

As I recommended, though, you're much better off hiring a known, trusted individual.

Answer 3

What is the best practice for enabling a remote consultant for such work such that we are protected against any malignant activities?

From a legal perspective: due diligence beforehand and strict penalties for breach of contract.

You start with the usual good hiring practices that also apply when hiring on-premise staff (and/or service providers) which include fact checking the provided resume, ask for education transcripts and certification numbers, check and call their references, interview, maybe even a background check or security screening etc. etc.

Then apply the carrot: pay fair value, offer attractive work, amazing colleagues, good working conditions and benefits etc. (if you pay peanuts you get monkeys.)

And the stick: breach the terms of your employment/service contract and we'll sick our lawyers on you and leave you bankrupt!

Unfortunately both the above become increasingly more difficult when crossing borders and timezones.

Once you decide on hiring somebody:

• clear directives and policies, people should be aware of what they should and may not do.
• the principle of minimal access applies, make it difficult for people to (accidentally or on purpose) do things they shouldn't. For the typical system administrator often that still means full access, but a security auditor for instance should not need full administrator access but can simply request the existing admins to run a script on his behalf that collects the details he needs to make his report. Such a script can easily be checked beforehand.
• trust, but verify. Simply have the existing staff check the work of a new joiner and as always collect audit information.
• etc. etc.

This question details what I usually ask my clients to do to establish remote access for me, which might be a starting point for you as well.

Answer 4

There is one systemic method of protecting yourself that comes to mind, which I have not seen mentioned.

Host your Linux instances as VM's on a virtualization hypervisor (VMware, Xenserver, Hyper-V, etc.).

DO NOT give the remote admin administrative access to the hypervisor. The remote admin would only get root access to the VM's themselves.

DO Implement a hypervisor based backup system (Unitrends, Veeam, vSphere Data Protection, etc.)

DO keep at least one snapshot per day of each Linux VM, going back as far in time as you feel is necessary.

DO NOT give the remote admin write access to the backup repositories.

If you do these things, you will have backup snapshots of each Linux instance over which the remote admin has no control. If the remote admin does something hinky, whether intentionally or accidentally, you can always mount a backup from before the hinkeness occurred to evaluate what happened and possibly recover to a clean state.

This won't be proof against a hypervisor side-channel attack, which could potentially be mounted from within a VM that the attacker has root access to.

If your backups don't go far enough back in time, this won't protect you.

You need to thoroughly trust whomever is in control of your hypervisor and the backup infrastructure.

If you're doing this in the cloud (AWS, Azure, etc.), the implementation details will differ, but the general concept would be the same.

In essence, divide responsibilities among parties who are not business partners with each other, in addition to only hiring people you trust.

Answer 5

Give him his own user account. Then find out exactly what he needs access to and grant just that access but nothing else. For example if he needs to reconfigure an Apache web server, use an ACL to give him write access to the Apache configuration files and configure sudo to let him restart the Apache service but not execute any other commands as root. As always, keep backups of anything that you're giving him access to (in this case, your Apache configuration files).