10 servers to administer and I'm a history major

Question

Through reasons that don't warrant exhaustive discussion, I find myself in charge of 10 servers:

• A domain controller-~500 hosts/~350 users
• IIS web server-This is where we make our money
• SQL server-The crown jewels
• Exchange server
• Linux box for data entry
• AV server
• Backup server
• A few others tossed around

The company where I work believes everybody is replaceable and therefore believes they can pay a minimum wage for any position. The IT manager and Sysadmin quit recently and I think I was the only person who did not take a big step backward when the call went out for volunteers. This also explains why someone with my background is in this position. That is the reality, as much as I wish it otherwise.

What are the things I should be doing to keep those systems running? There is no written procedure left behind and I crammed the A+ and Network+ certs in the last two months but that leaves me with some theory and no practical experience.

I am in the process of teaching myself powershell but from here to there is a long way. I have no scripting or programming experience.

What tasks should I be performing? What practices should I implement?

I understand I am probably hosed but a lifeline to get me through would be helpful.

Answer 1

Honestly, I would find another job, unless your current task is just to keep everything running until they hire a new SysAdmin. You are being setup for failure. You are doing the job of at least two people if this is all hosted locally and nothing is documented.

Don't worry about the scripting or programming anything just yet. Get a handle on keeping everything running.

Are you in charge of the corporate firewall too?

The quick and dirty daily tasks I see you needing to do are (in no particular order):

• check the nightly backups
• check the exchange queues to make sure they are processing
• check the SQL backups
• check the AV server for alerts if anything failed to scan or update

Answer 2

First off, look at Zoredache's link to the Beginning Sysadmin question. It's a lot of info but will get you where you need to go. That being said, it's a lot of data dumping.

Second, seriously, look for a new job. If your employer regards everyone as interchangeable widgets you're better off not being there anyway. But it's a bad economy, and that might not be an option.

In your specific case, I would:

1. Find a local IT firm you can escalate labor to if something breaks and you're over your head.
2. Make sure your maintenance contracts with your software vendors are up to date. Chances are most of your software packages (including whatever's on the IIS server that's making you money) were purchased rather than written internally. Make sure you have support with them - if anything breaks, you'll be calling them first.
3. Make sure you have good backups - both data and system state. If one of your servers toasts, you'll need to be able to restore it quickly. This may actually be the most important priority.
4. Do a hardware audit and make sure your servers are under warranty. If they're out, point out to management that there will be significant downtime if they die, and you only make money when they're up.
5. Pray
6. Perform a software audit, hope you find a lot of out of compliance software, report it to the Business Software Alliance (if you're in the US), and use the money you make from the finders fee to tide you over until you find a new job.

...6 may be optional.

Answer 3

Find a new job. Let that company fail. They clearly need a lesson.

Your stepping up to help will quickly become unappreciated and chances are, won't lead to anything else but more stress and abuse.

Answer 4

Start drinking? :-)

Besides that, look at getting a spare box or two that can be used to set up non-production servers. That will give you the opportunity to learn how to deal with some of this software in a slightly less dangerous environment than "if I break it we all stop working and making money".

Even if you run them inside virtualization software like VirtualBox, it's something that gives you hands on with the software without jeopardizing the production work.

Finally, if you do have to touch the production servers before you're more comfortable with them, document every step you take. Make sure you always know how you got to the screen you're on and how to get back.

Answer 5

Honestly, find a mentor!

Not knowing where you are, it's tough to recommend a forum, but meetups are pretty common with IT folk so google your local area for a Sysadmin (or specific techs) meetup. You'll get a whole bunch of new contacts out of this too; which might help in unexpected ways.

Update:

I'd also highly recommend you start writing a blog about this (pseudonym required IMO)!

Answer 6

So, you will be learning on production servers and this is not your choice. In this case, you should not be blamed on any service interruption. It is the company responsibility to find a replacement for the sysadmin.

You can learn many things, but it will take time and it may hurt the running services. You have to be extremely careful before doing any change and you should do it on a development/testing server first.

There are many related questions on this site such as: SysAdmin & Developer: Responsibilities, https://serverfault.com/questions/51006/how-did-you-become-a-sysadmin, and What's the difference between sysadmin and an IT Consultant?. I hope these will give you an idea what should be done as a sysadmin if you could not get out of this ASAP.

Answer 7

Protect your jewels!

Brent Ozar has an excellent "Oh crap, I just inherited a database server" script and even a training video now. He calls it the "60 minute database blitz" but I think my title is implied.

Remember this

Your backups are important but nowhere near as important as your restores.

What's that mean? Yes you need to take backups to have a chance of unbuggering a situation but if you don't test those backups, all you're doing is burning disk space.

Answer 8

If for some reason you decide you actually WANT to do sysadmin work, I heartily recommend Tom Limoncelli's Practice of System and Network Admistration. He provides excellent examples of how a well-operated organization might be run and if you get an opportunity to suggest changes, it's tough to go wrong with his recommendations. Book link to Amazon: http://amzn.to/sNUmsr

Answer 9

You never want to be in a position where you are forced to learn on production systems. Learning involves making mistakes, and mistakes on production systems cost your company money. They may not care about anything else (which is clearly indicated by the very fact you are in this position to begin with) but they will care about this.

1. If it is at all an option find another job. This will not end well. And honestly do you really want to work at a company that treats everyone as replaceable?
2. Request an emergency budget so you can find a consulting company to keep things running and act as a resource while you get up to speed. If this is denied, make sure you have it in writing somewhere for CYA purposes.
3. Read any documentation left behind by the previous IT staff.
4. Perform an inventory of both physical hardware and devices. Figure out what is in warranty and what isn't.

Your goal should be to find another job and leave without anything breaking. If something does break (and it probably will) be prepared to walk out. This means you should be extremely careful regarding CYA procedures. You need to assume that no on at the company has your back - make sure you're not in a position where you need to rely on someone's support.

While you're looking for a new job setup a environment to practice and learn with and do your best to familiarize yourself with the current infrastructure.

And for what it's worth... my degree is in Philosophy. You can succeed in this field if you want to, but not at this company. Good luck.

Answer 10

Reading everybody else's comment I would say:

1. Take this opportunity to learn
2. Tell your management without sugar coating that they are in a dire situation
3. Take this opportunity to learn more. After all sysadmin / DBA related jobs are well paid.
4. If you do get fired, collect some money from them and find a new job with your newly acquired skills.

I love the write a blog comment, you have pretty good writing skills. Make sure to put a link where we can find it.

Answer 11

There are quite a bit of scattered answers here. Here are some of my thoughts.

Some people say you should do it and others say to completely avoid it. My question to you is are you ready to devote hours on and working on these servers? Not only that, but are you also willing to learn things completely new?

Server administration isn't tough once you grasp it, but I've ruined servers by running the wrong commands, not following the "standards" (like taking daily backups), and et cetera. Every system admin will fail sometime, hey, it's how we learn, but you have to be prepared on how to handle it. It's a good amount of work and the biggest question here is if you're willing to take on the job.

Sure you can learn something new to put on your resumé, but if you are not willing to learn or devote yourself to you job, you're just going to be another person who "hates their job".

Answer 12

I will be frank, this is a handfull, even for more experienced guys, also learning most of the skills in server administration is learning from mistakes. I doubt you want to make mistakes on live-systems.

The danger of you not having any or few experience in server administration also makes it dangerous for your company. You are the victim of ignorant managers it seems. The only way to wake them up would probably be a sudden disaster, where disaster recovery is nescesary and nobody will be around who has the knowledge to make it run again and their crucial business processes stop working.

I'm not implying you should create a disaster, but just go and find another job, because you don't want to be held accountable when things go bad.

Answer 13

Seize the day! This is a great opportunity. Get a subscription to the Safari Tech Books database and read up on all of the how to books and certification books for the systems you administer. Start by focusing on backups and restore procedures. Get yourself test servers to play around with before laying hands on the production models. Test servers can be cheap desktop units, they don't have to be expensive or durable like production ones. Make yourself a goal to be successful in this position for two years and then go get a new job with a good company that is willing to pay you for your great knowledge and experience. Also, earn certifications for all the technology you are responsible for. You are poised to have a lot of fun and earn a lot of money (way more than a degree in history would get you).