At work, I'm the only IT guy (do it all, and do it now type guy) for the last 10 years. If I ever got hit by a bus they would be totally screwed. I've mentioned it several times to management/president type people, yet they ignore me. Too bad for them.
What can I do to alleviate their pain? (Or should I even care?)
(Yes, this should be a community wiki, but, I don't see the checkbox... maybe I don't have enough rep.)
Document the heck out of everything.
There was a thread on Slashdot recently about starting documentation, which inspired me to write down my thoughts about documentation.
My key points were:
Principle #1: It Is Never Done
Documentation is an on-going effort that will always lag behind what is in production. Changes are made ad-hoc, things moved around or discontinued or put into service at random. Documentation will never catch up.
You have to sell the people paying the bills on the value of spending time (and therefore, money) on keeping the running documentation up to date. Frequently those conversations go like this: "remember when I had to spend $TIME figuring out how $THING was broken? Well when I was finished, there was this tech note detailing $THING, so that the next guy to come along won't have to figure it all out."
You have to do it, even though you will never finish.
Principle #2: The Only Thing Worse Than No Documentation Is Wrong Documentation
This is more of a truism than a principle. Documentation can lull you into the false sense that something is in a known state and that if something goes wrong you can therefore have a running start at fixing it.
It is important to acknowledge this problem.
Principle #3: You Are Writing Documentation For Your Successor
Odds are 95% of anything you do document you will never have to refer to again. Documentation is a collection of wisdom for the future, not for you. So you have to assume that your audience knows little or nothing about the specifics of how things are the way they are.
And there will be a successor. I don't know about you, but I don't plan to be in these specific environments for the rest of my life. Opportunities come and go, and when they come, sometimes you go. But life goes on behind you, and the smoother you can make life for your successor the better. Otherwise you might have a collection of former customers who quietly say unflattering things about you. I like to say that it is the same 50 guys working everywhere in IT in Ottawa because you keep running into them everywhere. Helping your successor might open doors for you in the future.
Now to a certain extent there is always a degree of "blame the previous guy" when trouble comes up. That is part of the business. I've done it myself. But on several occasions when I had blasted the previous guy as some kind of moron, I have learned otherwise that he really had his act together and knew more about what was going on than I did at the time.
Principle #4: "Why" is often more important than "How"
When looking at a system most of us start thinking thinks like why the hell is this like this? There are almost always very specific reasons for the configuration choices made. In these circumstances, the "Why" dictates the "How", and you have to make sure that the reader understands the specific problems being solved when examining the smoking remains of your solution.
Principle #5: It has to be easy or you won't do it
This means you have to be very aware of your tools as well as those who are going to use your tools.
Keeping things up-to-date has to be easy. If you have to make any kind of effort, then you will find excuses to avoid doing it when it is best done, which is immediately after a change.
If your tools are not easy for others to use, then they won't use it. This can be especially crippling in a team environment, since the larger the team gets the more likely you will encounter a team member who does not like your choice of tools.
Personally, I like a wiki for documents. However the problem is that a wiki does not force a structure on you, so the structure must be imposed from outside. This always leads to conflict somewhere as somebody else has a better/different idea.
In some places I've used Word and Visio documents "published" to PDF, with the "latest" PDF being considered authoritative. This is good in that you then have a collection that you can hand to your employer/successor. The PDFs, if properly dated, can provide a historical record of what happened, although one which is not easy to navigate through. It is bad in that I don't like Word or Visio and have been forced to get a basic understanding of these tools in order to effectively communicate the ideas.
My current employer is toying with the idea of Word documents in a Sharepoint portal. We'll just have to see how far we get there
Of course you should care. After all, any job worth doing is a job worth doing well.
1.) It's already been said but it needs repeating for reiteration. Document, document, document. Use excel spreadsheets, notepaper, quill and parchment if you have to. Several thousand mead notebooks like in the movie "Se7en" if need be. Either way, lay it out clear, concise and easy-to-read for whomever is going to have to replace you when you get hit by a meteor.
2.) Once you've begun documenting everything, you should be in the writing mood. Time to start a side project detailing the last few years' changes made on the servers. Start building a change-management process, but go through it historically. Make sure you note how often you've changed those disks on some of those finicky servers. How much they cost, etc. These provide great metrics for you to rely on anyway, even if that meteor misses you and takes out the neighbor's dog instead.
3.) Implement a monitoring system that monitors and emails critical failures. What's that, you say? You have one already? Sweet! Now document it. How it works, what you monitor, why you monitor it.
4.) You've got a responsibility to take it to your management types again. And again, and again. As many times as you can. Be polite. Be respectful, but come bearing what the cost to the business would be, if that meteor came down and you disappeared.
This is not a responsibility that you can blow off, it's an ethical requirement of your job and comisserate to the position you hold as the proverbial `keeper and guardian of the keys to the kingdom'.
Think of it this way. If you can avoid and forget worriyng about this and feel OK doing that, then why aren't you combing the accounting fileshares for company salaries to see what and how much more those management types are making than you. Why aren't you exploiting confidential corporate data for your own use? Why aren't you reading people's email?
Simply put, (and hopefully) you aren't doing these things because of a good solid sense of morals and ethics. You know, the whole right from wrong sort of thing. Thus follows, if you have that, then you know that it's clearly your responsibility to document and prepare countermeasures against the worst-case scenerio.
That being, your uninterrupted-by-work and relaxing vacation trip to Hawaii. :)
(Sans meteor, that is.)
Be careful crossing the street, look both ways, ensure there are no buses with Keanu Reeves and Sandra Bullock inside charging down the road.
If you want to lower the learning curve for your replacement, the best thing you can do is to write documentation of the set up and of your processes. Possibly the easiest way to do that is to set up a wiki system somewhere and just keep adding to it. Some of it will become out of date, but what isn't will be invaluable.
The many posts above regarding the importance of documentation are spot on, but there is one other aspect to being "the guy" at work that you want to look out for.
If you are the only person that knows how to operate everything, vacations/births/emergencies are difficult to handle, and you can never be promoted or learn other positions in the company (if that interests you). If you are not able to grow and learn and expand your skills, you may end up in a position down the line where you are looking for a job, and your resume shows that you have spent 10 years as a COBOL/FORTRAN programmer or an OS\2/Novell/NT admin.
Growth and crosstraining are important to development as a sysadmin. Instead of being "the guy" that the whole network depends on, be "the guy" that is always willing to show the new guy what to do and who is always interested in learning more about the business.
The key here is detailed and complete documentation. This is something that is critical not only in case you are incapacitated but if they have a junior person come in and you want to go on vacation (or move on to greener pastures). Having proper docs and references can be immensely useful for someone else to come up to speed on the network.
I try VERY HARD to ensure that we have no SPoF knowledge-wise, including my own. That attitude is more valuable than any local knowledge I might have kept to myself.
I took over this position (Sys Admin / Lead Dev) from a guy who had his own specific way of doing everything. The 17 servers we have are all set up slightly different. There are so many procedures and manual handling of issues here and hardly any of it was documented (documentation pretty much included a single line explanation of each server and its role). This has made me re-evaluate a lot of the processes in the office. Everytime I learn something, its added to the wiki. I also delegate some server work to other devs so they can at least learn minimal info about small jobs.
Documentation writing sucks, but think what it'd be like to come into your position without any.
Documentation is a huge deal. Where I'm at, I have the opposite problem as you really, we recently put in a new POS/Order Tracking system (we're an engraving and promotions company) and I'm have the problem of getting people to write in the system the information about the customization. Mainly low-level people, as the manager will do it to her orders, because she used to do it to all orders, putting it into a VERY OLD access database. I so want to go paperless, and it's entirely possible in our business, but I just can't get people to type the work order information in, dammit.
So, keep plugging away at getting authorization. Or if you're feeling nice, do it on your own time. It might be a good thing for your future at the company, not to mention your successor.
re: should I care?
Unless your office is 100% dysfunctional, people are going to notice that you make things work more smoothly. That you care about the well-being of the company, and not just locking in your 'indispensibility'. The company that banks on you never leaving / never getting a better offer isn't functioning very well. It's someone's JOB to notice that you withhold that kind of thing from the company.
I personally LOVE working with people who document, and want the work they've finished to stand on its own, and not constantly require their involvement. When you document and put the things you've already done behind you, you open up more time to work on future projects.
We have basically the same situation. That's why the director has hired me to add one more node of knowledge.
But honestly, it is not your place to care. It is the business of management to assess risks and address them.
I've found that the Getting Things Done approach helps get over the initial mental barrier preventing you from documenting your work properly. Rather than think of documentation as a monolithic task which will prevent you from doing "real" work, subdivide it into bite-sized chunks. Whenever you run across a tidbit of information that your successor should know, write it down. Then reserve an hour or two each week to review, clarify and categorize your knowledge.
Don't document. Don't share knowledge with others. By documenting things, you make yourself less valuable. You will get paid less. Your bonus will be less. Your job security will be decreased. Why would you do this?
The proper balance is where you don't get disturbed unnecessarily during sleep or vacation. Otherwise, you devalue yourself.
