Write it down!
After many years of doing system implementations for customers, I have a certain phobia about communicating technical details. No matter how long the day has been, or how tired I am, I always send customers the addresses, user names, passwords, etc for any systems I setup before my head hits the pillow. As I told one client recently, I do this just in case I fall off a cliff before they get it.
I am sure that many reading this have experienced that strange feeling in the pit of your stomach that accompanies the realization that you don’t have a key password, or forgot the procedure necessary to correct a problem with a particular system.
So, quick quiz: pick any three of your key systems at random, firewalls, wireless gateways, servers, etc, and find your documentation with passwords, addresses, and backup locations for them. Can you do it?
If you can’t, welcome to the club. It happens to all of us. We get something running reliably, and forget it. As they say, out of sight, out of mind. Then, after many months, a system dies, and we are sent scrambling around looking for the details we need to fix it. As a good example, you may use a cloud-based backup system, such as iDrive. But, do you know the login password required to restore backups?
You may well not be ultimately to blame for the omission. Vendors are notorious for not passing along the details you need after they finish. I worked on a customer security camera system recently. The installing vendor had not provided the customer with all of the necessary password information when they finished. The vendor subsequently had a corporate split, and lost many of their own records. When the customer could not login to the system, the vendor was forced to initialize the camera system, including all 18 cameras, and reset the passwords. The scary part came thereafter however, when they declined to give the customer the new passwords -- to protect the customer from themselves.
In any event, whether you are to blame for the original problem or not, you will likely be forced one day to fix it. Systems break, and even the best system administration cannot change that. Information security issues have made this problem worse. We now face malware such as ransomware, which can quickly turn a working system into a space heater.
It today’s competitive business climate, none of us can afford to have a system go down, and not be able to it back up quickly. As such, I recommend that you take some time out of your schedule to gather the necessary details to make sure you can recover from a system issue:
Inventory what you have
Do a physical inventory (or a virtual one in the case of virtual systems), and record them, including serial numbers, system models, service tags, physical locations, etc.
Record system and application details
For each system, note some key details, incuding:
Login credentials, including the original administrator credentials, if different
What the system does
What backups are being performed, what software was used to make them, and where they are
Install file location for any applications
The location of any documentation regarding how to diagnose and correct application issues
Fix any gaps, now, not later
Don’t wait until a system dies and you are in a panic to get it back up. If you find that you are missing key details, track them down. You may have to involve a vendor, and even pressure them, if they are reluctant to help, but get everything together, sooner, rather than later.
Vendors are not the only issue. Many companies have employees with key details in their heads. These folks may forgot, or worse, refuse to give back when you need it. Case in point – some time back, I was called to hack my way into a key medical billing system. It seems that the employee who knew the password quit, and demanded a monthly retainer to keep the system running.
Decide where to store documentation
You would be surprised how many people I have seen store critical details about a system on the system itself. Bad idea. If the system goes, so much for your recovery details. Find a good place to keep your details, preferably two. I generally store the critical documents on a cloud service like Dropbox, with a paper copy (yes, believe it or not), in a secure location.
Have a backup plan
If you lose a key system, what will you do in the interim? It there a manual work-around? Do you have a backup system? Do you need to notify customers? Figure out your backup plan for each system, and write it down.
Drill baby, drill
The only way to know for sure that you can recover from a major failure is to simulate one. Have someone pick a random system, pretend it is fried, and simulate the process of getting it back into service using your documentation.
Bottom line – we live in a dangerous world, and it is getting worse, not better. The difference between you and another company who goes belly up after a crisis will be your good documentation.