A disaster averted?

A disaster: averted?

What happens to a small or medium sized company when it loses all corporate data in a disaster? See this page [1].

According to research by the University of Texas, only 6 percent of companies suffering from a catastrophic data loss survive, while 43 percent never reopen and 51 percent close within two years.

Our company, where some of us have slaved for more than a decade, was on the brink of losing all corporate data, due to negligence and carelessness of our own engineers, on Friday 2 Oct 2009. Just a few days ago.

What happened?

Some of our engineers wanted to test some new hard drives we have purchased. To do so, they shut down and dismantled our primary production server (the Dell SC1435), removed its disks, changed its disk controller settings, inserted the new test disks, and tried their tests. The new disks were found to be working fine. Then they removed the new disks and put in the old drives back into the server. The server would not boot.

Why?

Why did our engineers dismantle a production server? Let's look at the facts:

• These new disks could have been tested on any other server. There are at least one or two other totally unimportant desktop-class machines which could have been used for the testing. In that case, why was our most important production server dismantled?
• Our main production server does not have any complete and current backup. Our only backup of this server is out-of-date and incomplete.
• The primary production server has a RAID-0 setting done in firmware using the in-built disk controller. Such RAID configurations done in firmware usually cannot be re-created if you dismantle the system, remove the disks, and change the firmware settings. You should never play with the disks in a hardware-RAID-configured system.

So we have a lot of questions, but no answers.

How did we come back from the edge of disaster?

Some of our best engineers rushed to office, together with the original lot which had created the man-made disaster. They all struggled to get the original RAID-0 disk set to assemble and work together as before. Within two or three hours, they had succeeded. Their initiative, and their technical knowledge, plus a large chunk of luck, helped us survive.

Why don't we have backups of our primary corporate data?

Because we have never had the money, of late, to buy fresh hardware to take a full backup. Our old systems have small disks, and can't hold a full backup. One of our backup servers (apps1) is also showing hardware problems. Our data volumes have grown.

What next?

I have decided that there is no point running the company, paying salaries and other bills, if we can't find the money to protect our company from destruction. This means we need to find the money from somewhere to get additional hardware and maintain a full automated mirror of all data, like we do for our customers. Immediately.

Where we will get the money from is a question. The total budget will probably be just a few lakhs, which is a small fraction of what our company spends on revenue expenses in a single month. So wherever we get the money from, we will have to get it, and set up a full redundant server pair. We are currently living too close to the edge. This can't go on.