Systems Failure - Case Study

The Organisation

A range of businesses, covering multinationals to one-person companies, based in Central London.

What Happened

A London power company was experiencing severe problems in providing a stable supply. Voltages kept "spiking" and there was intermittent total loss of supply. None of these events was predictable (other than in its unpredictability).This event went on for several weeks until the service was stabilised.

Impact

Large companies which were affected had provided themselves with powerful generators to provide a second source of supply. They coupled these with sophisticated Uninterrupted Power Supply (UPS) facilities.

The smaller companies affected did not have the capacity to deal with the event in this way, as UPS and generators are expensive and outside their financial reach.

These companies suffered constant losses of processing and normal service to clients was disrupted. There were multiple cases of data corruption, as power-down sequences were not carried out and systems crashed as they lost power.Companies had to rework input, clean corrupted data and pacify disgruntled customers. Many lost work, had increased operating costs and lost client goodwill. It is possible that some companies went out of business due to these increased overheads.

Lessons

• Back up your data. Never rely on a single store of information.
• If your data stores are large, prioritise what is essential and make sure this is backed up in a structured manner that makes rebuilding priority systems and applications as quick and easy as possible.
• Consider a small-scale UPS. It may not allow you to keep your systems running, but it will allow you to perform a controlled power down. It will also reduce the impact of voltage 'spikes' and other variations in current.