Almost every organisation has backups. Few of them know, with certainty, that those backups work. The difference only shows on the day the organisation can least afford to find out.

Why do backups fail?

The failure is rarely in the backup tool itself. These are the patterns we keep seeing when we assess a new environment:

  • A backup that succeeds but restores incomplete: database files are copied while they are being written, producing an inconsistent copy that fails on restore.
  • Missing scope: the data is backed up but the configuration is not: servers, certificates, permissions and firewall rules. You restore the data, then find that rebuilding the environment takes days.
  • A copy in the same place: a backup on the same machine or at the same site protects you from accidental deletion, not from a fire or ransomware.
  • A copy deletable with the same credentials: modern ransomware looks for backups first. If the server's account can delete them, they are not protected.
  • A failure alert nobody reads: the backup job has been failing for two months, and the alert goes to a mailbox nobody opens any more.

Two numbers before any tool: RPO and RTO

Before choosing any solution, the organisation has to answer two questions in business terms, not technical ones:

RPO: how much data can we afford to lose?

The recovery point objective sets how much time's worth of data you accept losing. If the backup runs every 24 hours, you are implicitly accepting the loss of a full working day. Ask the business owners, not the IT team, how many transactions can be re-entered by hand before the loss becomes unacceptable.

RTO: how long can we afford to be down?

The recovery time objective sets how long it takes for the service to come back. This is where the most common mistake happens: the time is measured from the moment the restore starts, while in reality it starts from the moment of the incident. It includes detecting it, deciding to restore, preparing the alternative environment, transferring, and then verifying.

Together, the two numbers set the cost. The closer they get to zero, the more sharply the price rises, so the same targets are not applied to every system: the accounting system is not the document archive.

The 3-2-1 rule, and why it became 3-2-1-1-0

The classic rule: three copies of the data, on two different storage media, with one of them off-site. The spread of ransomware added two more items:

  • One immutable or isolated copy: a copy that cannot be deleted or encrypted, even with administrator rights.
  • Zero verification errors: automated checks of the copy's integrity, not an assumption that it succeeded.

Testing: the part that always gets skipped

A restore test is not checking that the backup file exists. A real test means an actual restore into an isolated environment, running the system, verifying the data, and measuring how long it took.

Three levels of testing, run at different intervals:

  • Automated integrity check of the backup: daily
  • Restore of a sample (a file, a table, a mailbox) into a test environment: monthly
  • Full restore of a critical system, measuring the real time: at least every six months

And document the result: how long did it take? What did not work? Which step was not documented? The biggest value of a test is not proving the backup works, but finding the missing steps at a time when finding them costs nothing.

The question we ask in every assessment: when did you last actually restore a complete system? The most common answer is silence, or "at the last outage".

A recovery plan is not the backup file

Backup is a technology. A disaster recovery plan is a written operational decision that answers: who declares a disaster? In what order are systems restored? Who communicates with customers and staff? And where is the plan itself if the server holding it is the one that failed?

A plan that is printed or kept outside the environment, and tested at least once a year, is worth many times more than an expensive backup tool with no plan.

The bottom line

Start with two questions in business terms (how much data can we afford to lose, and how long can we afford to be down), then choose the technology that meets them, then test it on a documented schedule. Doing it in the reverse order is why so many organisations discover on the day of the incident that what they had was an assumption, not a backup.