Guides & Checklists
Restore drills: proving your backups actually come back
A practical quarterly restore drill programme for agency IT: what to sample, how to time it against RTO, and what evidence to keep.
Backup software reports success when a job finishes writing data somewhere. That report does not tell you whether the data can be read back, whether the target platform will accept it, whether the encryption keys still exist, or whether the restored application will actually start and serve users. The distance between a green dashboard and a working system is where most recovery failures live, and the only dependable way to close it is to restore something real, on a schedule, and time how long it took.
A restore drill programme does not have to be elaborate. It has to be routine, sampled across the kinds of data the agency actually holds, performed against an isolated target, and documented well enough that an auditor or a cyber insurance underwriter can read the record without needing a verbal briefing.
Sample across four classes of data every quarter
Testing one file share repeatedly proves very little. Each quarter, pick one system from each of these classes and rotate the selection so that over a year or two everything important has been exercised at least once.
- Files and unstructured data. Shares, department drives, GIS layers, scanned records. Restore a folder and a single file, then check permissions, ownership, and timestamps, not just that the bytes arrived.
- Databases. Restore to a point in time on a separate instance, run the engine's integrity check, and validate against known records: a specific case number, a parcel, a payroll run.
- Servers and virtual machines. Do a full machine restore, boot it in isolation, and confirm that services start and that the application answers on its own port.
- SaaS and vendor-hosted systems. Mail and collaboration, ERP, permitting, public safety records. Take an export and reimport it somewhere, or require the vendor to demonstrate a restore.
Restore into an isolated target, never over production
Restores under test should land in a segmented network with no route to production, with DNS overrides in place and outbound mail, payment interfaces, and scheduled jobs disabled. A restored permitting or utility billing system that reaches a live integration can send real notices to real residents.
Treat the restored copy as production data for security purposes. It carries the same classification and the same obligations, including any criminal justice, health, or federal tax information rules that apply to the source system. Plan how the test environment will be secured while it exists and how it will be destroyed afterward.
Time the restore against the stated RTO
Measure wall clock time from the moment someone decides to restore to the moment a qualified person declares the system usable. Include the unglamorous steps: locating the restore point, provisioning a target, retrieving data from cloud or tape, rehydrating archive-tier storage, running database recovery, and validating. Those steps routinely dominate the total and are invisible in backup job logs.
Then compare the measured time to the recovery time objective in your continuity plan. If the measured time is longer, the drill has produced a decision, not a failure. Either the agency funds the change that closes the gap, or the department head formally accepts a revised objective. Leaving an unachievable RTO on paper is the worse outcome.
The failures drills catch that dashboards do not
- Unlicensed or unavailable restore targets. The image restores, then the operating system, database engine, or application refuses to run without a license key or an activation path that no longer exists.
- Missing keys and credentials. Backups are encrypted and the key material sits in a system that is itself down, or in a password vault that was never backed up separately.
- Applications that come back without their data. The app server restores cleanly, the database is a different vintage, and nobody documented the dependency order or the connection configuration.
- SaaS exports nobody has ever reimported. Exports frequently omit attachments, permissions, version history, workflow state, or audit metadata. You learn this on the day you try to load one.
- Scope gaps. Certificates, firewall rules, service accounts, scheduled tasks, and interface configurations often sit outside the backup job entirely.
- Runbooks that only work with one person present. If the drill stalls without a specific staff member, that is a finding worth writing down.
Record evidence the way reviewers will ask for it
Keep one short record per drill: date, system and data class, the restore point used and its age, who performed the work, the target environment, start and end times, elapsed time against the stated RTO, the validation steps and their results, defects found, tickets opened, and a sign off from the system owner. Retain the records under your agency's retention schedule.
Testing of contingency and recovery capability appears across common frameworks, including the contingency planning and system recovery controls in NIST SP 800-53, the recover function of the NIST Cybersecurity Framework, and the data recovery control in the CIS Critical Security Controls. Cyber insurance applications and renewals commonly ask how often restores are tested and whether backups are isolated or immutable. Specific control language and underwriting questions change, so confirm current expectations with your assessor, auditor, or broker rather than relying on a summary.
What to ask vendors and put in the contract
For hosted and SaaS systems, ask directly: who is responsible for backup, what a full export actually contains, what restore commitments exist, whether the agency may pull its own export on demand, and whether the vendor will participate in an agency-run restore test.
In the contract, seek provisions for vendor participation in a periodic restore test, delivery of a complete export in a documented, non-proprietary format on request and at termination, notice when the vendor's own backup or restore testing fails, and retention periods that match the agency's records schedule rather than the vendor's default.
Start small. Pick one system this month, book ninety minutes, restore it into isolation, time it, and write the one-page record. The first drill usually finds enough to justify the second.
HQT provides cloud and enterprise IT services to state and local government, including backup architecture, recovery planning, and restore testing support.
Frequently asked questions
How often should we test backup restores?
A quarterly cycle works well for most agencies: each quarter, restore one file share, one database, one server or virtual machine, and one SaaS or vendor-hosted dataset. Rotate which system is chosen within each class so that critical systems are exercised at least annually. Systems with short recovery time objectives, such as public safety or revenue applications, deserve more frequent testing than the general schedule.
What is the difference between backup verification and a restore drill?
Verification is usually an automated check that a backup file is readable or that its checksum matches, and it runs inside the backup platform. A restore drill puts the data back onto a target system, starts the application, and has a person confirm that real records are correct and usable. Verification tells you the backup is intact; only a drill tells you the agency can actually recover.
Do we still need to test restores for SaaS systems the vendor backs up?
Yes. Vendor backups protect the vendor's platform against its own failures, but they rarely protect you against accidental deletion, a bad bulk update, or the need to leave the vendor. Take your own export, reimport it somewhere, and confirm what it contains, since exports often omit attachments, permissions, version history, or workflow state. Contract terms about export format, export frequency, and restore commitments should be confirmed before renewal.
Where should we restore data during a test?
Restore into an isolated environment with no network route to production, with DNS overrides in place and outbound mail, payment, and interface connections disabled, so the restored system cannot contact residents or push data to live systems. Treat the restored copy as production data for security and compliance purposes, since it carries the same classification as the source. Plan in advance how the test environment will be secured and then destroyed.
What evidence do auditors and cyber insurers usually want from restore testing?
Keep a dated record for each drill showing the system tested, the restore point used, who performed the work, start and end times, elapsed time compared to the stated recovery time objective, validation steps and results, defects found, and an owner sign off. Retain those records under your agency's retention schedule so they can be produced during an assessment or a renewal. Specific expectations vary by auditor, framework, and insurer, so confirm the current requirements with them directly.