Cloud & Enterprise IT
Disaster Recovery & Continuity
Backup, recovery, and continuity planning that has been tested rather than assumed.
What this covers
Ransomware, wildfire, earthquake, flood, and plain hardware failure all threaten agency systems. Recovery depends on backups that are protected, restores that are practiced, and plans that name who does what.
Typical activities
- Business impact analysis to set recovery time and recovery point objectives by system
- Backup architecture with immutable or offline copies resistant to ransomware
- Disaster recovery design using cloud regions, secondary sites, or a mix
- Restore testing and tabletop exercises with documented results
- Continuity of operations plans that cover people and processes, not only technology
- Alignment with emergency management plans and mutual aid arrangements
What you receive
Recovery objectives by system, an implemented backup and recovery design, test records, and a maintained continuity plan.
Why it matters
Agencies have a public duty to keep services running. A recovery plan that has been exercised is the difference between a difficult week and a difficult year.
Signs the current arrangement has not been tested
Any of the following can be confirmed internally in an afternoon, before a vendor is involved.
- The last restore anyone performed was a single file or mailbox rather than a full system with its dependencies.
- Backup jobs are monitored for completion, but nobody checks whether the protected set still matches what production runs today.
- The backup console, the virtualization platform, and the production directory share one identity provider and one set of administrator accounts.
- Protected copies sit on the same array, in the same building, or in the same cloud account as the systems they protect.
- Recovery objectives exist on paper but were never priced, so nobody has decided what faster recovery is worth paying for.
- The continuity plan names staff who have left, or an alternate site the agency no longer holds.
Failure modes to check for
Each of these can coexist with a clean set of backup reports.
- A recovery site that depends on the same directory, management plane, or administrator accounts as production, so reaching one is reaching both.
- Restore tests that stop at the database or virtual machine and never reach name resolution, certificates, licensing, interfaces, and the print and payment paths a clerk needs before the service is usable.
- Coverage reported as backup success rather than restore success.
- Immutability asserted in a policy rather than enforced by the platform through object lock or retention lock under credentials separate from production.
- A single recovery objective applied to every system, which generally means the cost tradeoff was never put to program owners.
- Runbooks only their author can follow, which surfaces the first time the author is unavailable.
- Exercises that end at "IT restores the system" without rehearsing the counter and field work a department would run in the meantime.
Systems held by vendors and other agencies
Dispatch, records, permitting, finance, and case management are often hosted by a vendor or by another government, and the agency's recovery position there is whatever the agreement says. Read each one for recovery time and recovery point commitments, for whether the vendor's backups exist for the customer's benefit or only for the vendor's, for the right to an independent export, for the right to take part in a test, and for notification timelines after an incident. The absence of a commitment is itself a finding.
Productivity suites deserve separate attention, because the provider keeps the service running while retention of mail, files, and sites is governed by configuration and licensing that change over time. Where a city depends on county or state facilities, someone else holds the restoration priority list, so ask to see it and confirm where the agency sits on it. Public safety answering points, radio, and utility control systems carry their own continuity requirements under different staff, so reconcile with those plans rather than writing a second set.
What to specify in a solicitation, and how to check afterward
Specific requirements produce comparable proposals. Consider making an observed restore an acceptance criterion: a named system rebuilt from protected copies into an isolated environment, timed, and witnessed by agency staff. Require a demonstration that a protected copy cannot be deleted or shortened using production administrator credentials. Require recovery runbooks and any infrastructure code in editable formats in agency controlled repositories. Ask what the vendor's own continuity arrangements are, and whether the vendor will hold a copy of agency data. If the agency may later seek reimbursement under a disaster assistance program, confirm the documentation that program expects, since eligibility rules are set by the program rather than by the contract.
Afterward, judge the work on measured behavior rather than on the binder. Track actual recovery time against the objective across successive exercises, check whether newly provisioned systems enter protection automatically or only when someone remembers, and count the undocumented dependencies each exercise uncovers.
Working with HQT on this
HQT scopes this kind of work as a fixed-price or time-and-materials engagement with a written statement of work, named staff, and a defined acceptance process. We are candid about what we can take on and prefer a well-run project to an oversized one. Contact us to discuss a requirement, a draft scope, or an upcoming solicitation.
Frequently asked questions
How often should an agency test its disaster recovery plan?
No single frequency fits every agency, and the binding answer comes from the rules that apply to the particular system and the data it holds. Federal contingency planning guidance and the contingency planning controls in NIST Special Publication 800-53 call for testing at a frequency the organization defines, and the control baselines built on them commonly set that at no less than annually. A more useful trigger than the calendar is material change, such as a new system, a platform or identity migration, a change of hosting vendor, or turnover among the staff who would actually run the recovery. Highest priority services tend to do better with frequent narrow tests than with one large annual event.
What is the difference between disaster recovery and continuity of operations planning?
Disaster recovery is the technical restoration of systems and data. Continuity of operations covers how essential government functions keep running while systems are unavailable, including orders of succession, delegations of authority, alternate facilities, vital records, manual procedures, and staff notification. An agency can restore a server quickly and still fail residents if counter staff have no way to take a payment or issue a permit in the meantime. The two plans should reference each other and share the same assumptions about how long an outage could last.
Do we need our own backups for SaaS and vendor hosted systems?
Usually yes, or at least an independent export that the agency controls and can read without the vendor. Vendor platforms are engineered for availability, which is not the same as protecting the agency from a mass deletion, a bad configuration change, a contract dispute, or the vendor's own incident. Check what the contract actually commits the vendor to, what export formats exist, and how quickly a usable copy could be obtained. Where an independent copy is not practical, record that as a risk leadership has accepted rather than leaving it undiscussed.
How do we set recovery time and recovery point objectives we can actually meet?
Start by asking program owners what happens to residents and staff hour by hour without a service, rather than asking IT for a number. Group systems into a small number of tiers, price the architecture each tier would require, and let leadership choose with the cost in front of them. Objectives that have never been priced or demonstrated in a test are aspirations rather than commitments, and they are the ones that fail publicly.
How can we verify that our backups would survive a ransomware attack?
Test deletion, not only restoration. Have someone holding production administrator rights attempt to delete a protected copy or shorten its retention, and confirm that the platform refuses. Also confirm that the backup system authenticates separately from the production directory, that at least one copy is offline or immutable, and that a restore can run into an isolated environment that does not depend on the systems presumed compromised. Keep the results as evidence for auditors and insurers.