Synthetic data for testing agency systems without using real records

Reference article · 6 min read

Most agency systems are tested with a copy of production. Someone restores last night's backup into the test environment, and the same resident names, addresses, case notes, benefit amounts, and identification numbers now exist in a second place with weaker access controls, thinner logging, and a longer list of people who can reach them. It is the fastest way to get realistic test data. It is also one of the most common ways sensitive records end up somewhere nobody is watching.

Synthetic and masked datasets are the alternative. Built well, they give you the volumes, formats, and awkward edge cases you need for load testing, user acceptance testing, demos, training, and vendor troubleshooting, without putting real records where they do not belong. Built poorly, they produce a clean, well-behaved dataset that passes every test and then fails in the first week of production, because real records are not clean.

Why the production copy is a live exposure

• Lower environments are rarely held to the production standard. Production may sit behind multi-factor authentication, network segmentation, privileged access management, and complete audit logging. Test and development environments often carry shared accounts, broad developer access, untracked snapshots, and database dumps sitting on laptops.
• The number of people with access grows. Contractors, integration partners, QA staff, and vendor support engineers routinely receive access to non-production systems that they would never be granted in production.
• Regulated data stays regulated. Protections that attach to health information, criminal justice information, student records, federal tax information, or payment card data generally follow the data rather than the environment label. Requirements differ by program and change over time, so confirm the specifics with the governing authority or your counsel before assuming a test system is out of scope.
• The copy may itself be a record. A dataset sitting in a test environment can still fall under retention schedules, public records obligations, and litigation holds, which means it has to be found, produced, or defensibly disposed of.
• Breach notification does not care which server it was. State notification statutes generally turn on the exposure of defined categories of personal information, not on whether the host was labeled production.

Decide how realistic the data actually needs to be

Fidelity is not one setting. Write down what each environment needs before you build anything.

• Load and performance testing needs accurate volume, row distribution, index selectivity, and skew. The content of any individual record is close to irrelevant.
• User acceptance testing needs business plausibility and edge cases so program staff can recognize the scenarios they handle.
• Demos and training need readable, safe-to-screenshot data that will not appear in a slide deck by accident.
• Vendor troubleshooting needs the specific record shape that reproduces the defect, which is usually one record, not a database.

Practical ways to generate the data

• Mask a production extract. Replace sensitive values in place using deterministic transformations. This preserves real structure and real messiness, which is its strength. It also carries re-identification risk and almost always leaves sensitive content in free-text notes fields.
• Generate from the schema by rule. Drive generation from constraints, code tables, and business rules. This is the safest option and scales to any volume, but it only produces the cases you thought to describe.
• Generate statistically from production profiles. Fit distributions on real data, then sample from them. Useful for volume realism, and it requires care so the generator does not memorize and reproduce rare individual records.
• Curate an edge case library by hand. A few hundred deliberately ugly records, each tied to a defect or a known scenario, is often the single most valuable test asset an agency owns.
• Subset instead of cloning. Pull a referentially complete slice of production, then mask it. Smaller datasets refresh faster and expose less.

Keeping related tables consistent

Referential integrity is where most test data efforts fall apart. A masked parent record with an unmasked child record is worse than either alone.

• Use deterministic pseudonymization so the same input value always maps to the same output value, and apply the identical function everywhere that value appears, including denormalized copies and audit tables.
• Keep any reversal key out of the test environment entirely, held under production-level control or discarded.
• Preserve formats, lengths, and check digits where the application validates them, so masked values still pass front-end and batch edits.
• Preserve cardinality and skew. If one percent of cases have more than fifty associated documents, the test data needs that tail or performance testing will mislead you.
• Keep dates internally ordered. Shift a case timeline by a consistent offset rather than randomizing each date independently.
• Treat free text, scanned attachments, and log tables as unsafe by default and replace rather than mask them.
• Run the batch jobs, reports, and interfaces against the result. Foreign key checks alone will not catch a reconciliation report that no longer balances.

Where synthetic data quietly fails you

Synthetic data reflects what you already know about your data. The records that break systems are the ones nobody documented: names longer than the field anyone tested, addresses with no street number, a person carrying two active case numbers, sentinel dates like 99991231 carried over from a mainframe conversion, trailing spaces, mixed character encodings, and statuses that no current business rule can produce but that still exist in thousands of rows.

Three consequences follow. Performance tests on tidy data understate production cost. Analytics and model work should not be validated on synthetic records alone. And masked data can still be re-identified when enough quasi-identifiers survive. The practical answer is usually a small, tightly controlled environment with production-fidelity data and production-grade controls for the narrow cases that need it, plus synthetic data everywhere else.

What to ask vendors and put in the contract

Ask how the vendor builds test data today, whether any environment they control currently holds your production records, and what they do when reproducing a defect requires a real case. Ask whether the product ships a seed dataset or a generator, and whether it can mask an extract on export.

In the contract, state that production data may not be copied outside agency-controlled environments without written approval, and make data handling obligations explicitly cover development, test, demo, training, and support environments. Require a documented refresh and destruction process, extend breach notification duties to non-production systems, and make a usable synthetic seed dataset plus its generator a named deliverable rather than an afterthought.

How to start

1. Inventory where production copies already exist, including vendor environments, analyst workstations, and old snapshots.
2. Pick one system, ideally one with a painful upcoming release.
3. Write down the fidelity each environment genuinely requires.
4. Build a referentially complete subset with deterministic masking, then add a hand-curated edge case library on top.
5. Automate the refresh so teams stop asking for production restores as a favor.
6. Close the loop. Every production defect that test data failed to catch becomes a new record in the library.

HQT builds and modernizes systems for state and local government agencies, including test data strategy, masking, and environment design as part of that work.

Frequently asked questions