Data conversion is the project, not the last two weeks of it

July 15, 2026 · 5 min read · HQT

Project schedules for a replacement system usually show the application built across most of the timeline and the data moved near the end. Conversion looks like a transfer: read from the old database, write to the new one. It is not. Legacy data carries the history of everything anyone ever did to it, including the workarounds staff invented when the software would not let them do their jobs.

We plan conversion as its own project, with its own schedule, acceptance criteria, and share of the budget. Here is how we sequence it.

Why the estimate comes in low

Conversion estimates are written against the schema: count tables and columns, map them to the new model, multiply. The schema is the easy half. The values are where the work is.

Free text fields hold structured data because there was no field for it. Retired code lists get quietly reused for something else. Identifiers are formatted differently depending on which decade the record was created in. The same person or parcel appears several times with small spelling differences. Workflow statuses contradict the documents attached to the case. References to scanned images no longer resolve.

None of those are engineering problems. Each is a decision belonging to someone with authority over the program, and collecting those decisions takes calendar time no column count predicts.

Profile the data before anyone commits to a date

The first activity should be profiling, not mapping: counts by entity and year, distinct value lists for every coded field, null rates, format variants for identifiers and dates, broken reference checks, and duplicate detection on the entities that matter most. Run it against a recent copy of production; a curated test extract hides the records that cause trouble.

The output is a defect list, and every item needs a named decision owner and a due date. An estimate written after profiling can be defended. One written before it is a guess the schedule absorbs later.

Profiling extracts and every later copy hold the same sensitive content as production. They need the same access controls, encryption, and disposal handling as the system of record, including in vendor environments.

Treat conversion as software, not as an event

Conversion logic belongs in source control and runs as a pipeline, unattended, from raw extract to loaded target, producing the same reports every time. It has to be rerunnable from scratch, because it will run many times before the run that counts.

The discipline that matters most is that nothing gets fixed by hand in the target. A record corrected in a spreadsheet after a load disappears on the next run. Corrections go into the transformation rules, or back into the source system so the next extract carries them.

Decide what does not move

Not all of it has to come across. Retention requirements come from approved records schedules and from program rules attached to specific funding or regulation, and they vary by record series. Federal agencies work from schedules approved by the National Archives; state and local agencies work from their state's schedules and local policy. A litigation hold or a pending records request overrides the schedule. Those calls belong to the records officer and counsel, not the project team.

Once that is settled, the usual pattern is to convert active records, archive closed records where they stay searchable and producible, and keep the legacy system read only for a defined period. Running the old system costs something too, including patching and network isolation for unsupported software, so its retirement date belongs in the plan.

Validate at three levels

1. Reconciliation. Control totals by entity, year, and organizational unit, with financial amounts matching exactly. Differences get explained record by record, not waved off as rounding.
2. Record level comparison. Automated field by field comparison on a sample chosen to include the hard cases, referential integrity checks in the target, and verification that attached documents open.
3. Business validation. Staff work real cases in the converted environment before cutover. A clerk pulling up the files they know are bad finds problems no automated check was written to catch.

Anything that does not match becomes an exception with a reason code and an owner. A run that reports no exceptions usually means the checks are too weak.

Rehearse the cutover

Full rehearsals on a copy of production, timed end to end, give the runtime, which usually sets the outage window. They also surface the steps nobody wrote down.

The cutover plan needs a freeze window, a method for carrying over records changed during the freeze, a rollback point with a named person authorized to call it, and a queue with an owner for the records staff find in the weeks after go live that did not convert cleanly.

What this means for an agency

Fund and schedule conversion as its own phase. In a solicitation, make early profiling a deliverable whose findings both sides see, ask vendors how they validate rather than whether they will, and write reconciliation reports and staff led business validation into the acceptance criteria. Name a data owner with authority to decide what a bad record should become, and give that person real time on the project.

A new system that runs correctly on data staff do not trust is still a failed project. Conversion is where that trust is built or lost.