Data retention in analytics platforms is not your records schedule

Reference article · 6 min read

A records retention schedule tells an agency how long it must keep the official record of a business activity, and what to do when that period ends. A retention setting in a data platform tells a job how long to keep rows in a table, files in a bucket, or entries in a log before something deletes them. The two are related, but they are not the same decision, and treating one as a substitute for the other produces two familiar failures: purging analytical data the agency was obligated to preserve, or keeping every extract forever because nobody could say which copy was the record.

The workable goal is modest. For any dataset in the analytics environment, you should be able to say whether it is the official record, a copy of a record held authoritatively somewhere else, or derived data that is not a record at all. Retention rules then follow from that answer rather than from storage anxiety. Classification rules vary by state and by records series, so confirm the specifics with your records officer, your state archives or local government records program, and your counsel.

The record, the copy, and the derived table

Retention schedules generally apply to records by content and function, not by the software that happens to hold them. That principle cuts both ways in analytics work.

• System of record. The transactional application that creates and maintains the data is usually where the official record lives, and its retention is governed by the applicable schedule for that series.
• Convenience copy. A warehouse table replicated from that system is commonly treated as a duplicate kept for reference or reporting convenience. Many jurisdictions allow such copies to be destroyed when no longer needed, but the classification is your records officer's call, not the data team's.
• Derived data. Aggregates, scores, models, and published dashboards may be records in their own right when they document a decision, are reported externally, or support a grant or audit obligation.
• The copy that quietly became the record. If the source system purges on schedule while the warehouse keeps full history, the warehouse now holds the only surviving evidence. That is a records decision made by accident, and it should either be reversed or formalized.

Discoverable is not the same as being a record

When counsel asks whether warehouse tables are discoverable, the question is usually about production, not classification. Data in the agency's possession, custody, or control can fall within the scope of a public records request or litigation discovery whether or not it carries a records series designation. Calling something a convenience copy does not put it out of reach.

That is the strongest practical argument for bounded retention in analytics systems. Every copy you keep is a copy that may have to be located, searched, reviewed for exempt content, and produced. Unbounded ad hoc extracts multiply that work without adding analytical value.

Set analytics retention on operational grounds

Once classification is settled, set platform retention from operational need and write down the reason for each number.

• Raw ingest and landing zones. Keep only as long as you need to reprocess or replay a load, plus a margin for failed pipeline investigations.
• Detail fact tables. Keep for the longest analytical window someone actually uses, which is usually driven by trend and seasonality reporting or by grant and program reporting periods.
• Aggregates. Keep longer than detail. They are smaller, less sensitive, and answer most historical questions.
• BI caches and materialized extracts. Give them short lifetimes. A cache is a performance mechanism, not an archive.
• Platform and access logs. Drive these from security monitoring and incident investigation needs, and from any applicable state or program requirements.

One rule deserves emphasis: if you intend warehouse retention to exceed the source system's retention, treat that as a deliberate decision to create a new record and take it to your records officer.

Find the copies before you write the policy

A retention policy that only covers the warehouse leaves the shadow archive intact. Inventory the places copies actually land: non-production environments refreshed from production data, CSV exports on shared drives and in mailboxes, scheduled report attachments, notebook outputs, object storage with versioning or soft delete enabled, database snapshots, and any replicas held inside vendor platforms. Versioning and soft delete are especially easy to miss, because deletion appears to succeed while a recoverable copy remains.

Backups are usually managed as disaster recovery rather than as an archive, but that treatment holds up best when backup sets expire on a documented, uniform cycle. Discuss the boundary with counsel rather than assuming it.

Legal hold has to reach the delete jobs

A hold is worthless if automated deletion keeps running. Confirm that you can suspend scheduled purges in the warehouse, storage lifecycle rules, log expiry, and backup expiry, and that you can scope the hold narrowly enough to avoid freezing everything. Name who can trigger a hold, how the analytics team is notified, and how release is documented. Test the process once before you need it.

What to ask vendors, and how to start

Ask prospective and incumbent vendors:

• Where is agency data physically stored, including caches, backups, and subcontractor systems?
• Can retention be configured per dataset, and can deletion be verified rather than merely requested?
• Can lifecycle and purge automation be suspended for legal hold, and at what granularity?
• What happens to data at contract termination, on what timeline, and in what format can it be exported first?
• Is deletion logged in a way an auditor can review?

In the contract, require deletion within a stated period after termination, written confirmation of destruction, export of agency data before deletion, legal hold support, and a prohibition on secondary use of agency data. To start, pick your three largest or most sensitive analytics datasets, classify each with your records officer, document the retention you actually need and why, then turn off or bound the copies your inventory turned up.

HQT builds and supports data platforms for state and local government, including retention, access, and lifecycle controls in analytics environments.

Frequently asked questions