Data catalogs and data inventories: making agency data findable

Reference article · 6 min read

A new analyst asks a simple question on the first week of the job: what data does this agency actually have? The answer usually lives in people. Someone in the permitting division knows which table holds the current address, and which one looks current but stopped being updated when the old system was replaced. When that person retires, the knowledge leaves with them. At the same time, the records officer or privacy officer is being asked a different question by a different audience: what categories of personal information does the agency maintain, under what authority, how long must it be kept, and where are the copies? Both questions are about knowing what exists. They are not the same question, and they are rarely answered well by the same document.

Agencies get into trouble when they try to build one artifact that serves both. A catalog written for analysts becomes unreadable once it is loaded with legal citations. An inventory written for counsel becomes useless once it is stuffed with column names and load schedules. The workable approach is two related lists, each kept deliberately small, joined by a common list of systems so a change to one prompts a review of the other.

The technical data catalog

The catalog exists so that a competent person who does not already know your systems can find data, understand what a row means, and avoid using it wrongly. Its audience is analysts, developers, report writers, GIS staff, and the program manager who wants a number for a council packet.

A useful entry is short and honest. At minimum:

• What a row represents. Write it in plain language: "one row per issued building permit, one row per inspection visit." Not "permit data."
• Business owner and technical steward. Name the role and unit accountable for the content, and separately the person or team who runs the pipeline.
• Refresh and timeliness. How it is loaded, how often, and how far behind the source it can be.
• Known limits. What is missing, which years are unreliable, which fields are free text, which code values were retired, what happened when the source system was converted.
• Access and sensitivity. Its classification, who may see it, and how to request access.
• Lineage. Where it came from and which reports, dashboards, or downstream tables depend on it.

Automated harvesting from the database gives you names and data types quickly. It does not tell anyone what a row means or where the data lies. Fifty entries written by people who use the data beat five hundred harvested stubs.

The statutory data inventory

The inventory exists to answer obligations, not analytical questions. Its audience is the records officer, the privacy officer, counsel, the public records coordinator, and auditors. The drivers differ by jurisdiction, but the common ones are state records and archives law with approved retention schedules, the state public records act and the duty to conduct a reasonable search, sectoral privacy rules such as HIPAA, FERPA, CJIS Security Policy requirements, and IRS Publication 1075 for federal tax information, breach notification statutes, and in some places open data or transparency ordinances that require agencies to list the datasets they hold.

A minimum entry usually includes the record series or system name, the categories of information held with personal identifiers called out, the legal authority for collecting and using it, the retention schedule citation and the disposition at the end of that period, the system of record and every known copy, who the data is shared with and under what agreement, the responsible custodian, and any exemption the agency would assert in response to a records request.

These requirements vary by state and they change. Confirm the specifics with your state records program, your privacy officer, and agency counsel rather than relying on a general description.

Where the two lists meet

Join them on a single authoritative list of systems and applications, usually the one your IT asset or configuration records already maintain. The catalog describes tables and datasets underneath those systems. The inventory describes record series and information categories held within them.

Copies are the hard part and the reason the technical catalog is worth building first. Personal information rarely stays in one system. It moves into a warehouse, a dashboard extract, a monthly spreadsheet, a vendor's reporting environment, a backup. Lineage in the catalog is the most reliable input the inventory can get, because it shows where the data traveled.

Keeping either one current

Annual campaigns produce a document that is accurate for a month. Attach updates to events instead:

• Gate the work. No new production data feed without a catalog entry. No new system or major integration through intake and change review without an inventory entry. Put it on the checklist that already exists.
• Name roles, not people. Ownership assigned to an individual disappears at the next reorganization.
• Trigger on change. Schema changes, vendor replacements, new interfaces, and decommissioning each prompt a review.
• Publish where people work. A catalog nobody can search from their normal tools is a filing cabinet.
• Retire entries. Removing a decommissioned system is part of keeping the list trustworthy.

What to ask vendors and what to require in the contract

Before selecting any system that will hold agency data, ask whether the vendor can export a machine readable list of stored data elements including fields you configure yourself, whether metadata is available through an API or a standard dataset description vocabulary, where hosted metadata resides and who can see it, and what a full extract of data and documentation looks like at contract exit.

Then write it down. Useful terms include a data dictionary as a named deliverable updated with each release, the agency's right to export data and metadata in a machine readable format at any time and at termination, advance notice of schema changes that affect integrations, disclosure of hosting locations and subprocessors so the inventory can stay accurate, deletion and retention obligations aligned to the agency's approved schedule, and confirmation that the agency may publish its own metadata without restriction.

How to start without buying anything

Pick one program area, ideally one that is losing an expert or facing a records obligation it cannot currently meet. List systems before tables. Write ten catalog entries by hand in a spreadsheet using the fields above, then test them by handing a real question to someone who was not involved. Work the same ten systems through the inventory fields with your records officer. Only after that exercise will you know whether a catalog product earns its cost, and what it needs to do.

HQT works with state and local agencies on data platform, integration, and analytics work, including cataloging and documenting the systems an agency already runs.

Frequently asked questions