Getting data out of a vendor system you do not control

Reference article · 5 min read

Most agency data now lives in systems the agency does not operate. A vendor hosts the application, administers the database, and decides what leaves it. That works until the data is needed somewhere else: a warehouse for reporting, a sister system that needs case records, a public dashboard, or a successor platform at the end of the contract. At that point the constraint is rarely technical. It is whatever access the contract allows and whatever the vendor is willing to build.

The routes below are the ones that exist in practice, each with its own profile for freshness, operational load, and risk. The closing sections cover the access language to settle before go-live.

Decide what access you actually need

Vendors answer "can we get our data" differently depending on what is being asked. Write the requirement down first.

Freshness. Nightly is fine for most reporting. Minutes matter for operational handoffs between systems. Real time is expensive and rarely necessary.
Scope. The fields in the user interface are a subset of what the system stores. Decide whether you need the full record set, including lookup tables, audit trails, and attachments.
History. Current state is not the same as history. Ask whether you need to reconstruct a record as of a past date, and whether deleted records must remain visible.

The extraction routes compared

Vendor API. A documented interface, usually REST, returning records on request. Best for selective, current data and event-driven integration. Watch for rate limits, slow pagination, and endpoints that expose only part of the model. Many are built for transactions rather than bulk movement, so a full history pull can take days.
Scheduled file export. The vendor drops flat files, usually CSV or JSON, to secure file transfer or object storage on a schedule. This is the most durable and lowest-maintenance route for reporting: a file either arrived complete or it did not. Latency is the schedule, and deletes are the weak spot, so ask whether exports are full snapshots or deltas.
Reporting replica. A read-only copy of the database, refreshed on a lag, that staff query directly. It gives full fidelity and lets analysts write their own queries without touching production. It also exposes the vendor's internal schema, which changes without notice unless the contract says otherwise, and it is often priced separately.
Direct database access to production. Occasionally offered in single-tenant hosting. It is the fastest path to complete data and the riskiest, since one poorly written query can degrade the live system. Where it exists it should be read-only, on a separate credential, and covered by the application's security terms.
Change data capture. The vendor streams inserts, updates, and deletes from the database log into a queue or landing zone. Latency is low, load on the source is light, and deletes are handled cleanly, unlike most file exports. It is also the most demanding to operate: ordering, replay after an outage, and schema changes all need handling.
Report downloads and screen scraping. Staff exporting spreadsheets from the reporting module, or a script driving the interface. Treat this as a stopgap: fragile, hard to audit, and frequently prohibited by the terms of use.

Operational load is the part that gets underestimated

Every route becomes a production system somebody has to run. Budget for credential rotation, alerting when a run fails silently, reconciliation counts against the source, and regression testing after each vendor release. A nightly file drop with a row count check often serves an agency better than a streaming pipeline nobody is staffed to maintain.

Meaning is the quieter problem. Column names rarely match program definitions, status codes get reused, and soft-deleted rows look like live records. Ask for a data dictionary in writing, and validate a sample against cases staff already know before anyone publishes numbers from the extract.

What to ask before award

Which routes do you support today for a customer of our size, and which of them appear in a price list?
Is interface and export documentation available now, and may we review it during evaluation?
How much notice do we get before a schema or interface change, and how long does the prior version stay available?
At termination, what exactly do we receive, in what format, on what schedule, and at what cost?

Contract language to place before the system goes live

Ownership. The agency owns its data and derived records. The vendor holds them as custodian and takes no license beyond operating the service.
Standing access. A named extraction method, with frequency, scope, and format specified, available for the life of the agreement at no separate charge.
Documentation. Interface and schema documentation kept current, with advance written notice of breaking changes and a stated deprecation window.
Exit. A complete export in a non-proprietary format on request and at termination, including attachments and history, delivered within a stated number of days, plus transition assistance.
Verification. The right to test extraction before go-live and periodically afterward, so the exit clause is not first exercised in the contract's final week.
Security. Access paths covered by the same encryption, logging, and incident notification terms as the application, with credentials issued to the agency rather than shared.

Terms vary by jurisdiction, and records retention and disclosure obligations shape what the agency must be able to produce, so confirm wording with counsel and purchasing. If the system is already live, the renewal or the next change order is the moment to raise it: ask in writing, keep the response, and price the difference into the renewal.

HQT works with state and local agencies on data extraction, integration, and reporting around hosted vendor systems, including the access requirements that belong in the contract.

Frequently asked questions

Does the agency own its data in a hosted vendor system?

Ownership depends on what the contract says, not on where the data is stored. Common practice for public agencies is a clause stating that the agency owns its data and any derived records, and that the vendor holds them only as a custodian for operating the service. Without that language, practical access may be limited to whatever the vendor's standard product offers. Confirm the specific wording with your counsel.

What is the difference between a vendor API and a nightly file export?

An API returns records on request and suits selective queries and system-to-system integration, but it is often rate limited and slow for bulk history. A scheduled file export delivers a complete set of files on a fixed cadence and is usually simpler and cheaper to operate for reporting. Many agencies use both: files to load the warehouse, and the API for operational handoffs between systems.

Can we get direct access to the vendor's database?

Sometimes, though usually as a read-only reporting replica rather than access to the production database. A replica gives full fidelity without risking the live system, but it exposes the vendor's internal schema, which can change with each release. Direct production access is rare in multi-tenant hosting, and where it is offered it should be read-only, on a separate credential, and covered by the same security terms as the application.

What should a data access clause include before go-live?

Name the extraction method, frequency, scope, and format, and state that the access is included at no separate charge. Require current interface and schema documentation, advance notice of breaking changes, and a complete export in a non-proprietary format at termination with a transition assistance period. Add the right to test extraction before go-live so the exit path is proven while the agency still has leverage.

What can we do if the system is already live and access is limited?

Start by documenting what the vendor provides today and requesting the access you need in writing, then keep the response. The renewal or the next change order is the practical moment to add data access terms, because that is when the agency has something to negotiate with. In the meantime, a scheduled file export is usually the easiest route for a vendor to stand up quickly.