Data & Analytics
Open Data & Transparency Portals
Publishing agency data in usable formats for residents, researchers, and journalists.
What this covers
Open data portals make budgets, contracts, permits, crime statistics, and service data available to the public in machine-readable form. Beyond transparency, they reduce public records request volume by publishing what is frequently asked for.
Typical activities
- Dataset inventory and prioritization based on request history and public interest
- Privacy review and de-identification of each dataset
- Portal platform selection or configuration, including open-source options
- Automated publishing pipelines so data stays current
- Metadata, data dictionaries, and usage guidance
- Public engagement and feedback channels
What you receive
A prioritized dataset catalog, a configured portal, automated publishing pipelines, and a governance process for adding datasets.
Considerations for public agencies
Public records law, privacy law, and CJIS restrictions define what can be published. We build the review step into the pipeline so that nothing goes public without a documented decision.
Signals that this work is due now
Agencies usually notice a pattern before they write a policy:
- The same extract is requested again and again under your state public records law, and each response is a staff member rerunning a query by hand.
- Outside groups are scraping your web pages, which means you are already publishing the data in a form that breaks when a page template changes.
- A department circulates figures nobody outside it can reproduce, because the underlying extract is not published.
- You already have a portal, and much of its catalog has not changed since the day it was loaded.
- An ordinance, executive order, or grant condition requires publication, and compliance today depends on one person remembering.
The last two deserve attention first: a visibly stale catalog is harder to recover than one that was never launched, and publication resting on one person stops when that person changes jobs.
Practices that hold up, and failures that recur
Treat each dataset as a small product: a named steward in the department that owns the source system, a refresh cadence published next to the data, a machine readable primary format with bulk download, and a pipeline whose failures alert a person. Column names and types are a public interface, so document them, version them, change them with notice, and keep superseded versions available so an older analysis can still be reproduced.
The recurring failures are easy to name. Tables posted as PDFs, which push extraction work onto every user. A catalog maintained by hand as a second copy, drifting from the system of record until the two disagree in public. No visible last-updated date, so a reader cannot tell current from abandoned. Aggregation that looked safe at launch and becomes identifying once the file is joined to another public dataset. After launch, the measures worth watching are datasets refreshed on their stated cadence, time to repair a failed pipeline, and whether request topics move away from what you publish.
Fitting publication to the systems you already run
Most of the work is plumbing between systems you already operate: the financial system behind budget and payment data, permitting, records management, the service request platform, and GIS. Read from a replica or a scheduled extract rather than production, since a public endpoint on an operational database becomes an availability problem the first time a dataset draws attention. Check vendor agreements before scoping, because some price API access or bulk export separately, cap call volume, or treat the schema as proprietary. If your authoritative layers already live in ArcGIS Online, some may be publishable through an ArcGIS Hub site, and harvesting that catalog beats holding copies that drift from it. Where a domain standard exists, publishing to it gives data users tools that already work: GTFS for transit schedules, Open311 for service requests, the Open Contracting Data Standard for procurement.
What to put in the solicitation
- Export of every dataset and its metadata in non-proprietary formats on demand and at termination, with pipeline code delivered into a repository the agency owns.
- Pricing that does not vary with download volume or API calls, so heavy public interest does not become an unplanned cost.
- A documented API with published rate limits alongside bulk file download, and catalog metadata conforming to a named profile and version, such as DCAT-US, so other catalogs can harvest your listings.
- Accessibility conformance to WCAG 2.1 Level AA, evidenced by a conformance report covering downloadable files and not only portal pages. The Department of Justice rule under Title II of the ADA sets that standard for state and local government web content and mobile applications, with compliance dates that depend on the size of the entity, so confirm which applies to you.
- Persistent URLs, a redirect policy for datasets that are renamed or retired, and retention of superseded versions.
- Pipeline failure monitoring that names who is alerted and the expected time to repair.
Write acceptance against a working pipeline carrying one of your own datasets rather than a demonstration catalog.
Working with HQT on this
HQT scopes this kind of work as a fixed-price or time-and-materials engagement with a written statement of work, named staff, and a defined acceptance process. We are candid about what we can take on and prefer a well-run project to an oversized one. Contact us to discuss a requirement, a draft scope, or an upcoming solicitation.
Frequently asked questions
Will an open data portal actually reduce our public records requests?
It can reduce the repeating portion of the workload, meaning the same salary roster, permit list, contract register, or incident extract that people ask for over and over. It will not reduce complex or case specific requests, and publishing more can produce new questions once the public can see what exists. The test worth applying is whether the topics of incoming requests move away from what you have published, which means recording request topics before launch as well as after.
Should we use an open source platform such as CKAN or a hosted commercial portal?
Both are workable, and the choice usually turns on who carries the operations rather than on feature lists. Open source options such as CKAN or DKAN avoid license cost and keep the stack under your control, but your team owns hosting, upgrades, and security patching. A hosted product moves that load to the vendor, and in exchange makes export rights, API pricing, and metadata portability contract terms you need to settle before signing.
We already publish GIS layers through ArcGIS Hub. Do we need a separate open data portal?
Usually what you need is one place for the public to look, not a second copy of your layers. If your authoritative geographic data already lives in ArcGIS Online, federating or harvesting that catalog into a single entry point avoids the situation where two versions of the same layer disagree in public. Non spatial data such as budgets, contracts, and permits still needs somewhere to live, so the common result is one public catalog that points at both sources.
How do we handle a dataset that is only identifying when it is combined with other data?
This is the mosaic effect, and it is the risk most often missed, because each field looks harmless on its own until the file is joined with another public dataset such as a parcel roll or a business license list. Practical responses include generalizing dates and locations, suppressing small counts, and settling the question dataset by dataset rather than by a blanket rule. Because upstream systems add and rename columns over time, the decision has to be revisited whenever a schema changes rather than cleared once before launch.
How much ongoing staff time does a portal take to keep running?
The recurring work is stewardship rather than publishing: answering questions from data users, reviewing schema changes before they break a pipeline, repairing failed refreshes, and clearing new datasets for release. Automated pipelines keep that modest, but no realistic plan puts it at zero. The usual arrangement is a named owner for each dataset in the department that produces it, plus a small coordinating role that keeps the catalog and the review process consistent.