Technology
Data warehouses, lakes, and lakehouses for agencies
The main data platform architectures, what each is good for, and a sensible default for a public agency.
Agencies that want reporting across systems need somewhere to bring the data together. Three architectures dominate, and the terms are used loosely.
Data warehouse
A structured database organized for reporting, typically with data modeled into facts and dimensions. Data is cleaned and shaped on the way in. Strong for known reporting needs, business intelligence tools, and governed, consistent metrics. Less flexible for unstructured data and exploratory analysis.
Data lake
Storage for raw data in its original form, structured or not, at low cost. Flexible and cheap to fill, but without structure and governance it becomes a swamp: lots of data, little that anyone trusts.
Lakehouse
An architecture that stores data in open formats on low-cost storage while adding the structure, transactions, and governance of a warehouse. Modern cloud platforms support this pattern natively.
A sensible default for an agency
Most agencies' needs are reporting and dashboards across operational systems, with occasional analysis and a growing interest in machine learning. A cloud data warehouse, or a lakehouse on a cloud platform, with a governed modeled layer for reporting and a raw layer for exploration, serves this well. The specific product matters less than the modeling and governance.
What matters more than the architecture
- Reliable ingestion pipelines with monitoring
- A modeled layer with agreed definitions
- Access controls that respect data sensitivity
- Documentation and a catalog
- An owner for the platform and for each dataset
Cost considerations
Cloud data platforms charge for storage and for compute. Compute is where surprises happen. Scheduling, query optimization, and usage monitoring keep it predictable.
Regulated data
Public safety, health, and human services data carry restrictions. The platform design should separate and protect regulated data while allowing aggregate reporting, with the rules documented.