Security logging and SIEM: what to collect and how long to keep it

Reference article · 5 min read

A security information and event management platform, or SIEM, is only as useful as the log sources connected to it and the length of time those logs stay searchable. Agencies usually confront both questions at an awkward moment: a quote arrives priced on gigabytes per day that nobody has measured, or an incident response team asks for six months of authentication records and learns the source was never onboarded and retention was left at the vendor default.

Logging decisions are reversible going forward and permanent going backward. You can turn on a new source tomorrow, but you cannot create last spring's sign-in records after the fact. That asymmetry should shape both the onboarding order and the retention period you set at the start.

Onboard log sources in order of investigative value

Most agency investigations ask the same questions: whose account was used, what did the device do, and where did the data go. Connect the sources that answer those first, then expand.

Identity. Directory and single sign-on logs: sign-in success and failure, multi-factor prompts and denials, password and MFA method changes, privileged group changes, and application consent grants. Nearly every investigation starts here.
Endpoint. Detection and response telemetry from workstations and servers, including process execution, persistence, script activity, and detections with their outcomes.
Firewall and remote access. Perimeter allows and denies, VPN sessions, and remote access gateway activity, which establish where connections originated.
Cloud administrative activity. Tenant and subscription audit logs for productivity suites and infrastructure providers: role assignments, configuration changes, key and token creation, mailbox rule changes, and sharing changes. These records often expire quickly in the native service.
Email security. Delivery, quarantine, and phishing detection events, since email remains a common route for initial access.

After those five, add domain controller events, DNS, web proxy traffic, backup systems, remote monitoring tools, and the applications holding regulated data.

Retention is several decisions, not one

Retention is usually two numbers: how long logs stay hot, meaning indexed and immediately searchable, and how long they stay in archive, retrievable but slower and cheaper. Set the hot period to cover how far back an investigation reaches and the archive period to cover legal and records obligations.

Investigation reach. Intrusions are often discovered well after initial access, and insider or fraud cases surface months later during an audit. A hot window measured in months rather than weeks is what makes those investigations possible.
Criminal justice information. The FBI CJIS Security Policy sets audit and accountability requirements, including a minimum retention period for audit records. The policy is revised periodically, so confirm the current requirement with your state's CJIS Systems Agency.
Payment card data. Agencies accepting card payments fall under PCI DSS, which sets a minimum log history requirement plus a shorter window that must stay immediately available for analysis.
Health information. Agencies covered by HIPAA carry documentation retention obligations measured in years, and many apply the same period to security records. Confirm scope with counsel.
State records schedules. Many states treat system and audit logs as records with a defined disposition, which may be shorter or longer than you assumed. Your records officer should sign off on the setting.
Litigation and public records holds. A hold suspends normal disposition, so the platform needs a way to preserve a subset of data without keeping everything forever.

Some states borrow the structure of federal event logging guidance, which pairs roughly a year of active storage with a longer cold storage tail. That shape is a reasonable default.

Why ingestion volume drives cost

Most SIEM pricing is a function of how much data arrives, measured per gigabyte per day, per event, or in prepaid capacity tiers. A handful of sources generate most of the volume: firewall traffic, DNS, web proxy, and verbose application debug output. Identity and cloud audit logs, which carry the highest investigative value, are usually a small fraction of the total.

Filter and route at the collector so high volume, low value events land in cheap storage rather than the indexed tier. Confirm whether retention is priced separately from ingestion, and whether searching archived data carries its own fee. Ask what happens when you exceed the licensed volume: a hard stop that drops events is very different from an overage charge.

Platform or managed service

A platform without staff to watch it produces alerts nobody reads. Around the clock coverage takes more people than most agency security teams have, so many agencies pair a platform with a managed detection and response provider. Either way, decide who owns the detection content, tuning, and dashboards, and whether they transfer at the end of the term.

What to ask before signing

What is our actual daily volume by source, measured over at least a month, and what does the quote assume?
What are the hot and archive retention periods, and what does changing each cost?
Can we export raw logs at any time, in a documented format, without a fee?
Which of our sources have supported connectors today, including public safety and utility systems?
Who tunes detections, how often, and what alert volume should we expect after tuning?
For a managed service: what are the response commitments by severity, what may the provider do without approval, and how is performance measured?

In the contract, pin down retention periods and volume tiers, data ownership and export on termination, notification timelines when the provider detects an incident, and a transition assistance period. Confirm CJIS requirements and background screening for provider personnel able to view criminal justice data.

HQT works with state and local agencies on security monitoring architecture, log source onboarding, and retention design aligned to records and compliance obligations.

Frequently asked questions

How long should a local government keep security logs?

There is no single answer, because the period is driven by several separate obligations. Most agencies set a hot, searchable window long enough to support an investigation that reaches back months, then keep an archive copy to satisfy compliance and records rules. Criminal justice, payment card, and health data each carry their own minimum periods, and state records retention schedules may apply to logs as records. Confirm the applicable periods with your records officer, your CJIS Systems Agency, and counsel before configuring the platform.

Does the CJIS Security Policy set a log retention requirement?

Yes. The FBI CJIS Security Policy includes audit and accountability controls that require generating audit records for defined events and retaining them for a minimum period. The policy is revised periodically and the control language changes between versions, so agencies should confirm the current requirement through their state CJIS Systems Agency rather than relying on a summary or an older policy version.

Which log sources should we connect to a SIEM first?

Identity logs first, because nearly every investigation begins with an account. Then endpoint detection and response telemetry, firewall and remote access logs, cloud administrative audit logs for your productivity suite and infrastructure provider, and email security events. Those five answer most of the questions an investigator asks, and cloud audit logs in particular often expire quickly in the native service if they are not exported.

Why is our SIEM quote higher than we expected?

Most SIEM pricing scales with data volume rather than the number of users or devices, so the quote depends on how many gigabytes per day arrive. High volume sources such as firewall traffic, DNS, and web proxy logs can dominate the bill while contributing less investigative value than identity and cloud audit logs. Measure actual daily volume by source over at least a month, and ask whether retention, archive search, and overages are priced separately.

Should we buy a SIEM platform or use a managed detection and response service?

The deciding factor is staffing rather than the technology. A platform generates alerts continuously, and around the clock triage takes more analysts than most agency security teams have, which leads many agencies to pair a platform with a managed provider. If you use a managed service, establish in the contract who owns the detection content and tuning, how quickly the provider must notify you by severity, and that raw logs remain exportable to the agency.