Certificate lifecycle management: ending expired certificate outages

Reference article · 5 min read

A permit portal goes down on a Saturday morning. Every server is healthy and nothing was deployed. Eventually someone looks past the error page and finds a TLS certificate that expired at midnight. The renewal reminders went to the mailbox of an employee who left the agency.

That pattern repeats across VPN concentrators, RADIUS servers, records management interfaces, and the links between a dispatch center and state and federal systems. The cause is rarely technical difficulty. It is bookkeeping. The durable fix is a complete inventory, a named owner for every certificate, and automated renewal wherever the endpoint supports it. A bigger spreadsheet buys a few quiet months and then fails the same way.

Find every certificate, not just the ones you bought

The outages come from the certificates nobody recorded. Use several discovery methods at once, because each catches what the others miss.

External scanning. Scan your public address ranges and published hostnames, including nonstandard ports.
Certificate Transparency logs. Publicly trusted certificates are recorded publicly. Monitoring those logs for your domains surfaces certificates a department bought on a purchase card without telling IT.
Internal scanning and CA records. Look beyond port 443 at directory services, remote desktop, mail submission, and appliance consoles, and query each internal authority for what it issued.
Cloud and platform stores. Managed load balancers, API gateways, and container platforms hold certificates that never touch a server filesystem.
Devices and clients. Laptops, mobile data computers, printers, access controllers, and industrial control equipment may all carry client certificates.
Signing certificates. Code signing, document signing, and single sign-on federation signing certificates expire too, and the last one takes down every application behind it.

Give every certificate a named owner

An inventory without owners produces alerts nobody acts on. For each certificate, record the system it serves, that system's business owner, the technical contact, where the private key lives, and what has to be restarted after replacement.

Send notifications to a shared mailbox, never an individual address, and treat renewal as scheduled work with lead time: open the ticket early enough to absorb a change advisory board, a vendor dependency, and a maintenance window.

Automate issuance and renewal where the endpoint allows it

Automation is the only approach that holds up as certificate counts grow and validity periods shrink.

Web servers, proxies, and load balancers. Most support the ACME protocol, directly or through a client, against a public authority or an internal one that speaks ACME.
Domain joined systems. Windows autoenrollment issues user and machine certificates without human involvement once templates and permissions are right.
Network and mobile devices. Enrollment protocols such as EST and SCEP, driven by device management or network access control, cover most of this population.
Appliances that accept only manual upload. Keep these on a separate list with longer lead times and a written runbook, and ask the vendor about API support at every contract renewal.

Test the full path in a lower environment first. The common failure is a certificate that renews on disk while the service keeps presenting the old one because nothing reloaded it, so monitor what the endpoint actually serves, from outside.

Expect validity periods to keep getting shorter

The maximum lifetime of publicly trusted TLS certificates has been reduced repeatedly, and the industry body that sets those rules has approved further step reductions over the next several years. Confirm current maximums and effective dates with your certificate authority, since that schedule is set outside any single vendor.

The consequence is simple: any renewal that requires a person will recur far more often than it does now. Treat automated enrollment as an architecture requirement for new systems, the way you treat logging or single sign-on.

Deal with the internal CA nobody documented

Many agencies run a certificate authority stood up years ago, with an offline root that may not still be findable and a practice statement that was never written. Do not start by turning it off.

Inventory what it has issued and what still depends on it.
Check the fundamentals: signature algorithm, key sizes, private key protection, and whether the root actually reaches every trust store you care about, including mobile devices and contractor laptops.
Decide the destination: a documented two tier hierarchy you run, a managed PKI service, or platform native issuance for individual use cases.
Plan trust distribution and a migration window well before the old root nears expiration. Letting a root expire on its own is how organizations discover which systems hardcoded a trust anchor.
Write policies proportionate to what the authority issues. Where it supports criminal justice, health, or tax systems, confirm applicable requirements with your security office and counsel.

What to ask vendors and put in the contract

Ask during evaluation, not after installation:

Does the product support ACME, EST, SCEP, or an API for replacing certificates, or only manual upload?
Can a certificate be replaced without a service restart or outage window?
Does any client component pin a certificate or a specific issuer?
For hosted services using our domain name, who requests and renews the certificate, and what notice do we get?
What happens at expiration: hard failure, or a warning that still passes traffic?

Then put the durable parts in writing. Require the vendor to maintain valid certificates on everything it hosts, treat expiration at the highest incident severity, and give advance notice of any certificate or issuer change that needs action on your side. Make an automated enrollment protocol an acceptance criterion for new systems, and require documentation of every certificate delivered: purpose, chain, key location, and renewal procedure.

A reasonable first month is narrow: run discovery, put the results in one system of record, name an owner for everything expiring in the next quarter, and automate the renewals that need no vendor cooperation. HQT works with state and local agencies on infrastructure and security engineering, including certificate discovery, PKI design, and renewal automation.

Frequently asked questions

How do we find all the certificates in use across our agency?

Combine methods, because no single one is complete. Scan external address ranges and internal networks on more than port 443, monitor Certificate Transparency logs for your domains to catch certificates bought outside IT, query every internal certificate authority for issued certificates, and export certificate lists from cloud platforms, device management, and network access control. Load all of it into one system of record rather than several departmental spreadsheets.

What is the best way to track certificate expiration dates?

Track them in a single inventory that holds the expiration date, the system served, a named business and technical owner, the private key location, and the reload or restart step. Send alerts to a shared mailbox or ticket queue so they survive staff turnover, and generate a work ticket with enough lead time for change approval and maintenance windows. The stronger control is monitoring what each endpoint actually presents from outside, since a renewed file on disk does not always become the served certificate.

Can we automate certificate renewal on government systems?

Yes, for most modern endpoints. Web servers, reverse proxies, and many load balancers support the ACME protocol; domain joined Windows systems can use autoenrollment; network gear and mobile devices commonly support EST or SCEP through device management or network access control. Appliances that only accept a manual upload should be tracked separately with longer lead times and a documented runbook, and automation support should be a question you ask before buying the next one.

Are certificate validity periods getting shorter, and what does that mean for us?

The maximum lifetime of publicly trusted TLS certificates has been reduced several times, and the body that governs those rules has approved further phased reductions. Confirm the current maximum and the effective dates with your certificate authority, since the schedule is set industry wide rather than by one vendor. The practical effect is that any renewal requiring a human will repeat far more often, so automation capability becomes an architecture requirement rather than a convenience.

What should we do with an old internal certificate authority nobody documented?

Do not shut it down first. Inventory what it issued and what still trusts it, then review the signature algorithm, key sizes, private key protection, and whether its root is actually distributed to every relevant trust store. From there decide whether to rebuild a documented hierarchy, move to a managed PKI service, or migrate individual use cases to platform native issuance, and plan trust distribution well before the existing root nears expiration.