Engineering
Three integration failures that nobody notices until the data is wrong
File transfers, direct database access, and APIs each fail in a way that tells nobody. Here is how to make each failure visible.
Most integration failures are not dramatic: nothing crashes, and nobody gets paged. Data stops moving, or moves incorrectly, while every system involved reports that it is healthy. The failure surfaces later, when a clerk notices a record that never arrived, or a resident is told about a balance they already paid.
Three patterns account for much of what goes wrong between systems: file transfers that run without supervision, database accounts nobody documented, and APIs that nobody watches.
The silent file transfer
The nightly file drop is the oldest integration pattern still in production. One job writes a file, another picks it up, and the two ends never speak to each other again. The failures are quiet by construction:
- The export does not run. Nothing reports the absence, because the receiving side only reacts to files that arrive.
- The file arrives empty or truncated, often because it was read while it was still being written.
- The file arrives with yesterday's contents, because the export failed and the old copy was never cleared.
- The source system adds a column during an upgrade, and a loader that reads by position shifts every field after it.
The controls are ordinary:
- Alert on absence. Treat "no file by 06:00" as an event, not only an error return.
- Compare record counts against a plausible range, and halt the load at zero or far outside it.
- Write to a temporary name and rename on completion, so a partial file is never picked up.
- Validate the header or schema on arrival and reject a mismatch instead of loading it.
- Stamp each file with the source run identifier, so a repeat delivery is visible.
- Name an owner on both ends. A transfer with no owner on the receiving side is not monitored, whatever the runbook says.
Undocumented database access
The second pattern is one system reaching directly into another's database. It usually started as a reporting connection, a vendor connector, or a script someone wrote to solve a real problem.
The coupling is invisible. It lives in a configuration file on a server owned by another team, so the people planning an upgrade cannot know it exists. The upgrade lands, a table is renamed, and something breaks in a part of the organization that was never in the meeting. It fails in the other direction too: a heavy query from the consumer slows the source system during business hours.
Access at the database layer also goes around the role checks and audit logging the application provides. That matters when the data is regulated. The FBI's Criminal Justice Information Services (CJIS) Security Policy, the HIPAA Security Rule, IRS Publication 1075, and the Access Control and Audit and Accountability families of NIST SP 800-53 all call for access control and audit logging over the data in their scope. Which applies is a question for your privacy or security officer. The engineering point holds either way: a direct connection is a path around the controls the application is credited with enforcing.
Inventory every account that can reach the database and what each one is for. Issue each consumer its own least-privilege credential instead of a shared login. Publish views or a read replica as the interface. A view is a contract that can be held stable across upgrades. Base tables are an implementation detail.
The unmonitored API
An API at least gives both sides an error channel. It still fails quietly:
- A call returns a success status with an empty payload, and the caller records success.
- Pagination stops after the first page, so the same first batch moves every night and nothing else does.
- A retry without an idempotency key creates duplicates.
- A token expires, or a vendor retires a version, and the error is swallowed by a catch block that logs nothing.
- Rate limiting throttles the job, records are dropped, and the run still exits cleanly.
"The job ran" is not the same as "the data moved." Reconciliation separates them: how many records were eligible at the source, how many arrived, and what happened to the difference. Failed items belong in a queue a person works, not in a log nobody opens. Volume that drops to zero should raise the same alert an error does.
Test the alarm, not the integration
All three failures share one property: nothing tells anyone. The useful question is not whether an integration is monitored, but whether a person would find out within a day if it stopped working.
That is testable. In our own systems and reference builds we revoke a credential, block a file, or force an error response, then watch whether an alert reaches a person inside the window set for that integration. When none arrives, the integration is not finished.
Where to start
List every integration, its mechanism, its owner on each side, and how a failure would be detected today. An inventory that turns up connections nobody can account for has done its job.
Then work in order of consequence. Add absence alerting and record-count checks first to the transfers carrying money, safety, or eligibility data. Replace direct database access with a documented interface at the next upgrade of either system. Put the rest into procurement language before the next contract is signed: a published interface rather than base-table access, documented failure behavior, notice before a version is retired, and an obligation to surface failed records.