Why integrations fail silently, and how to make them fail loudly

Silent failures look different on the surface, but they fall into a small number of shapes. Most integrations that have been in production for a few years exhibit several of them.

The swallowed error

An exception is caught, written to a log nobody reads, and processing continues. The code handled the error; the business never finds out it happened.

The success that isn’t

A call returns a success status with an error in the body, or an empty result that looks the same as nothing to do. Or a missing value is replaced with a default, and wrong data flows on as if it were right.

The partial batch

A scheduled job processes most of its records, skips the ones that failed validation, and reports that it finished. The skipped records are found later, by whoever needed them.

The retry that duplicates

A request times out after the other side has already processed it. The retry creates a second order, a second shipment or a second payment.

The silence

The partner stops sending, or a scheduled job stops running. Nothing fails, because nothing happens — so nothing alerts.

Schema drift

A counterparty adds, renames or repurposes a field. Parsing still succeeds, but the data now means something different.

Integration work tends to be scoped around the path where everything goes right: connect to the other system, map the fields, test with sample data, ship. Failure handling is the part that determines whether an integration works in production, and it is the part most easily left out, because nothing about the demo requires it.

The problem compounds when each integration is built separately, by whoever was available at the time. None of them share failure handling, each fails in its own way, and when one breaks, the person who wrote it is the only one who can say what it was supposed to do.

None of these techniques is new or exotic. What matters is that they are designed in deliberately, and that they work together:

Idempotent processing

Every message or request carries a unique key, and the receiver records the keys it has processed and ignores repeats. That makes retries safe: processing the same message twice has the same effect as processing it once. Martin Fowler’s catalogue calls this an idempotent receiver.

Retries with backoff, jitter and a limit

Transient failures are retried with increasing delays, a random element so that many clients do not retry in lockstep, a maximum number of attempts, and a timeout on every call. The Amazon Builders’ Library explains why each part matters.

Quarantine for what cannot be processed

Work that fails permanently — bad data, a broken rule, a record the other side rejects — is moved aside with the reason attached, rather than dropped or retried forever. In messaging systems this is a dead-letter channel; the principle applies to batch jobs too.

Replay

Once the cause is fixed, quarantined work is reprocessed from where it was set aside. Because processing is idempotent, replaying something that partly succeeded the first time is safe.

Validation at the boundary

Incoming data is checked against an explicit contract as it arrives. What does not conform is rejected loudly — never quietly coerced into defaults — and contract tests catch schema drift before production does.

Reconciliation

A scheduled comparison between the systems on each side of an integration — counts, totals, identifiers — that detects drift no individual error message would reveal.

Technical alerts — error rates, latency, a process that crashed — are necessary and not sufficient. The failures that cost the most tend to produce no errors at all. Alert on what the business expects to happen: no orders received from a partner in the last two hours of a working day, fewer invoices generated than the day’s orders predict, a nightly job that has not reported success by morning.

The last of those is the answer to the silence failure mode: a check that something happened, rather than a check that nothing went wrong. Google’s Site Reliability Engineering book makes the broader case for alerting on symptoms users can see rather than on internal causes.

An alert nobody owns is a silent failure with extra steps.

Every alert needs someone responsible for acting on it and a short written procedure for what to check first. Without both, alerts accumulate until they are ignored, and the integration is back to being monitored by its customers.

The durable fix is structural. Define a contract for what external systems can do, and implement it with an adapter per counterparty, so that partner-specific formats, identifiers and failure behaviour never reach the core of the application. Failure handling — idempotency, quarantine, replay, alerting — then lives in the boundary once, and every adapter inherits it.

The first integration built this way carries the cost of establishing the boundary. Every integration after it reuses the structure instead of rebuilding it inside the core application, and a new partner becomes an adapter rather than a project.

Automation is not a strategy if the underlying integration is broken.

Existing integrations do not need to be rebuilt to stop failing silently. The order matters more than the scope:

Inventory what exists

For each integration: what it moves, who depends on it, how a failure is detected today, and who can fix it. The answers tend to be uncomfortable, and they are the plan.

Add outcome and absence alerts

The cheapest step, and the one that stops customers from being the monitoring system.

Reconcile where drift costs money

Orders, payments, stock, invoices — wherever two systems disagreeing has a price.

Make processing idempotent, then retry

In that order. Automatic retries on processing that is not idempotent turn transient failures into duplicates.

Move integrations behind the boundary as they change

The next change to each integration pays for moving it behind the shared boundary, so the structural work never needs a budget of its own.

Integrations that fail silently are also one of the signs that a system as a whole has become expensive to change — see seven signs your software has become too expensive to change.