How to Make an Integration Fail Gracefully: Retries, Alerts and Reconciliation

Every integration will fail eventually. The other system will be down for maintenance. A password will expire. Someone will change a field. A record will arrive that breaks an assumption. The question is not whether it happens, but what the integration does when it does.

Integrations that are built only for the happy path tend to fail silently, which is worse than failing loudly. Here is how to build ones that fail gracefully.

Make it safe to run twice

An idempotent process gives the same result whether it runs once or three times. This matters because the safest response to many failures is to run it again. If running twice creates duplicate invoices, you cannot do that. Use unique keys, check whether a record already exists before creating it, and update instead of insert where it makes sense.

Retry the right errors

Some failures are temporary: a timeout, a busy server, a brief network problem. Retrying after a short, growing delay usually fixes them. Other failures are permanent: invalid data, a missing required field, a rejected login. Retrying those only repeats the failure. Treat the two differently, and put a limit on retries so the process does not loop forever.

Quarantine bad records instead of stopping

When one record out of five thousand is bad, the process should set it aside, record why, and carry on with the rest. A table of rejected records with the reason and the original data lets someone fix and resubmit them. A process that halts on the first error means one bad row blocks everything.

Log with context

A log line that says error is useless. Record which record, which step, what the other system returned, and when. When something goes wrong at 3 am, the person fixing it should be able to understand what happened without recreating it.

Alert a human, with enough to act on

Alerts should reach someone who can do something, and say what failed and what to check first. Too many alerts and they get ignored, so alert on what needs action, such as repeated failures or a job that did not run, not on every transient blip.

Watch for what did not happen

The scariest failure is the one that produces no error because the job never ran. Monitor that expected jobs completed on time and that data is as fresh as it should be.

Reconcile

Counts and totals should match between systems. A daily check that compares the number of orders sent with the number received, or the total value on each side, catches problems that no individual error message will. When they do not match, you want to know the same day.

Plan for the boring failures

Credentials expire, certificates lapse, and vendors change their interfaces with a few weeks' notice. Keep a calendar of expiry dates and a contact for each system, and write a short runbook: what the integration does, how to tell if it is healthy, how to restart it, who to call.

For the basics of how these systems talk to each other, see what a REST API is and why your operations team should care, and for the design principles behind a good integration, see ERP integration best practices.

If you rely on integrations that nobody is quite sure are working, reach out. A review of the failure handling is often the cheapest way to avoid an expensive surprise.