Chaos engineering payment systems: start with the DNS TTL

A sixty-second DNS time-to-live produced a ninety-three-second failover window. During those thirty-three unaccounted seconds, roughly thirty-seven thousand requests at 400 transactions per second went to an endpoint that was already dead.

Nothing in that sentence is a bug. The TTL was set deliberately. The health check worked. The replacement endpoint came up correctly. The gap came from intermediate resolvers caching the record past its stated lifetime, which is behaviour DNS is entitled to have. This is the case for chaos engineering payment systems rather than reviewing them: no code review finds a defect that exists only in the space between two correct components.

The figures come from a published account of chaos experiments on ECS-hosted payment infrastructure, and they are unusually specific for this kind of write-up. Most resilience articles describe practices. This one gives numbers, which is what makes it useful.

The incident that justified the budget

The programme started the way these programmes always start. A payment processor's settlement service went dark for four hours during peak reconciliation. The cause was a routine ECS task replacement during a deployment, combined with a subtle dependency on a single Redis node, which together produced cascading timeouts.

Read that again and notice what is absent: there is no faulty component. A deployment ran as designed. A cache node was where it had always been. The dependency between them was undocumented because nobody had written it down, and nobody had written it down because nothing had ever exercised it at the same time as a deploy.

Four hours of settlement downtime is not four hours of inconvenience. It is a reconciliation backlog that arrives at the same moment as the next day's volume.

What the experiments actually found

Three findings are worth stating in full, because each is a class of problem rather than a one-off.

Retry policies amplify load precisely when load is the problem

Injecting five hundred milliseconds of latency, with a retry policy of three attempts using exponential backoff and jitter, increased sustained database connection usage by approximately 2.4 times over baseline. Backoff and jitter are the correct pattern. They are recommended everywhere, including by us. They still multiply connection pressure by nearly two and a half when the dependency slows rather than fails.

If your connection pool is sized for normal operation with modest headroom, a latency event does not degrade the system gracefully. It exhausts the pool, and pool exhaustion looks like a database outage to everything upstream.

The sizing implication is concrete. If you size the pool at peak concurrency plus a comfortable twenty or thirty per cent, a 2.4 times multiplier eats that headroom immediately. Either size for the amplified case, or cap total in-flight retries across the service rather than per call site, so that three attempts per request cannot become three times your entire request volume.

Spot interruptions leave transactions in a state your schema cannot express

A settlement job processing around fifty thousand accumulated transactions in seventy-five seconds was interrupted at the fifty-eight-second mark. Fourteen thousand transactions were left in an ambiguous state.

Ambiguous is the important word. Not failed, which you can retry. Not succeeded, which you can move past. Ambiguous, which means somebody reconciles them by hand, or the job re-runs and you find out whether it is genuinely idempotent under partial completion. Most batch jobs are idempotent in the sense that running them twice from a clean start is safe. Far fewer are idempotent when killed at second fifty-eight of seventy-five.

Availability zone degradation defeats the scheduler that is supposed to handle it

During an AZ failure, ECS tasks entered a start-stop loop: the cluster's desired task count was never satisfied because the scheduler kept placing tasks where they could not survive. The orchestrator was doing its job. Its job simply did not include noticing that the destination was the problem.

The consequence is worse than reduced capacity. A cluster in a placement loop reports itself as actively scaling, so autoscaling alarms stay quiet and dashboards show tasks starting rather than a service down. You lose the zone and the signal that you lost it at the same moment. The remediation is capacity provider strategies that exclude a degraded zone rather than retrying into it, plus an alert on task start rate as a first-class metric — a number most teams never graph.

The configuration values that came out of it

The remediations are specific enough to copy, which is rare:

Setting	Value	What it prevents
deployment_minimum_healthy_percent	100	Capacity dipping below demand mid-deploy
health_check_grace_period_seconds	120	Killing tasks that are still warming up
stopTimeout	120	In-flight settlement work truncated on shutdown
Route 53 TTL	10 seconds	The failover gap described above
JVM networkaddress.cache.ttl	Matched to DNS	The runtime caching a record the resolver already dropped

That last row is the one teams miss. Lowering the Route 53 TTL achieves nothing if the JVM holds its own resolved address indefinitely, which older defaults do. You will have fixed the infrastructure and left the application pointing at a dead host.

How to start without breaking production

The staged progression described is conservative, and correctly so for anything touching money:

• Stage one: staging, with production traffic shadowed onto it.
• Stage two: production, but only services off the transaction path.
• Stage three: secondary transaction-path services, in a low-traffic window between three and five in the morning.
• Stage four: primary services, with full rollback automation.

Reaching stage four typically takes six to twelve months. If that sounds slow, compare it with four hours of settlement downtime and the reconciliation that follows.

Three starter experiments are recommended, and they are the right three: ECS task replacement under load during a deployment, database connection pool exhaustion, and measuring service discovery failover timing. AWS Fault Injection Service handles the injection; the discipline is in deciding the hypothesis and the abort condition before you run it.

Define steady state before you break anything

An experiment without a stated steady state is an outage with better paperwork. Before injection, write down the metric that represents healthy operation and the threshold at which you abort. For a payment path that is usually authorisation success rate and settlement lag, not CPU.

Two rules make the difference between a programme that survives its first bad run and one that gets banned. The abort condition must be automated, because a human watching a dashboard at 3am will hesitate. And the experiment must be announced to whoever answers the pager, every time, even when you are confident, because the cost of one person spending twenty minutes debugging your injection is the cost of the whole programme's credibility.

Record the result whether or not it was interesting. A hypothesis that held is evidence you can point at during the next architecture review, and it is the only way the exercise compounds.

When chaos engineering payment systems is not worth it

Chaos engineering is a practice for systems whose failure modes you have run out of other ways to discover. If you are pre-launch, or running a single service with one database and no autoscaling, you do not have emergent behaviour yet. You have bugs, and you should find those with tests.

The threshold is roughly: more than one service in the transaction path, more than one availability zone, and an orchestrator making placement decisions you do not directly control. Below that, the money is better spent on test coverage and automation. Above it, integration tests stop being able to reach the interesting failures, because the interesting failures are between the components rather than inside them.

What this means if you run payment rails

We integrate payment flows into commerce platforms — FlashNow handles vendor payouts and consumer transactions across three platform roles — and the pattern that generalises is not the DNS number. It is that every one of these failures was a timing assumption nobody had written down.

Sixty seconds of TTL assumed resolvers honour TTLs. Three retries assumed the dependency fails rather than slows. A seventy-five-second batch assumed it would be allowed to finish. None of those assumptions were unreasonable and none were documented, so none were tested.

The practical move, before any of this becomes a programme: write down the timing assumptions in your transaction path. How long may a dependency take before you give up. How long a shutdown may take. How stale a DNS answer may be. How long a batch may run. Most teams cannot produce that list, and the exercise of writing it finds the gaps faster than the first experiment will.

Then pick the cheapest assumption to test, and test it at three in the morning.