A deadlock doesn't care how many times you retry it

A fix for a capacity problem that itself needs capacity to run is not delayed, it is deadlocked — and no amount of retrying breaks a deadlock, because every retry just asks the full system for the one thing it doesn't have. One week this summer at Organ, we found that shape twice inside our own agent platform: once in the outage that started it, and once in the task we dispatched to fix it.

The week in three numbers

• 1 — the number of EC2 instances running the entire platform's task infrastructure: a single t3.xlarge, Auto Scaling Group pinned at min = max = desired = 1. That instance fits roughly two concurrent agent tasks. Every scheduled department-head wake-up, every developer-workflow dispatch, every background job competes for the same two slots.
• 7 of 7 — dispatched tasks that failed inside two short windows on a single day (three at 07:11–07:12 UTC, four at 22:40–22:56 UTC), and zero failures outside those two windows. The failures wore three different labels — JSON_EXTRACTION_FAILED, PROCESS_CRASHED: exited with code null, and ECS task did not reach RUNNING before the deadline — but every one traced back to the same saturated host. A null exit code means a process was killed by a signal, not that it crashed on its own; a container SIGKILLed mid-run produces exactly the "missing session ID" signature that gets mislabeled as an LLM output bug.
• 9 of 9 — simultaneous recovery containers that fired to catch up on a 40.7-hour platform outage, and the number of those nine that survived: zero. All nine ended FAILED or TIMED_OUT. The recovery replay hit the same one-instance ceiling as the tasks above and amplified the outage instead of ending it.

The connective tissue across both incidents is the same sentence, written almost verbatim in that week's engineering review: we are generating our own failure rate. Nothing external attacked the platform either time. The platform's own scheduling — a batch of dispatches, a burst of catch-up replay — exceeded the platform's own fixed capacity, and the system reported that collision as three unrelated bugs instead of one saturated box.

What changed that week

1. The task sent to fix ECS capacity failures was itself killed by an ECS capacity failure. One of the three tasks in the 07:11 burst was dispatched specifically to fix start-deadline failures on the runner fleet. It never got far enough to write a line of code — it died during task-spec generation, in the same burst, on the same saturated host, wearing the mislabeled JSON_EXTRACTION_FAILED tag. Naming this precisely — a deadlock, not three independent regressions — is what turned "8 failures, no obvious pattern" into a single, fixable root cause, and it changes what gets built next: not eight separate patches, but one concurrency cap.

2. Bounding concurrent dispatch was identified as the single highest-leverage fix and sent for build. Rather than chase the three error labels individually, engineering traced all seven failures to one mechanism — department-head wake-ups batch three to four task dispatches at once against a fleet sized for two — and dispatched the fix at the source: cap how many tasks launch together, so the remediation stops competing with itself for the resource it's trying to free up.

3. A new watchdog was built specifically to catch a recovery that makes things worse, not just an outage that starts. The existing liveness detector was itself one of roughly 40 schedules that had silently gone dark months earlier, which is why the 40.7-hour outage ran undetected. Its replacement checks for the exact failure shape found that week — a catch-up replay where every recovery attempt dies — and is built to always write an entry, even on a healthy day, so silence can never again be mistaken for "nothing wrong." It also runs deliberately away from the 09:00 cluster where the original nine-container thundering herd occurred.

4. The outage's root cause was corrected on the record before a fix got built on the wrong premise. An earlier theory held that nothing was actually due to run during the outage window, which would have made it a non-event. Direct evidence — timestamped rows created by the eventual recovery, each one carrying the exact scheduled time it should have fired — disproved that: the window was a real 40.7-hour gap, not 32 as first estimated, and both of the safety detectors meant to catch a gap that size stayed silent through the entire thing. Getting the outage's actual shape right, in public inside our own workspace, is what let the watchdog above be designed against the failure that happened instead of the failure that was assumed to have happened.

What an indie hacker can take from this

Know your concurrency ceiling before your own schedule finds it for you. If every cron job, background worker, and CI run you own shares one box, you have a real, fixed number of things that can run at once — and if you ever schedule more work than that number, in your own calendar, you will manufacture an outage without any external cause at all. Count your slots. Count what you've scheduled against them. Those two numbers colliding is not bad luck, it's arithmetic.

A recovery mechanism that replays a whole backlog at once is not a recovery mechanism, it's a second outage waiting for a trigger. When work piles up during downtime and then all of it fires back at once against the same fixed capacity that just failed, the catch-up burst can lose 100% of its own attempts — worse than doing nothing, because it also occupies the capacity anyone trying to actually fix the problem would need. Cap and stagger any replay logic before you need it, not after you watch it fail nine-for-nine.

When several failures cluster in the same narrow time window, check the clock before you check the code. Three differently-worded errors that all happen inside a 23-minute burst are a much stronger signal of a shared timing or capacity cause than three separate bugs are of pure coincidence. It's tempting to fix the label you can see — retry this exception, patch that error message — when the real fix is one line in your infrastructure config limiting how much can run at once.