GalleryExhibit 02
The circuit breaker that could never close again
Asking a circuit breaker whether a request was allowed consumed the one probe that would have let it recover.
The client layer of an app that calls several remote endpoints while a session is live, with a breaker in front of each group.
What you can check here. Both breakers are in the case and in lib/sims/circuit-breaker.ts, and the tests assert that the broken one stays stranded rather than only that the fixed one recovers.
The object
See it happen
Fail three calls, wait out the backoff, then press “Pre-flight check” before sending anything.
Nothing yet — try a control above.
The breaker in the case is the same class the tests drive, on a virtual clock you advance by pressing a button. No network is involved; “fail a request” just records a failure.
What you are looking at
A caller that asks and then walks away leaves the breaker stuck half-open forever.
- Fail three requests to open the circuit, then let the backoff elapse.
- Press “Pre-flight check” — the answer is yes, and it costs you the probe.
- Now send a real request: it is refused, and no amount of waiting helps.
Wall text
What happened
Each group of remote endpoints sat behind a three-state circuit breaker: closed, open after three consecutive failures, then half-open once a backoff had elapsed to let a single probe through. A successful probe closes the circuit; a failed one reopens it with a doubled backoff. It is a textbook design and it was implemented correctly.
In the app, several call sites pre-flighted with canAttempt() before deciding whether it was even worth assembling a request — request pacing, input too short to be worth sending, a budget check that had already been answered. Only some of them then went on to call.
canAttempt() performed the open → half-open transition as a side effect of being asked. So a caller that asked and then bailed out left the breaker half-open with no probe in flight — and in half-open state canAttempt() returns false. Nothing was ever going to report back, so nothing was ever going to close the circuit. A caller that did proceed was rejected too, by the client's own guard, because the pre-flight had already consumed the transition.
Breakers were module-scoped, so every endpoint behind one stayed dead until a full page reload. A degraded-mode fallback designed to be temporary had quietly become permanent.
Root cause
A query that mutates. canAttempt() reads like a question and behaves like a claim, so every caller who merely wondered whether a request was worth making silently took the last one.
The two responsibilities are genuinely different: *may a request be issued right now* is a property of the breaker, and *I am issuing one, hold the slot for me* is a transaction. Merging them means the answer depends on who else has asked.
The fix splits them. canAttempt() becomes a pure query — and explicitly returns false in half-open, because a probe is already in flight and it alone decides the next transition. beginAttempt() reserves the slot, and is called from exactly one place: the line where a request really goes out.
The change
Code
canAttempt(now: number = Date.now()): boolean {
if (this.state === 'closed') {
return true
}
- if (this.state === 'open' && now >= this.nextProbeAt) {
- this.state = 'half-open'
- return true
+ if (this.state === 'half-open') {
+ // A probe is already in flight; it alone decides what happens next.
+ return false
}
- return false
+ return now >= this.nextProbeAt
}
/**
* Reserve the attempt slot for a request that is actually being issued
* now, moving an elapsed open circuit to half-open. The caller MUST then
* reach recordSuccess() or recordFailure() on every path — a reserved
* probe that never reports back leaves the breaker half-open forever.
*/
beginAttempt(now: number = Date.now()): boolean {
if (!this.canAttempt(now)) {
return false
}
if (this.state === 'open') {
this.state = 'half-open'
}
return true
}
- if (!breaker.canAttempt()) {
+ // beginAttempt, not canAttempt: this is the point where a request
+ // really goes out, so this is where the probe slot gets reserved.
+ if (!breaker.beginAttempt()) {
throw new BreakerOpenError(spec.breaker)
}
The test that catches it
Proof it stays fixed
This is the exhibit where a passing test suite was part of the problem. Unit tests around the breaker class existed and stayed green the whole time, because they called canAttempt() once per backoff window and then recorded a result — a reasonable way to test a state machine, and not how the application used it.
So the museum's version tests the *endpoint*, not the class in isolation, and reproduces the shape the app actually had: pre-flight, decide not to call, pre-flight again, and only then send a request. Both implementations are in the case, driven by a virtual clock, and the same two objects are what the tests drive.
The assertion that matters is the negative one. It is not enough to show the fixed breaker recovers; the test also pins that the broken one is *permanently* stranded — ten idle minutes later, still refusing. A test that only checked the happy path would have passed against both.
it("is stranded by a pre-flight that never becomes a request", () => {
const endpoint = new SimEndpoint("broken");
openIt(endpoint);
// Backoff elapses; a caller asks, then decides not to call.
expect(endpoint.preflight(30_000)).toBe(true);
expect(endpoint.breaker.state).toBe("half-open");
// Ten minutes later, a real request is still refused.
expect(endpoint.call(630_000, "ok")).toBe("refused");
expect(endpoint.breaker.state).toBe("half-open");
expect(endpoint.preflight(630_000)).toBe(false);
});
How it went
Discovery to regression test
Observed
Found by reading, not by failing
Nothing reported it. It surfaced while reading the call sites rather than the class: they do not all issue a request, and
The simulation ↗canAttemptmutates. The Broken state in the case reproduces the sequence in four button presses.Final fix
Split the query from the claim
lib/sims/circuit-breaker.ts ↗canAttempt()becomes side-effect free and returns false while a probe is in flight;beginAttempt()reserves the slot at the one place a request is actually issued.Pinned by a test
Test the endpoint, not the class
Eight cases drive both implementations through the app's own pre-flight-then-bail pattern, and assert that the old one never recovers. The class-level tests would have stayed green, which is the lesson.
tests/unit/sims/circuit-breaker.test.ts ↗
Verify it yourself
Sources
- Exhibit datacontent/exhibits/circuit-breaker-half-open.ts ↗
- Simulationcomponents/sims/breaker/breaker-sim.tsx ↗The state machine, the readouts and the virtual clock.
- Logiclib/sims/circuit-breaker.ts ↗Both breakers, in one class with a version flag.
- Testtests/unit/sims/circuit-breaker.test.ts ↗Eight cases, including the permanent-stranding one.