The Test Wasn't Flaky. The Server Was Quitting.

For most of July, the last gate before production on this site failed about four times in ten. It is a Playwright suite I call the prod smoke: it builds the real Worker bundle, serves it with wrangler dev --local, and drives a browser at it. When it goes red, the deploy job is skipped and nothing ships.

The failures looked like this. Four specs would go down together — CSP headers, front-door cache, agent terminal, search — all with ECONNREFUSED. Four unrelated parts of the site, broken at once, in a suite that had passed an hour earlier on the same commit.

That shape is worth learning to recognize, because it is not four defects. It is one. When everything downstream of a server fails at the same instant, you do not have four bugs; you have a server that stopped answering. Every minute I spent looking at the CSP test was a minute spent on a symptom.

What follows is the part I got wrong for three weeks, and the thing that turned out to be true.

The wrong answer, written down and committed

I had a diagnosis early. Buried in the run logs was this:

kj/async-io-unix.c++:263: disconnected: ::write(...): Broken pipe

That is workerd, the runtime under wrangler dev, complaining about a write to a socket the client had already closed. It appeared in failing runs. It looked exactly like a crash. I believed it, and on 19 July I committed a supervisor script to restart the server around it, in c376ab9, with a message that stated the cause plainly: “workerd intermittently dies mid-run on a client disconnect.”

The supervisor worked. The failures continued.

The problem with a wrong diagnosis that ships with a partial fix is that it stops looking like a hypothesis. It becomes the thing everybody knows. It went into a commit message, then into comments in three files, and every time I came back to the flake I started from it.

Here is what finally killed it. I stopped reading the logs for the error I expected and started counting:

Run	Broken pipe lines	Server exits	Result
CI, 3 Aug	2	0	passed
CI, 4 Aug	1	8	failed
46 local runs	0	12	4 failed

A passing run had two Broken pipe lines. A failing run had one. Forty-six instrumented runs on my own machine produced twelve failures and not a single Broken pipe line. In one run, the first two exits both happened before the first Broken pipe line appeared.

The two events were unrelated. Broken pipe is noise that a dev server emits when a browser closes a tab, and it appears in healthy runs. I had spent three weeks on a coincidence.

The number that did correlate was the one I had not been counting: every run where the server process exited failed, and every run where it did not, passed. Restart count was not predictive either — one CI run restarted the server eight times and still went green. What mattered was whether the process was gone at the moment a test needed it.

An error with no message

So why had I not noticed the server exiting? Because of how it announced itself:

✘ [ERROR]

That is the whole line. No message, no stack, no code. wrangler dev prints an empty error and the Node process exits 1. In a CI log with thousands of lines of build output, a bare glyph is invisible, and it is not something you would ever think to grep for.

The stack existed, but only in wrangler’s own debug log, which nothing was capturing. The fix was two lines in the workflow: set WRANGLER_LOG: debug on the prod-smoke job, and upload the log directory as an artifact whether the job passes or fails. Do that before you theorize. I had been reasoning about a process whose last words I had never read.

With the log in hand the sequence was legible, and it was not a crash at all.

What was happening

wrangler dev runs your code in workerd, but it puts a small Worker in front of it called the ProxyWorker, which forwards each request to the UserWorker — your actual code — and pipes the response back.

When Playwright tears down a browser context, it closes sockets. If a response is still being written when that happens, the ProxyWorker’s fetch to the UserWorker rejects with a disconnect: Network connection lost. The ProxyWorker reports that to the controller, where it arrives as an error named Error inside ProxyWorker.

Wrangler keeps an allowlist of error strings that are known to be harmless during development. There are two on it: Failed to send message to… and Could not connect to InspectorProxyWorker. Error inside ProxyWorker is not one of them. So it falls through to emit('error'), and with no handler attached to that event, Node does what Node does.

Note what is not happening here. workerd does not die. It survives the whole thing, which is why it goes on holding port 4321 after its parent is gone — and why the naive cleanup step I will get to below made everything worse. The thing that quits is wrangler’s own Node process. There is no crash to catch and no flag that makes this non-fatal.

The one-line change that caused it

None of this was new code on my side. The suite broke because I upgraded wrangler from 4.110 to 4.114 in d63dea6, in a routine batch of dependency updates, and deploys started failing three days later.

The relevant code is in the ProxyWorker’s error handler. It has to decide whether an error it just caught is still worth reporting, because by the time a network error surfaces, the UserWorker may already have been replaced by a hot reload — in which case the error is stale and should be dropped. In 4.113, that check is one line:

// only report errors if the downstream proxy has NOT changed
if (userWorkerUrl.href === newUserWorkerUrl?.href) {

Now look at what those two URLs are. userWorkerUrl is built from the incoming request, so it carries that request’s path:

const userWorkerUrl = new URL(request.url);
Object.assign(userWorkerUrl, proxyData.userWorkerUrl);

newUserWorkerUrl is rebuilt from the connection parts alone, so its path is /.

For any request to a real page, those two hrefs are never equal. http://localhost:4321/blog/some-post does not equal http://localhost:4321/. The comparison is false, the error is judged stale, and it is silently dropped. On every path except the root, this check swallowed the error by accident.

In 4.114 the comparison became origin-based, and wrangler’s own source says why:

isSameUserWorkerOrigin compares origin (not href) so a genuine error on a non-root path isn’t misread as a reload

That is a correct fix for a real bug. Origins now match on every path, so the error is reported instead of dropped — and because Error inside ProxyWorker was never on the non-fatal allowlist, “reported” means the dev server exits.

My suite had been depending on a bug. When upstream fixed it, a dropped connection on any page went from silently ignored to fatal. Both of those behaviors are wrong; I had just been on the comfortable side of the trade.

It was never one bad test

While the diagnosis was wrong I kept trying to find the test responsible. There wasn’t one.

I suspected my heaviest spec, which loads about 1.3 MB of Mermaid chunks and has one case that aborts requests mid-flight. Running it alone: zero failures in eight attempts. The agent-terminal spec alone: zero in five. The full suite: about half.

Then I stopped bisecting and measured the actual trigger — a page torn down while responses were in flight:

• A page with no subresources survived 80 teardowns.
• A normal page died within 6 to 22.
• Letting the page settle before teardown cut the rate by roughly 6×.

It is dose-response. Every teardown with something in flight is a roll of the dice, and a browser test suite does that hundreds of times. That also explains why concurrency looked like a lever and wasn’t: running with --workers=1 failed at the same rate over eight runs. Serializing the tests does not reduce the number of teardowns, only their overlap.

If you are hunting a flake and single-spec runs come back clean, consider that you may be looking for a rate rather than a culprit.

A real bug that fixed nothing

One detour is worth publishing because the code looked obviously correct.

The supervisor needed to free the port before restarting, since workerd outlives its parent. It did this:

lsof -ti:4321 | xargs kill -9

lsof -i:PORT matches a socket if either endpoint uses that port — local or foreign. A browser connected to the dev server has 4321 as its remote port, so it matches. That command was killing Chromium’s connections during every restart. The correct filter is -sTCP:LISTEN, which returns only the listener; I confirmed it with a listener and one client, where the bare form returns both PIDs and the filtered form returns one.

The same mistake was in the CI cleanup step. It was a genuine bug, I was right to fix it, and it changed nothing: four of six runs still failed afterward. Fixing a real bug you found while looking for a different one is not evidence you found the right one — which is the same error as the Broken pipe, wearing better clothes.

The fix, and why it is three things

The fix is a version pin:

"wrangler": "4.113.0"

Exact, not a caret. A caret floats straight back to 4.114 on the next install and undoes everything.

The pin alone is not enough, and this is the part that would have cost someone else an afternoon. Two more entries are required:

• @cloudflare/vite-plugin held at ~1.46.0. The plugin pins miniflare and workerd exactly in its own dependencies, and 1.46 is the line whose pins match wrangler 4.113. Let it float to 1.47 and the tree resolves two copies of miniflare and two of workerd. The tell is a pnpm warning naming two workerd versions at install time.
• sharp overridden to ^0.35.3. Plugin 1.46.0 carries sharp 0.34.5, which has a high-severity advisory. I had raised the audit gate from critical to high a few days earlier, which is the only reason this surfaced before it shipped rather than after.

Measured across full runs of the suite:

wrangler	server exits	outcome
4.114.0	7 in 6 runs	4 red
4.113.0	0 in 14 runs	12 green

The one non-green run under 4.113 was an unrelated timeout, with zero exits.

A pin is a debt

Pinning to an older version to dodge an upstream fix is not a good place to end up. It is a deliberate decision to stay on the far side of a change that was, on its own terms, correct.

So the pin carries its reasoning where the next person will hit it — in the Dependabot ignore entry, alongside the measurements and an explicit condition for removing it: not “when a newer version exists,” but when upstream restores non-fatal handling for a client disconnect. A newer version number is not evidence. 4.118 was still reported failing, and it also pins a miniflare 5 alpha.

The whole-suite retry stays in CI as a backstop, because a pin is a bet and I would rather not re-learn this from a red deploy. On the first green run after the pin landed, the retry never fired — and the run before it had restarted the server four times and still passed.

Three things I would keep from this:

When everything fails at once, suspect the thing underneath all of it. Four specs failing together was never four problems.

An error you can explain is not the same as the error that is happening. Broken pipe had a plausible story, appeared in the right logs, and was a coincidence. The number that mattered was one I was not counting until I stopped assuming.

Capture the logs before you theorize. The actual cause printed a bare ✘ [ERROR] with no message, and its stack existed the whole time in a debug log nothing was uploading. Three weeks of guessing, and the evidence was one workflow line away.