---
title: "The Test Wasn't Flaky. The Server Was Quitting."
description: "A prod-smoke suite failed for weeks and the error I spent that time chasing was never the cause. The real one was an upgrade that made a dropped connection fatal."
date: 2026-08-06T00:00:00.000Z
tags: ["cloudflare", "wrangler", "testing", "debugging", "ci"]
author: "Cameron Rye"
canonical_url: https://rye.dev/blog/the-test-wasnt-flaky-the-server-was-quitting/
---
For most of July, the last gate before production on this site failed about four times in ten. It is a Playwright suite I call the prod smoke: it builds the real Worker bundle, serves it with `wrangler dev --local`, and drives a browser at it. When it goes red, the deploy job is skipped and nothing ships.

The failures looked like this. Four specs would go down together — CSP headers, front-door cache, agent terminal, search — all with `ECONNREFUSED`. Four unrelated parts of the site, broken at once, in a suite that had passed an hour earlier on the same commit.

That shape is worth learning to recognize, because it is not four defects. It is one. When everything downstream of a server fails at the same instant, you do not have four bugs; you have a server that stopped answering. Every minute I spent looking at the CSP test was a minute spent on a symptom.

What follows is the part I got wrong for three weeks, and the thing that turned out to be true.

## The wrong answer, written down and committed

I had a diagnosis early. Buried in the run logs was this:

```
kj/async-io-unix.c++:263: disconnected: ::write(...): Broken pipe
```

That is workerd, the runtime under `wrangler dev`, complaining about a write to a socket the client had already closed. It appeared in failing runs. It looked exactly like a crash. I believed it, and on 19 July I committed a supervisor script to restart the server around it, in `c376ab9`, with a message that stated the cause plainly: *"workerd intermittently dies mid-run on a client disconnect."*

The supervisor worked. The failures continued.

The problem with a wrong diagnosis that ships with a partial fix is that it stops looking like a hypothesis. It becomes the thing everybody knows. It went into a commit message, then into comments in three files, and every time I came back to the flake I started from it.

Here is what finally killed it. I stopped reading the logs for the error I expected and started counting:

| Run | Broken pipe lines | Server exits | Result |
|---|---|---|---|
| CI, 3 Aug | 2 | 0 | **passed** |
| CI, 4 Aug | 1 | 8 | failed |
| 46 local runs | **0** | 12 | 4 failed |

A passing run had two Broken pipe lines. A failing run had one. Forty-six instrumented runs on my own machine produced twelve failures and not a single Broken pipe line. In one run, the first two exits both happened *before* the first Broken pipe line appeared.

The two events were unrelated. Broken pipe is noise that a dev server emits when a browser closes a tab, and it appears in healthy runs. I had spent three weeks on a coincidence.

The number that did correlate was the one I had not been counting: **every run where the server process exited failed, and every run where it did not, passed.** Restart count was not predictive either — one CI run restarted the server eight times and still went green. What mattered was whether the process was gone at the moment a test needed it.

![A monitor showing lines of code, where one line is a blank glowing white bar with no text, examined under a magnifying glass.](/images/blog/generated/the-test-wasnt-flaky-the-server-was-quitting-illustrates-the-concept-of-an--1786026627037.jpg)

## An error with no message

So why had I not noticed the server exiting? Because of how it announced itself:

```
✘ [ERROR]
```

That is the whole line. No message, no stack, no code. `wrangler dev` prints an empty error and the Node process exits 1. In a CI log with thousands of lines of build output, a bare glyph is invisible, and it is not something you would ever think to grep for.

The stack existed, but only in wrangler's own debug log, which nothing was capturing. The fix was two lines in the workflow: set `WRANGLER_LOG: debug` on the prod-smoke job, and upload the log directory as an artifact whether the job passes or fails. Do that before you theorize. I had been reasoning about a process whose last words I had never read.

With the log in hand the sequence was legible, and it was not a crash at all.

![Two isometric server towers labelled ProxyWorker and UserWorker, joined by a pipe that has snapped in the middle, marked with a red lightning bolt and a broken-link icon.](/images/blog/generated/the-test-wasnt-flaky-the-server-was-quitting-depicts-the-architectural-setu-1786026647497.jpg)

## What was happening

`wrangler dev` runs your code in workerd, but it puts a small Worker in front of it called the ProxyWorker, which forwards each request to the UserWorker — your actual code — and pipes the response back.

When Playwright tears down a browser context, it closes sockets. If a response is still being written when that happens, the ProxyWorker's `fetch` to the UserWorker rejects with a disconnect: `Network connection lost.` The ProxyWorker reports that to the controller, where it arrives as an error named **`Error inside ProxyWorker`**.

Wrangler keeps an allowlist of error strings that are known to be harmless during development. There are two on it: `Failed to send message to…` and `Could not connect to InspectorProxyWorker`. `Error inside ProxyWorker` is not one of them. So it falls through to `emit('error')`, and with no handler attached to that event, Node does what Node does.

Note what is *not* happening here. workerd does not die. It survives the whole thing, which is why it goes on holding port 4321 after its parent is gone — and why the naive cleanup step I will get to below made everything worse. The thing that quits is wrangler's own Node process. There is no crash to catch and no flag that makes this non-fatal.

## The one-line change that caused it

None of this was new code on my side. The suite broke because I upgraded wrangler from 4.110 to 4.114 in `d63dea6`, in a routine batch of dependency updates, and deploys started failing three days later.

The relevant code is in the ProxyWorker's error handler. It has to decide whether an error it just caught is still worth reporting, because by the time a network error surfaces, the UserWorker may already have been replaced by a hot reload — in which case the error is stale and should be dropped. In 4.113, that check is one line:

```ts
// only report errors if the downstream proxy has NOT changed
if (userWorkerUrl.href === newUserWorkerUrl?.href) {
```

Now look at what those two URLs are. `userWorkerUrl` is built from the incoming request, so it carries that request's path:

```ts
const userWorkerUrl = new URL(request.url);
Object.assign(userWorkerUrl, proxyData.userWorkerUrl);
```

`newUserWorkerUrl` is rebuilt from the connection parts alone, so its path is `/`.

For any request to a real page, those two hrefs are never equal. `http://localhost:4321/blog/some-post` does not equal `http://localhost:4321/`. The comparison is false, the error is judged stale, and it is silently dropped. **On every path except the root, this check swallowed the error by accident.**

In 4.114 the comparison became origin-based, and wrangler's own source says why:

> `isSameUserWorkerOrigin` compares origin (not href) so a genuine error on a non-root path isn't misread as a reload

That is a correct fix for a real bug. Origins now match on every path, so the error is reported instead of dropped — and because `Error inside ProxyWorker` was never on the non-fatal allowlist, "reported" means the dev server exits.

My suite had been depending on a bug. When upstream fixed it, a dropped connection on any page went from silently ignored to fatal. Both of those behaviors are wrong; I had just been on the comfortable side of the trade.

## It was never one bad test

While the diagnosis was wrong I kept trying to find the test responsible. There wasn't one.

I suspected my heaviest spec, which loads about 1.3 MB of Mermaid chunks and has one case that aborts requests mid-flight. Running it alone: zero failures in eight attempts. The agent-terminal spec alone: zero in five. The full suite: about half.

Then I stopped bisecting and measured the actual trigger — a page torn down while responses were in flight:

- A page with no subresources survived **80** teardowns.
- A normal page died within **6 to 22**.
- Letting the page settle before teardown cut the rate by roughly **6×**.

It is dose-response. Every teardown with something in flight is a roll of the dice, and a browser test suite does that hundreds of times. That also explains why concurrency looked like a lever and wasn't: running with `--workers=1` failed at the same rate over eight runs. Serializing the tests does not reduce the number of teardowns, only their overlap.

If you are hunting a flake and single-spec runs come back clean, consider that you may be looking for a rate rather than a culprit.

## A real bug that fixed nothing

One detour is worth publishing because the code looked obviously correct.

The supervisor needed to free the port before restarting, since workerd outlives its parent. It did this:

```bash
lsof -ti:4321 | xargs kill -9
```

`lsof -i:PORT` matches a socket if **either** endpoint uses that port — local or foreign. A browser connected to the dev server has 4321 as its *remote* port, so it matches. That command was killing Chromium's connections during every restart. The correct filter is `-sTCP:LISTEN`, which returns only the listener; I confirmed it with a listener and one client, where the bare form returns both PIDs and the filtered form returns one.

The same mistake was in the CI cleanup step. It was a genuine bug, I was right to fix it, and it changed nothing: four of six runs still failed afterward. Fixing a real bug you found while looking for a different one is not evidence you found the right one — which is the same error as the Broken pipe, wearing better clothes.

![Three stacked slabs labelled Version Z, Version Y and Version X. The top two are red and breaking apart; the bottom one is intact and held in place by a pin.](/images/blog/generated/the-test-wasnt-flaky-the-server-was-quitting-illustrates-the-concept-of-pin-1786026667492.jpg)

## The fix, and why it is three things

The fix is a version pin:

```json
"wrangler": "4.113.0"
```

Exact, not a caret. A caret floats straight back to 4.114 on the next install and undoes everything.

The pin alone is not enough, and this is the part that would have cost someone else an afternoon. Two more entries are required:

- **`@cloudflare/vite-plugin` held at `~1.46.0`.** The plugin pins `miniflare` and `workerd` exactly in its own dependencies, and 1.46 is the line whose pins match wrangler 4.113. Let it float to 1.47 and the tree resolves two copies of miniflare and two of workerd. The tell is a pnpm warning naming two workerd versions at install time.
- **`sharp` overridden to `^0.35.3`.** Plugin 1.46.0 carries sharp 0.34.5, which has a high-severity advisory. I had raised the audit gate from `critical` to `high` a few days earlier, which is the only reason this surfaced before it shipped rather than after.

Measured across full runs of the suite:

| wrangler | server exits | outcome |
|---|---|---|
| 4.114.0 | 7 in 6 runs | 4 red |
| 4.113.0 | **0 in 14 runs** | 12 green |

The one non-green run under 4.113 was an unrelated timeout, with zero exits.

## A pin is a debt

Pinning to an older version to dodge an upstream fix is not a good place to end up. It is a deliberate decision to stay on the far side of a change that was, on its own terms, correct.

So the pin carries its reasoning where the next person will hit it — in the Dependabot ignore entry, alongside the measurements and an explicit condition for removing it: not "when a newer version exists," but when upstream restores non-fatal handling for a client disconnect. A newer version number is not evidence. 4.118 was still reported failing, and it also pins a miniflare 5 alpha.

The whole-suite retry stays in CI as a backstop, because a pin is a bet and I would rather not re-learn this from a red deploy. On the first green run after the pin landed, the retry never fired — and the run before it had restarted the server four times and still passed.

Three things I would keep from this:

**When everything fails at once, suspect the thing underneath all of it.** Four specs failing together was never four problems.

**An error you can explain is not the same as the error that is happening.** Broken pipe had a plausible story, appeared in the right logs, and was a coincidence. The number that mattered was one I was not counting until I stopped assuming.

**Capture the logs before you theorize.** The actual cause printed a bare `✘ [ERROR]` with no message, and its stack existed the whole time in a debug log nothing was uploading. Three weeks of guessing, and the evidence was one workflow line away.