A mutation killed by a timeout is not a killed mutant

A note on the code below: every snippet in this post is illustrative. It is written to show the mechanism as plainly as possible, not copied out of the production codebase. The findings and the reasoning are real; the exact lines are a reconstruction.

A mutation campaign on a critical module reported a high kill rate. Almost every mutant died. The number was excellent and it was worth nothing, because a meaningful share of those mutants had not been caught by the assertions written to catch them. They had been caught by a timeout, by a teardown that blew up, or by wreckage the previous mutant left in the database.

All of those increment the same counter. The report says "killed" either way.

This is the failure mode that makes mutation testing dangerous rather than merely expensive: it is one of the few practices that can hand you a confident, quantitative, and entirely false picture of your test suite. A low coverage number tells you where you are weak. A high kill rate built from accidental kills tells you that you are strong exactly where you are not, and it hides the gaps the campaign was run to find.

What the number is supposed to mean

Mutation testing deliberately breaks your code — flips a comparison, deletes a branch, changes a constant, removes a call — and re-runs the suite. If the suite fails, the mutant is "killed". If it passes, the mutant "survived", and you have a behaviour change no test noticed.

The implied claim behind a kill is precise, and it is stronger than people usually read it as:

There exists a test that specifically distinguishes the original behaviour from the mutated behaviour, and it does so by evaluating an assertion about that behaviour.

Almost everything useful about mutation testing lives in that sentence. "The suite went red" is a much weaker claim, and it is the one the tooling actually measures.

The taxonomy that matters

Split every kill into two piles.

Legitimate. The assertion that targets the mutated behaviour evaluated, and it failed, and it failed for the reason you intended. The evidence you wanted is the failure message: expected 500, received 250 names the behaviour and the discrepancy.

Accidental. The suite went red for a reason unrelated to the assertion:

• Timeout. The mutant removed a release, a resolve, or a break condition. A test hung. The runner killed it. No assertion about the mutated behaviour was ever evaluated — the process died before reaching it.
• Teardown failure. The test body passed. Cleanup threw, because the mutant broke the thing cleanup depends on. Recorded as a failing test.
• Crash or setup error. The mutant made a module fail to initialise, so hundreds of unrelated tests failed. That is not coverage of the mutated line; it is coverage of import.
• Residue. The mutant left an object, a lock, or an open transaction behind, and a later test — often in a different file — failed because of it.
• An unrelated test. Something red for an incidental reason: a snapshot, a log-line assertion, an ordering dependency. It went red near the mutant, not because of the specific behaviour change.

The reason a legitimate kill is evidence and an accidental one is not comes down to counterfactuals. A legitimate kill tells you: if this behaviour regresses in production code, a named assertion will catch it and tell you what changed. An accidental kill tells you: if this behaviour regresses, something might go red, in a way that names an unrelated file and reports a timeout.

That second outcome is not a test. It is a rumour.

A concrete one

A cleanup path in a database helper released a pooled client and rolled back a savepoint. A mutant deleted the release.

Reported: killed. What actually happened: the pool exhausted its connections a few tests later, an unrelated suite hung waiting for a client, and the runner killed the file after the timeout. The test written specifically to assert that the client is released never executed its assertion — the process was gone before it ran.

Rewriting that single test so it asserted directly and quickly on release state, independent of pool exhaustion, turned the same mutant from "killed after the full timeout window" into "killed in under six seconds by its own assertion". Same counter, entirely different meaning. Only the second version tells you the release is tested.

The tell is almost always duration. A kill that takes the timeout limit is suspicious by construction; a kill that takes as long as the test normally takes is usually real. Recording the duration and the failing assertion's identity for every mutant costs a few lines in the harness and converts the whole exercise from a number into evidence.

Corollary one: blast radius bigger than the test

Some mutants do not stay inside the test that targets them.

A mutant that leaves a stray temporary table, an open transaction, or an unreleased advisory lock is still doing damage after its own run finishes. In a campaign that runs mutants sequentially against a shared database, that residue lands on the next mutant. Its result is now a measurement of the previous mutant's mess.

This is genuinely corrosive because it is silent and directional: the contamination flows forward, so the later results in a run are less trustworthy than the earlier ones, and nothing in the report distinguishes them. It also means the ordering of your mutant list changes your kill rate — which is a straightforward reductio, since mutation order carries no information about test quality.

The instinct on discovering this is to harden the test that the mutant broke. That is not enough, and the reasoning is worth stating carefully:

When a mutant's blast radius exceeds the test that targets it, the fix belongs in the shared harness, not the individual test. Otherwise the next mutant inherits the mess.

Concretely, that means:

• Clean up in the harness, unconditionally. Not in the test body, which the mutant may have prevented from reaching cleanup. A global teardown that drops temporary objects, closes stragglers, and rolls back open transactions runs even when the test died badly.
• Serialise work against shared state. A suite that scans globally for stray objects fails falsely when a concurrent session legitimately holds one. That is a false failure, which is another accidental kill.
• Verify the database is unchanged before and after the campaign. A before/after comparison of the schema and of any global object catalogue turns "I think it cleaned up" into a check.
• Isolate mutants from each other where you can afford to. A fresh schema per mutant is expensive; it is also the only structural fix, and worth it for the modules where the answer really matters.

Corollary two: the test that sampled the wrong point

The opposite error is subtler and I like it more, because the test looks thorough.

Consider a retry delay that is deliberately continuous at its crossover, so delays never jump:

const BASE_MS = 500;
const LINEAR_LIMIT = 4;

export function retryDelayMs(attempt: number): number {
  if (attempt <= LINEAR_LIMIT) return BASE_MS * attempt;   // 500 1000 1500 2000
  return BASE_MS * 2 ** (attempt - 2);                     // 2000 4000 8000 ...
}

At attempt = 4 both branches return 2000. That is by design — the whole point of choosing those constants was to avoid a discontinuity at the switchover.

Now the test. A conscientious engineer tests the boundary, because boundaries are where bugs live:

it('caps linear growth at the crossover', () => {
  expect(retryDelayMs(LINEAR_LIMIT)).toBe(2000);
});

And the mutant: delete the linear branch entirely, so every attempt uses the exponential formula.

The test passes. It passes at the one input where the two branches are indistinguishable, which is precisely the input the engineer chose because it was the interesting one. Attempts 1 through 3 now return 250, 500, and 1000 instead of 500, 1000, and 1500 — the retry schedule is wrong everywhere except the single point under test.

The mutant survives, correctly. The instructive part is the near-miss: had anything else in the suite incidentally gone red, this would have been logged as a kill and the blind spot would have been recorded as coverage. The lesson generalises past mutation testing entirely — a boundary test samples a function at the one input where its branches agree, which makes it the weakest possible discriminator between them. Test the boundary and a point either side of it.

Mutation testing is unusually good at surfacing this, because a surviving mutant forces the question "what input would have distinguished these?" — a question that ordinary line coverage never asks, since the line was executed either way.

The discipline that came out of it

1. Record why each mutant died, not just that it died: the failing assertion's identity, the duration, and whether the failure was in the test body, in setup, or in teardown.
2. Treat accidental kills as survivals. Not as a footnote — as survivals, in the headline number. The number's only job is to be trustworthy.
3. Prove the intended assertion is the one that fires. Before the campaign, confirm the targeting test fails against a hand-made version of the mutant, with the expected message. That is the difference between "my suite catches this" and "my suite catches this for this reason".
4. Harden the harness, not just the test, whenever a mutant escapes the test that targets it.
5. Look for equivalent mutants explicitly. Some survivors cannot be killed because they do not change behaviour. Adjudicating that honestly — with an argument, not a shrug — is part of the work, and it is where the crossover case above hides.

The through-line, and it is the same one as the isolation probe that exited zero and proved nothing: a green or red signal is a fact about a process, not about the property you wanted to establish. The work is in the binding between them, and that binding is where verification quietly turns into theatre.

---

Related reading

• Engineering notes — the original write-up, plus fail-closed infrastructure guards
• Tenant isolation — the same failure shape in a security control
• Whitepaper — mutation testing in the context of the delivery pipeline