Engineering notes

Things that were harder, subtler, or more embarrassing than expected. Written up because the failures are more instructive than the successes.

The environment variable that did nothing

The public site URL was configured as a runtime container environment variable and set correctly in infrastructure code. The deployed page served a different, stale hostname for months.

Three things combined:

1. The framework inlines NEXT_PUBLIC_* variables at build time, replacing them in the bundle. They are not read at runtime.
2. The container image was built without the variable defined, so the hardcoded fallback was compiled in.
3. The page was statically prerendered, freezing the resulting metadata into the image.

Every layer was individually reasonable. Together they produced configuration that looked authoritative, deployed cleanly, and had no effect — with the infrastructure code confidently asserting a value the running site contradicted.

It was found by comparing the deployed HTML against the deployed configuration and noticing they disagreed. The general lesson: verify configuration at the layer that consumes it, not the layer that declares it. A variable set in a deployment manifest proves the manifest, not the behaviour.

The fix on the replacement site was to stop pretending the value was dynamic. It is a build-time constant in one file, with a test asserting the built output carries it.

A mutant killed by a timeout is not a passing test

Mutation testing deliberately breaks the code and checks that the suite notices. The metric is the kill rate.

The trap is why a mutant died. A mutant can be killed by:

• the assertion that targets it — real evidence
• a test timeout, a teardown failure, or leftover state from a previous mutant — no evidence at all

Both increment the same counter. Counting the second kind as proof hides precisely the coverage gaps the campaign was run to find, and produces a confident number that means nothing.

Worse, a mutant whose blast radius exceeds the test targeting it can contaminate later mutants in the same run — so the residue of one bad result quietly corrupts the ones after it. Hardening the individual test is not enough; the shared harness has to clean up, or the next mutant inherits the mess.

The discipline that came out of it: record the reason for each kill, treat accidental kills as survivals, and serialise database work so one suite's stray object cannot fail another's global scan.

Fail-closed infrastructure guards

Terraform plan review catches what the reviewer thinks to look for. It reliably misses the boring invariants nobody re-reads on the fortieth change.

So the invariants became unit tests that parse the variable files directly:

• a database password is never a plain environment variable, only a vault reference
• no key appears in both the environment map and the secret map — the module emits one entry per key per map, so an overlap silently produces duplicates
• a development identifier never appears in production configuration
• the public URL and the auth callback URL are byte-identical, since a mismatch breaks sign-in in a way that only shows up at runtime

They fail closed: a missing or malformed value trips the suite rather than passing quietly. Several were written after the corresponding incident, which is the honest origin of most good guards.

Choosing the less impressive platform

An earlier design targeted Kubernetes. It was replaced with Azure Container Apps.

Kubernetes would have been more impressive on a resume and strictly worse for this system: it added a control plane to operate, an ingress stack to maintain, and a large surface of configuration for workloads that are four container types with straightforward scaling requirements.

Container Apps covered the requirement — scale to zero, HTTP autoscaling, managed certificates, workload identity, internal-only services — with materially less to run.

The interesting cost was not technical. Existing architecture decision records still referenced the Kubernetes design, and stale ADRs are worse than no ADRs, because they are read as current. Superseding a decision is part of making it.