Operational evidence

A control that works and a control you can demonstrate works are different deliverables. This page is about the second one: what the platform treats as evidence, why an exit code is not evidence, why ambiguity is a first-class outcome rather than a rounding error, and where the evidence is currently missing.

It is the page most likely to make the rest of this documentation credible or not, so it carries the honest status rather than deferring it.

The distinction

Implementing a control and demonstrating that it works in the deployed system are separate claims requiring separate evidence.

The distinction is routinely collapsed, and understandably so. A team writes a row-level-security migration, reviews it, watches the deployment succeed, and records the control as complete. Every step in that chain feels like verification. But every step is an assertion about intent — the policy is correct, the migration applied, the pipeline reported success.

The question a probe answers is different in kind. Not is the policy correct but does this database, right now, under this role, refuse this specific read.

The failure that made this concrete

Tenant isolation is verified by a one-shot job: connect as the least-privilege runtime role, attempt cross-tenant reads and writes that must fail, attempt in-tenant operations that must succeed, emit a structured verdict.

The first implementation reported success. It also proved nothing.

The job completed and the platform reported a successful exit. But the executing pod had been garbage-collected before its result envelope could be captured. What survived was an exit status — evidence that a process ended, not evidence of what it concluded. A passing exit code from a probe whose output is gone is indistinguishable from a probe that did nothing at all.

This is a more interesting failure than a control that is simply broken, because every dashboard showed green. The verification pipeline was working exactly as designed. The design was wrong.

What counts as durable evidence

Four properties, each of which the original probe lacked.

It outlives the process that produced it. Evidence held only in the memory, filesystem, or log buffer of an ephemeral execution resource is evidence with a lifetime shorter than the question it answers. Capture happens while the executing resource still exists, not afterwards from whatever remains.

It is bound to the run that produced it. Adjudication refuses to report a pass unless it can point at evidence originating from that specific execution. A pass inherited from an earlier run is not a pass. This is the property most often missing in practice, because a result that looks right rarely prompts anyone to ask which run produced it.

It does not depend on an interface nobody promised. A tempting shortcut was available: recover the verdict from platform log aggregation by correlating on an execution-name field. That would have been considerably less work. It was rejected because the field is undocumented — and undocumented fields get renamed, because nothing promised otherwise. A verification whose correctness rests on a vendor implementation detail is a verification with an expiry date nobody has written down, and it does not fail loudly when the date arrives.

It is safe to retain. A verification job runs with database access in an environment whose logs may be aggregated, retained, and read by people with different authorisation than the database itself. So the probe emits a single whitelisted result line — a status, the check results, a pass count — and no tenant identifiers, no SQL, and no connection detail. This is a deliberate reduction in debuggability, paid for the guarantee that the evidence artefact cannot itself become a disclosure channel attached to the very control it verifies.

Ambiguity is an outcome, not a rounding error

The probe emits four verdicts, not two:

Verdict	Meaning
Pass	Every check passed, and the result is bound to this run.
Proven violation	A check proved a violation.
Operational failure	The run broke before a verdict could be reached.
Ambiguous	Evidence is absent, unreadable, or cannot be attributed to this run.

Two rules govern how they combine. A proven violation outranks an unknown. And an unknown never yields a pass.

The explicit ambiguous verdict — a distinct outcome, not a pass — exists because both alternatives are wrong. Rounding missing evidence to success is exactly how the original failure occurred — the run exited zero, so it was treated as a pass. Rounding it to failure would be technically safe and practically useless: operators would learn within a week that the red signal usually means the capture path hiccupped, and would stop reading it. A signal that is ignored is not a control.

There are two distinct paths into ambiguity — evidence that never existed, and evidence that exists but cannot be tied to this run — and they share one correct conclusion: we do not know. The original design had no such state. It had an exit code and an assumption.

What the probe actually checks

The checks are the claims. A probe that tested less would prove less, so they are listed rather than summarised.

It connects only as the least-privilege runtime role, never an administrative one, and everything runs inside transactions that are unconditionally rolled back. Nothing it does is committed.

1. Preflight — is this the role we think it is? The current user is the runtime role; it is not a superuser; it does not hold the attribute that bypasses row-level security; it cannot create roles or databases; it does not replicate; it does not inherit privileges; it has no inherited role memberships; it owns zero relations and zero schemas; it holds no over-broad structural grants. The target table exists, has row-level security enabled, and is not owned by the runtime role. This phase exists because every later result is meaningless if the connection is privileged — a "pass" from a superuser connection proves only that superusers can do things.
2. Own-tenant operations must succeed. Insert, select, and update each affect exactly one row under the tenant's own context. A probe that tested only denial would pass against a database where nothing works at all.
3. Cross-tenant operations must fail. The other tenant's row is invisible; a cross-tenant insert is denied with the insufficient-privilege SQLSTATE; a cross-tenant update and delete each affect zero rows.
4. Empty context must deny. With no tenant context set, reads return zero rows and writes are refused.
5. Structural operations must be denied. Creating and altering a table are both refused. Identifiers used here are internally generated and validated against an allowlist, so the probe cannot become an injection vector itself.
6. Post-rollback state must be clean. Context did not leak across transaction boundaries; an unscoped full-table read returns zero rows; an unscoped insert is denied; the synthetic row did not survive the rollback.

Note the asymmetry in the last phase. The zero-row unscoped read is treated as corroborating, not as proof — an empty result could also mean an empty table. The denied unscoped insert is the content-independent guarantee, because it holds regardless of what the table contains.

Fail-closed CI guards

The same standard applies to infrastructure. Terraform plan review catches what the reviewer thinks to look for and reliably misses the boring invariants nobody re-reads on the fortieth change.

Those invariants are therefore unit tests that parse the infrastructure variable files directly and assert:

• a database password is never a plain environment variable, only a vault reference
• no key appears in both the environment map and the secret map, which would emit duplicate entries
• a development identifier never appears in production configuration
• the public URL and the authentication callback URL are byte-identical, since a mismatch breaks sign-in only at runtime

They fail closed: a missing or malformed value trips the suite rather than passing quietly. Several were written after the corresponding incident, which is the honest origin of most good guards.

A second guard runs over everything written to be published — the documentation site, the whitepaper, marketing copy — matching shapes rather than a list of known values: identifier formats, cloud service hostnames, resource identifier paths, the internal naming convention, credential shapes, private repository references, CI run identifiers. A deny-list of real values would itself be a catalogue of what to protect, and could only catch what someone remembered to add. A shape catches the ones nobody thought of.

That guard also asserts its own liveness: every rule must still match a synthetic fixture of what it is meant to catch, and its allowlist is asserted so that widening it fails a test rather than being a one-line fix under time pressure. A guard that silently stops matching is worse than no guard, because it produces confidence without coverage — which is the probe's failure in different clothing.

The same trap, one level down

Test suites are validated by mutation testing: deliberately break the code and confirm the suite notices. The metric is the kill rate.

The trap is why a mutant died. A mutant killed by the assertion targeting it is evidence. A mutant killed by a test timeout, a teardown failure, or residue from a previous mutant is not. Both increment the same counter.

Counting the second kind as proof conceals precisely the coverage gaps the campaign was run to expose, and yields a confident number that means nothing. Worse, a mutant whose blast radius exceeds its targeting test can contaminate subsequent mutants in the same run, so one bad result quietly corrupts the ones after it.

The discipline that came out of it: record the reason for each kill, treat accidental kills as survivals, and serialise database work so one suite's stray object cannot fail another's global scan. The parallel to the probe is exact — in both cases a green signal was being produced by something other than the thing it claimed to measure.

Current status

Implemented. The isolation control itself: policies on the tenant-scoped tables, specifying both read and write constraints on the platform tables, forced so that table ownership does not exempt, on a least-privilege runtime role that owns nothing, with tenant context bound to the transaction. The redesigned probe, including the four-verdict adjudication described above. The fail-closed infrastructure and publication guards.

Verification in progress. The redesigned probe has not yet produced a clean end-to-end pass bound to durable evidence from its own run. The first run returned an explicit ambiguous verdict — a distinct outcome, not a pass — after the one-shot execution environment was reclaimed before its result envelope could be captured, and the redesign has not yet produced a clean passing run.

So the honest status of tenant isolation is implemented; end-to-end proof in progress — not proven, and not verified. Any material from this project claiming verified tenant isolation contradicts this page, and this page is the one that is right.

Not yet done. Making the proof continuous. A single passing run proves a moment; the control's value is continuous, so the evidence should be too. The probe should run on a schedule and on every change to a tenant-scoped table or policy, with an ambiguous verdict treated as a build failure rather than a warning. A control verified once is a control verified before the last forty changes.

The same standard, applied to two claims this page does not make. Both are stated here because this is the page about evidence, and evidence that has not been produced is the thing most easily left unmentioned.

The anomaly detectors run as a pipeline stage and raise events. Their accuracy has never been measured: there is no labelled evaluation set, so precision and recall are unknown. A detector with unmeasured accuracy is a hypothesis, and the honest description of a hypothesis is "detection runs and its error profile is uncharacterised" — not "anomalies are detected".

The quarantine reason code is stored as free text with no constraint at the database boundary; the vocabulary is enforced only by the application's type system. The type system is not present where the data lives, so a future writer could store an unrecognised code without anything failing. That is a real gap between what the schema guarantees and what the documentation implies, and constraining it is outstanding work.

Also worth stating: only a development environment exists. Staging and production are defined in code and not deployed, so nothing on this page is a claim about production behaviour, uptime, or scale.

Stating all of that is the whole point. A page arguing for the difference between a control that works and evidence that a control works, which then quietly claimed the evidence, would be a worked example of its own thesis.

Related reading: tenant isolation for the control this verifies, and engineering notes for the other failures that shaped the design.