Proving tenant isolation is harder than implementing it

A note on the code below: every snippet in this post is illustrative. It is written to show the mechanism as plainly as possible, not copied out of the production codebase. The failure, the diagnosis and the redesign are real; the exact lines are a reconstruction.

The tenant-isolation probe ran in the development environment. It connected to the database as the application's runtime role, attempted a handful of reads and writes that were supposed to fail, attempted a handful that were supposed to succeed, and exited zero. The platform recorded a successful execution. Every dashboard was green.

It proved nothing.

The probe ran as a one-shot job on a managed container platform. By the time anything downstream went looking for the structured verdict the probe had written, the executing pod had been garbage-collected. What remained was an exit status — a fact about a process, not a fact about the assertions that process made. A zero exit from a probe whose output is gone is indistinguishable from a probe that connected, did nothing, and returned.

That is a more interesting failure than a broken control, and a much easier one to ship, because nothing in the pipeline looked wrong. The verification machinery worked exactly as designed. The design was wrong.

The easy half

The control itself was the straightforward part, and it is worth describing briefly because it makes the rest of the story sharper.

The conventional way to isolate tenants is to scope every query in application code:

-- illustrative; the real table and column names are internal
SELECT * FROM <tenant_scoped_table>
 WHERE <tenant-id-column> = $1
   AND status = 'quarantined';

The correctness of the entire tenant boundary now depends on that first predicate being written by every engineer, in every query, in every future change, forever. The failure characteristics are unusually hostile: a missing predicate returns more rows rather than an error, single-tenant test fixtures pass either way, any file can introduce it, and the blast radius is cross-tenant exposure.

Moving the boundary into PostgreSQL changes the category rather than the probability. Illustratively — every name below is a placeholder, because the real table, column, policy, and tenant-context setting names are internal:

ALTER TABLE <tenant_scoped_table> ENABLE ROW LEVEL SECURITY;
ALTER TABLE <tenant_scoped_table> FORCE ROW LEVEL SECURITY;

CREATE POLICY <policy-name> ON <tenant_scoped_table>
  USING (<tenant-id-column> = current_setting('<tenant-context-setting>', true)::uuid);

A query that forgets its tenant predicate now returns the rows the current tenant context permits, not everything. The application can still have bugs; it cannot have this bug.

Two supporting decisions carry most of the weight, and both are routinely skipped.

The runtime role must not be able to exempt itself. Row-level security is bypassed by table owners and by superusers. An application connecting as the schema owner has written a policy that does not apply to it. So the application connects as a role that owns nothing: it holds the specific data-manipulation grants it needs on the tenant-scoped tables, and deliberately not ownership of any table or schema, not the attribute that exempts a role from row-level security, and not superuser. Its name is internal, and the example does not need one — the shape of the grant is the point.

Tenant context is bound to the transaction, not the connection. Connection pools reuse connections. Connection-scoped context leaks one request's tenant into the next request that happens to get the same socket:

BEGIN;
SELECT set_config('<tenant-context-setting>', $1, true);  -- true: transaction-local
-- ... unit of work ...
COMMIT;

That is the whole control. A migration, a role, and a discipline about where context is set. Reviewable in an afternoon.

Why a code review is not evidence

Here is the collapse that happens next, and it happens almost everywhere: the migration lands, two people review it, the control is marked complete, and the control inventory now says "tenant isolation: enforced".

What actually exists at that point is a design intention and two opinions. Not a demonstration that the deployed database, reached by the deployed application's role, with the deployed pooling behaviour, rejects a cross-tenant read.

The gap between those two things is where the interesting failures live. Policies get created on some tables and not others. A later migration adds a table and nobody enables the policy on it. A connection-pool library is swapped and the new one resets session state at a different point. Someone grants the runtime role a convenience privilege during an incident and never revokes it. Each of those is invisible to a review of the original migration and visible to a probe that runs against the deployed system.

So: a probe. Connect as the runtime role, attempt cross-tenant reads and writes that must fail, attempt in-tenant operations that must succeed, confirm the role cannot disable the policy protecting it, and emit a structured verdict.

Conceptually — this is a sketch of the idea, not the probe's wire format, and none of these keys or values is the real one:

{
  "probe": "tenant-isolation",
  "runId": "<the identifier of this execution>",
  "checks": [
    { "name": "cross-tenant read returns nothing",   "outcomeCategory": "pass" },
    { "name": "cross-tenant write is refused",       "outcomeCategory": "pass" },
    { "name": "in-tenant work succeeds",             "outcomeCategory": "pass" },
    { "name": "role cannot disable its own policy",  "outcomeCategory": "pass" }
  ],
  "outcomeCategory": "pass"
}

Correct probe. Wrong lifecycle.

The lifecycle bug

One-shot jobs on managed container platforms are ephemeral by design — that is the feature you are paying for. The executing resource is reclaimed once the process ends, on a schedule the platform owns and does not promise to you.

The original design read the verdict after the job reported completion. Those two events are in a race, and the platform wins it more often than not. When the read lost, the pipeline had a completed execution, an exit status of zero, and no envelope. And it reported success, because the only signal it still had said the process ended cleanly.

Rounding "I have no evidence" up to "pass" is the single decision that turned a verification into theatre. It is also completely natural: the alternative — a step that goes red when nothing is wrong with the system under test — feels like a broken pipeline rather than an honest one.

The reframing that fixed it is small and generalises well beyond this system:

An exit status tells you a process ended. It does not tell you what the process concluded. If your verification's output is the exit status, you have verified the scheduler, not the control.

The redesign

Three changes, in the order they matter.

Capture before reclamation. The verdict is captured while the executing resource still exists, rather than read afterwards from whatever survived. The evidence is written to durable storage as a step in the run, not harvested from the corpse.

Bind the verdict to its own run. Adjudication refuses to report a pass unless it can point at evidence originating from that specific execution. This closes a failure mode that is worse than the original one: a stale envelope from an earlier, healthier run sitting in the same location, silently adjudicated as today's pass. In pseudocode:

evidence = load_evidence_for(runId)

if evidence is missing:              -> ambiguous
if evidence.runId != runId:          -> ambiguous   # not from this run
if evidence.outcomeCategory is pass: -> pass
otherwise:                           -> fail

Make ambiguity a first-class outcome. An explicit ambiguous verdict — a distinct outcome, not a pass — is neither pass nor fail. Rounding missing evidence to success is how the original failure happened. Rounding it to failure would have been almost as bad in a different way — a verification that goes red for infrastructural reasons trains operators to ignore it, and an ignored control is an absent control with extra steps. A third outcome says the true thing: the control was not shown to be broken, and it was not shown to be working; go and find out why.

The shortcut I turned down

There was an obvious cheaper fix available. The platform's log aggregation retains job output well past the pod's lifetime. Correlate on the execution-name field, pull the verdict line out of the logs, done — no capture step, no storage, no new moving parts.

I rejected it because that field is undocumented. It is present, it is populated, it is exactly what you would want, and the vendor has never promised it exists or described what it contains. A verification whose correctness rests on a vendor implementation detail is a verification with an expiry date nobody has written down. It will keep passing right up until a platform release renames a field, and it will not fail loudly when it does — it will fail into the same "no evidence, assume pass" hole the redesign was built to close.

Depending on undocumented behaviour is a reasonable trade in plenty of places. It is a bad trade specifically in the thing whose entire job is to be trustworthy when everything else is not.

Where this actually stands

The control is implemented. The verification has been redesigned. It has not yet produced a clean end-to-end pass bound to durable evidence from its own run.

The honest status is implemented; end-to-end proof in progress — not proven.

Stating that plainly is not a caveat bolted onto the end of the post; it is the post. An article arguing for the distinction between a control that works and evidence that a control works, which then quietly claimed the evidence, would be committing the exact error it describes. The temptation to write "and now it passes" is strong, which is precisely the pressure that produced the original green dashboard.

What I would do differently

Design the evidence path before the assertions. I wrote a good probe and gave no thought to how its conclusion would outlive it. The assertions took an afternoon; the evidence lifecycle took several iterations. That ratio is the lesson.

Treat "no evidence" as its own state from the first line of code. Two-valued verdicts push ambiguity into whichever branch the author was less afraid of. Almost nobody is afraid enough of a false pass.

Ask what each signal is a fact about. Exit status is a fact about a process. A green pipeline step is a fact about the pipeline. A log line is a fact about what was printed. None of them is a fact about the database policy, and only one artefact in the whole chain was.

Write down what the control does not cover. Row-level security addresses application queries that omit a tenant predicate. It does not address a tenant context set from untrusted input, escalation to an exempting role, inference through counts and timing, or anything living outside the database — object storage, caches, message payloads, logs. A control with an understood boundary can be layered. A control believed to be total cannot.

Assume the shortcut is a trap when it is cheap and undocumented. Especially when the thing you are shortcutting is the evidence.

The uncomfortable general conclusion: a team can hold a perfectly accurate inventory of implemented controls and an entirely inaccurate picture of which ones are demonstrably working. Distinguishing the two is cheap to do deliberately and expensive to discover after an incident.

---

Related reading

• Tenant isolation — the control, the probe failure, and current status
• Architecture — where the database boundary sits in the platform
• Engineering notes — other things that were harder than expected
• Whitepaper — the long-form treatment, including what the control does not defend against