Enforcing Tenant Isolation Below the Application

An architecture for governed multi-tenant B2B data integration

Abstract — Multi-tenant platforms conventionally enforce tenant boundaries in application code, scoping every query by tenant identifier. This works until one query omits the predicate, at which point the failure is silent, returns more data rather than an error, and exposes one customer's records to another. This paper describes a platform that relocates the boundary into the database using row-level security bound to a least-privilege runtime role, making that failure mode unrepresentable rather than merely improbable. It then argues that implementing such a control is the easier half of the work: the harder half is producing evidence that survives the process which generated it. We describe a verification failure in which an isolation probe passed and proved nothing, the redesign that fixed it, and the general principle that a control which works and a control you can demonstrate works are different deliverables. Around that central argument the paper documents the surrounding system — intake, governed ingestion, schema inference and contract versioning, quarantine, anomaly detection, the event spine, deployment, and the security boundary — because an isolation claim is only meaningful in the context of what it is isolating.

---

1. Executive summary

IntegraHive is a multi-tenant business-to-business data-integration platform. Trading partners deliver data over REST, managed file transfer, or direct upload. The platform infers the schema each file actually carries, registers and versions that schema as a contract, stages the records, applies governance policy, quarantines what breaks the contract, and raises anomaly events — with tenant isolation enforced by the database rather than by application code.

Four design commitments shape everything that follows.

The tenant boundary belongs below the application. Row-level security policies, evaluated by PostgreSQL on every query and bound to a least-privilege runtime role that cannot bypass them, convert the most damaging bug class in a multi-tenant system from unlikely to unrepresentable. Section 8 develops the argument, including what the control does not cover.

Every intake channel converges on one pipeline. Governance written once cannot be skipped by choosing a different door. Section 4 covers the channels, Section 5 the lifecycle they share.

Nothing is silently dropped. A record that breaches its contract is quarantined with a reason code and diagnostic context, not discarded. A pipeline that discards malformed records optimises its own success metrics while destroying the evidence needed to repair the partner integration. Section 7.

Verification is a deliverable, not a checkbox. The paper's most transferable finding came from a failure: an isolation probe ran, exited successfully, and proved nothing, because the one-shot execution environment was reclaimed before its result envelope could be captured. What survived was an exit status — evidence that a process ended, not evidence of what it concluded. Sections 12 and 15 describe the failure, the redesign, and the fact that the redesign has not yet produced a clean passing run.

That last point sets the tone for the document. A development environment runs the full stack. Staging and production are defined in code and not deployed. Tenant isolation is implemented, and its end-to-end proof is in progress. Those statements appear here, in Section 15, and on every page of the documentation site, because a reader who finds one overclaim stops believing the rest — and because a paper that argues for the difference between a control that works and evidence that a control works, and then quietly claims the evidence, would be committing the exact error it describes.

2. Problem and design goals

2.1 The awkward position of a B2B integration platform

A business-to-business data-integration platform sits in an awkward position. It holds data belonging to many organisations that are frequently competitors, ingests that data through channels those organisations control, and must reconcile the schema each partner actually sends against the schema they agreed to send.

Three requirements follow, and they pull against each other:

Isolation must be absolute. A single cross-tenant leak is not a degraded experience; it is a business-ending event for a platform whose entire value proposition is handling competitors' data.

Ingestion must be tolerant. Partners send malformed files, rename columns without notice, and change types between quarters. A pipeline that rejects everything unexpected is unusable; one that accepts everything silently is worse.

Everything must be reconstructable. When a downstream consumer asks why a number changed, the answer must be recoverable from evidence rather than inferred.

The tension is direct. Tolerance argues for accepting whatever arrives and sorting it out later; reconstructability argues for capturing the exact bytes and the exact rules that applied at the moment of acceptance; isolation argues for a boundary so rigid that no amount of tolerance elsewhere can perforate it. A design that optimises any one of these in isolation degrades the others.

2.2 Design goals

The goals below are stated as testable properties rather than aspirations, because an aspiration cannot fail a build.

G1 — A missing tenant predicate must not be able to return another tenant's rows. Not should not; must not. The mechanism must hold when the application is wrong.

G2 — Capture must be separable from interpretation. The bytes a partner sent must remain byte-exact and reproducible after any downstream failure, so that a parsing bug is a re-run rather than a data-loss incident.

G3 — A schema must be a versioned artefact, not a mutable property. Any historical run must be interpretable against the contract that actually applied to it.

G4 — Rejection must be reviewable. Every record the platform refuses must carry a reason code and enough context to act on, and must remain available for reprocessing.

G5 — Configuration must be verified at the layer that consumes it. A value set in a deployment manifest proves the manifest, not the behaviour.

G6 — Infrastructure invariants must be executable. Anything asserted about the deployed system in prose should also be asserted by something that can fail.

G7 — A control's evidence must outlive the process that produced it, and must be bound to the specific run that produced it.

Sections 5 through 12 describe how each goal is met. Section 15 describes where they are met only partially, and says so.

2.3 What was explicitly not a goal

Naming the non-goals is as useful as naming the goals, because it explains choices that otherwise look like omissions.

The platform is not a general-purpose ETL engine; it is a governed intake boundary. Transformation beyond normalisation and coercion is deliberately downstream. It does not attempt to be schema-agnostic in the "everything is a JSON blob" sense — the entire value of contract versioning depends on committing to a typed shape. And it does not attempt exactly-once end-to-end semantics across the whole lifecycle; it attempts idempotent capture, explicit per-stage status, and re-runnability, which is a weaker guarantee that can actually be delivered. Section 13 explains why that trade is the right one.

3. System context

3.1 Actors and surfaces

Four kinds of actor interact with the platform, and each gets a different surface with a different authentication model.

Trading partners deliver data and inspect the fate of their deliveries. They authenticate machine-to-machine with scoped API keys for ingestion, and through a partner portal for human review. Partner identity is verified server-side and is never taken from a request header.

Operators run the platform for a tenant: reviewing quarantined records, approving or rejecting schema changes, promoting a contract version, investigating anomaly events. They authenticate through enterprise single sign-on.

Downstream consumers read accepted data through a versioned API.

The platform itself runs scheduled pulls, workers, and one-shot jobs under workload identity rather than under any user's credentials.

flowchart LR
  P1["Partner: REST push"] --> EDGE["Ingestion edge"]
  P2["Partner source: scheduled pull"] --> EDGE
  P3["SFTP / object store"] --> EDGE
  P4["Operator: direct upload"] --> EDGE
  EDGE --> PIPE["Governed ingestion pipeline"]
  PIPE --> DB[("Managed PostgreSQL, private network, RLS")]
  PIPE --> SPINE["Event spine"]
  SPINE --> WORK["Workers"]
  SPINE --> MLS["ML service"]
  MLS --> DB
  WORK --> DB
  DB --> API["Versioned API"]
  DB --> CONSOLE["Operator console"]
  DB --> PORTAL["Partner portal"]

Figure 1 — System context. Four intake paths converge on a single governed pipeline. The pipeline writes to managed PostgreSQL on a private network, where row-level security is the tenant boundary, and publishes events onto an internal spine consumed by workers and the machine-learning service. Three read surfaces — a versioned API, an operator console, and a partner portal — are served from the same database and are therefore subject to the same policy. The important structural fact in this figure is that there is no path from an intake channel to a read surface that bypasses the pipeline, and no path to the database that bypasses the policy.

3.2 Trust zones

The system distinguishes three trust zones, and most of the security design is about what may cross between them.

The public zone contains partner clients, browsers, and the internet. The application zone contains the web workload, workers, the machine-learning service, and one-shot jobs; it is reachable from the public zone only through external ingress on the web tier. The data zone contains managed PostgreSQL and the internal event spine; it has no public endpoint at all and is reachable only from the application zone across a private virtual network.

Tenant identity is established at the boundary between the public and application zones, and is carried into the data zone as a transaction-scoped database setting. That single sentence is the load-bearing design of the whole platform, and Section 8 unpacks it.

4. Partner intake channels

4.1 Three doors and an operator override

Partners deliver data three ways, and operators have a fourth path for backfills and remediation.

REST push. Partners send CSV, JSON, or XML to a keyed ingestion endpoint. The request is authenticated with a scoped API key rather than a user session, rate-limited per route, and the payload is checksummed on arrival before anything else happens to it.

REST pull. The platform fetches from a partner-controlled source on a schedule, with configurable authentication, headers, and pagination. This inverts who initiates but not what happens next: a pulled payload enters the same landing zone with the same manifest and the same checksum.

Managed file transfer. Secure transfer between sources and targets over SFTP and object stores. Credentials for these connections are held in a managed vault and referenced by workload identity; they are never configuration values.

Direct upload. Operators can upload for backfills, replay, and remediation. This path is not a bypass — it produces a run exactly like any other, and is subject to identical governance. An intake channel that skipped governance because a human initiated it would be the largest hole in the design.

flowchart TD
  A["REST push (partner-initiated)"] --> Z["Tenant-isolated landing zone"]
  B["REST pull (scheduled)"] --> Z
  C["Managed file transfer (SFTP, object store)"] --> Z
  D["Operator upload (backfill, replay)"] --> Z
  Z --> M["Manifest + SHA-256 checksum"]
  M --> DUP{"Checksum already seen?"}
  DUP -->|yes| DUPE["Run marked duplicate; no processing cost"]
  DUP -->|no| RUN["Run proceeds to inference"]

Figure 2 — Intake channels converge before anything is interpreted. All four paths write into a tenant-isolated landing zone and produce a manifest and a SHA-256 checksum before any parsing occurs. Deduplication happens at this point, against the checksum, so a redelivered file is recognised before processing cost is incurred and is recorded as a duplicate delivery rather than silently ignored. Recording the duplicate rather than discarding it matters: "we received this twice" and "we never received it" are different partner conversations, and only one of them is a platform problem.

4.2 Why convergence is enforced rather than encouraged

Per-channel pipelines drift. The file-transfer path acquires a quirk the REST path lacks; a governance rule is applied in one and forgotten in another; and the divergence surfaces only when two partners with identical contracts produce different results. By that point the difference is load-bearing for someone.

Convergence is therefore structural rather than conventional: the channels are adapters that terminate at the landing zone, and the pipeline has exactly one entry point. A new channel is a new adapter, not a new pipeline, which means the work of adding one is bounded and the governance surface does not grow with it.

The cost is real. A channel with genuinely different semantics — streaming, say, or a protocol with its own transactional guarantees — has to be forced into a file-shaped abstraction or given an adapter that buffers into one. That is a worse fit than a bespoke path would be. It is accepted deliberately, because the alternative trades a known inefficiency for an unbounded and invisible divergence risk.

4.3 Idempotence at the door

Partner deliveries repeat. A retry after a timeout, a scheduler firing twice, a partner re-uploading because they were unsure the first attempt worked — all produce the same bytes arriving more than once.

The checksum recorded at landing makes this cheap to detect and, more importantly, cheap to detect before interpretation. A platform that deduplicates after parsing has already paid the parsing cost and, worse, has already had the opportunity to half-apply the second copy. Deduplicating on the bytes means the second delivery is a status transition and nothing else.

This is idempotent capture, not idempotent processing. A partner who sends a genuinely corrected file with different bytes gets a new run, as they should. Reconciling two runs that overlap in content is a governance question, not a deduplication question, and it is answered by the contract rather than by a hash.

5. Governed ingestion lifecycle

5.1 Seven stages

Every run moves through the same seven stages regardless of how it arrived.

  land ──► infer ──► register ──► stage ──► govern ──► quarantine ──► detect
   │         │          │           │          │            │           │
 capture   read the  version the  parse &   apply       isolate,     surface
 intact    real shape  contract   prepare   policy     never drop     drift

Land. Data is received into a tenant-isolated landing zone. A manifest and SHA-256 checksum are recorded on arrival; duplicates are caught before processing cost is incurred. Nothing is parsed here. Capture is separated from interpretation so that when parsing fails later, the original artefact remains byte-exact and reproducible.

Infer. The landed file is profiled to determine its actual structure — columns, types, nullability, nesting — without assuming the partner sent what was agreed. This is the difference between a pipeline reporting load failed and one reporting a date column arrived as a string, where it was previously a date.

Register. The inferred schema is registered as a version against the dataset's contract. Changes are diffable against prior versions, making schema drift a reviewable event with history rather than an incident discovered downstream.

Stage. Records are parsed and staged, with run status moving from receipt through landing, parsing and loading, and row counts recorded at each transition. A run that halts names the stage it halted in, which is most of the diagnostic work.

Govern. Policy is applied before acceptance: schema-drift handling, field masking by policy rather than convention, and retention rules. Running governance before acceptance rather than as a later sweep ensures non-compliant data never becomes the thing downstream consumers have already read.

Quarantine. Records breaching the contract are quarantined with a reason code and diagnostic context. The design rule is isolate, never drop. Quarantined records remain reviewable, and a corrected contract can reprocess them.

Detect. Statistical and machine-learning detectors raise events for schema drift, value outliers, and threshold breaches. Detection is a pipeline stage rather than an external monitor, so an anomaly is attributable to the run that produced it.

5.2 Run status as a state machine

Run status is not a boolean and not a free-text field. It is a constrained set of six states, enforced by a database check constraint and mirrored as a TypeScript union so that the two cannot drift apart without a type error. The literal tokens are internal; the shape of the machine is the part worth arguing about, and it is drawn below with descriptive labels.

stateDiagram-v2
  [*] --> Received
  Received --> Landed: bytes persisted, manifest and checksum written
  Received --> DuplicateDelivery: checksum already seen for this dataset
  Received --> Failure: capture failed
  Landed --> Parsed: records normalised, quarantine set computed
  Landed --> Failure: unreadable artefact
  Parsed --> Loaded: governance passed, accepted rows committed
  Parsed --> Failure: governance blocked or load failed
  DuplicateDelivery --> [*]
  Loaded --> [*]
  Failure --> [*]

Figure 3 — Run status transitions. A run moves from receipt through landing, parsing and loading, with terminal states for a delivery recognised as a duplicate of one already accepted and for a run that failed. The value of a constrained enumeration over a boolean is that a halted run names the stage it halted in, which converts most incident triage from investigation into lookup. The duplicate state is a terminal success, not a failure: the platform did the right thing. Failure is reachable from three different stages, and the stage it was reached from is the diagnostic content — a run that failed while landing is an infrastructure or artefact problem, while a run that failed while parsing is a contract or data problem, and those go to different people.

5.3 Partial failure is the normal case

Ingestion is not transactional end to end, and pretending otherwise produces systems that fail confusingly. A run may land successfully, parse partially, and fail governance — three outcomes at three stages, each needing a different response.

Run status is therefore explicit per stage rather than a single success flag, and the artefacts of each completed stage survive the failure of a later one. A governance failure does not discard the parsed records; it marks them unaccepted. This makes reprocessing after a contract fix a re-run of the failed stage rather than a full re-ingestion, which matters when the partner file is large and the delivery window has closed.

The general principle is that stage boundaries are commit points for evidence, not for data. Landing commits the bytes and the manifest. Inference commits the observed schema and its statistics. Staging commits the normalised records and the quarantine set. Governance commits the acceptance decision. Each of those survives the failure of everything after it, which is what makes a partial failure investigable rather than merely annoying.

5.4 What "governed" actually means here

The word governance is used loosely across the industry, so it is worth being precise about the claim.

Governance in this platform means three specific things, each evaluated before acceptance rather than after:

1. Contract conformance — does each record satisfy the registered contract's column set, types, and nullability constraints?
2. Change classification — if the observed schema differs from the contract, what class of change is it, and does the dataset's policy permit it to be applied automatically, require review, or block the run?
3. Data handling — which fields are masked by policy, and how long do landed and staged artefacts persist?

What governance is not is a post-hoc audit sweep. The ordering matters more than the rules: a sweep that runs after acceptance can tell you that non-compliant data is in the warehouse, but by then a downstream consumer has read it, and the remediation is a retraction rather than a rejection. Evaluating before acceptance means the worst case is a delayed dataset, not a poisoned one.

6. Schema inference and contract versioning

6.1 Inference is deterministic, not learned

Schema inference is a deterministic computation over the parsed sample, not a model. This is a deliberate choice and it is worth defending, because the platform does run machine-learning workloads elsewhere.

A learned type classifier would be more tolerant of odd inputs and would occasionally be right where a rule-based classifier is wrong. It would also make the schema of a dataset a function of a model version, which means the same file could register two different schemas six months apart, and the diff between two contract versions would be uninterpretable — did the partner change, or did the model? Since the entire purpose of contract versioning is to answer exactly that question, non-determinism at the inference step would defeat the feature it feeds.

Inference produces a canonical type from a small closed vocabulary: string, int, bigint, decimal, boolean, date, timestamp, and json. Each value in a column is classified, and the column type is the reduction of those classifications. Numeric classifications widen along int → bigint → decimal; temporal classifications widen along date → timestamp; a set that mixes families reduces to string, which is the safe absorbing state. The split between int and bigint is decided by whether observed values fit in a 32-bit range.

Alongside the type, each column carries its ordinal position, an inferred nullability derived from observed empty values, a confidence figure computed as the fraction of values whose own classification widens into the column's chosen type, and a set of statistics: null percentage, distinct count, minimum and maximum, and a bounded sample of values.

6.2 The leading-zero guard

One rule in the inference logic deserves individual mention, because it encodes an expensive lesson that most integration platforms learn in production.

Any multi-digit value carrying a leading zero forces its entire column to string and blocks numeric coercion for that column.

The values this protects are identifiers that look like numbers: zero-padded account codes, postal codes, product SKUs, tracking references. Coercing them to an integer type is lossless from the type system's point of view and catastrophic from the business's, because 00123 and 123 are different accounts and the coercion silently merges them. The damage is not detected at ingestion — the numbers are valid — but downstream, when a join produces the wrong customer.

The guard is deliberately aggressive: a single leading-zero value in a column of a million otherwise-clean integers pins the whole column to string. That is the correct asymmetry. The cost of being wrong in the conservative direction is a column typed more loosely than it needed to be. The cost of being wrong in the permissive direction is silent identity collision.

6.3 Fingerprints and version identity

A schema version is identified by a fingerprint: a SHA-256 over the ordered tuples of column name, canonical type, and nullability.

The fingerprint deliberately excludes statistics. Null percentage, distinct count, and value ranges vary between every run of the same dataset; including them would mint a new schema version on each delivery and make version history meaningless. Excluding them means a version is created only when the shape changes, which is exactly the event a reviewer wants to see.

The fingerprint includes ordinal position implicitly, through ordering. Two files with the same columns in a different order produce different fingerprints. This is defensible for positional formats such as CSV, where column order is part of the contract, and is a known source of friction for formats where it is not. Section 15 lists it as a limitation rather than defending it as a feature.

6.4 Observed versus contracted

The register stage is where most integration platforms accumulate their worst technical debt, and the reason is almost always the same: they conflate the schema that arrived with the schema that was agreed.

The naive model treats a dataset's schema as a single mutable property — one current shape, updated in place when it changes. This loses the two facts that matter during an incident: what the shape was when a given run landed, and what changed between then and now.

IntegraHive maintains two distinct markers over an append-only sequence of versions:

• The current version is descriptive. It moves with each run and records what the platform most recently observed.
• The contract version is authoritative. It moves only by explicit promotion, carries an approver and an approval timestamp, and is what governance evaluates against.

Each marker is enforced by a partial unique index, so the database itself guarantees that a dataset has at most one current version and at most one contract version. This is a small detail with a large consequence: the invariant cannot be broken by a race between two concurrent runs, because it is not maintained by application logic.

flowchart TD
  RUN["Ingestion run"] --> INF["Infer observed schema"]
  INF --> FP["Compute fingerprint over name, type, nullability"]
  FP --> EX{"Fingerprint already registered for this dataset?"}
  EX -->|yes| REUSE["Reuse existing version"]
  EX -->|no| NEW["Append new version"]
  REUSE --> CUR["Move 'current' marker"]
  NEW --> CUR
  CUR --> CMP{"Does current equal contract?"}
  CMP -->|yes| OK["No drift; govern against contract"]
  CMP -->|no| DIFF["Compute diff; classify change"]
  DIFF --> POL["Apply dataset drift policy"]
  POL --> PROMO["Promotion moves 'contract' marker; records approver and time"]

Figure 4 — Observed and contracted schema are separate markers over one append-only history. Every run registers what it observed. Only an explicit promotion moves the contract. The consequence is that a partner changing their file cannot change what the platform considers correct — drift becomes a reviewable event with a named approver, rather than a silent redefinition of the agreement. It also makes historical interpretation exact: a run from six months ago references the version in force when it landed, so it can be read against the contract that actually applied to it rather than today's.

6.5 Classifying drift

When the observed schema differs from the contract, the difference is decomposed into typed change kinds — a column added, a column removed, a type changed, or nullability changed — and each carries a reason code that explains why the classifier reached its conclusion rather than merely reporting it.

The reason codes distinguish cases that a coarse "type changed" would flatten into one alarm. A widening that cannot lose information is different from a narrowing that can. A change that alters meaning rather than representation is different from either. A change the classifier reached with low confidence is different again, and so is one blocked by the leading-zero guard. A newly appearing column that looks sensitive is treated differently from one that does not, and a new required column is treated differently from a new nullable one.

Each change is classified into one of three dispositions, and the dataset's policy decides which are acceptable:

• auto — apply without human involvement. Reserved for changes that are provably safe, such as a strictly widening type change or a new nullable column.
• review — hold for a human decision. The run does not proceed to acceptance on the changed shape until an operator approves or rejects.
• block — refuse. The run fails at governance and the contract stands.

flowchart TD
  D["Observed differs from contract"] --> K{"Change kind"}
  K -->|column added, nullable| SAFE["Provably safe"]
  K -->|column added, required| RISK["Requires decision"]
  K -->|column removed| RISK
  K -->|type widening, lossless| SAFE
  K -->|type narrowing, lossy| RISK
  K -->|semantic type change| RISK
  K -->|leading-zero guard tripped| RISK
  K -->|low classifier confidence| RISK
  SAFE --> P{"Dataset drift policy"}
  RISK --> P
  P -->|auto| APPLY["Applied; resolution recorded as auto-applied"]
  P -->|review| PARK["Parked for operator; approved or rejected"]
  P -->|block| STOP["Run fails at governance; contract unchanged"]

Figure 5 — Drift classification and policy are separate steps. The classifier answers what changed and how risky is it; the per-dataset policy answers what should happen about changes of that risk. Keeping them separate means a tenant can operate one dataset permissively and another strictly without two classifiers, and means the classification recorded against a historical run remains meaningful even if the policy is later changed. Every outcome is recorded with a resolution — applied automatically, parked, approved, or rejected — so the question "who allowed this change?" always has an answer.

6.6 The cost of this model

Versioning everything costs storage and a moderately more complex read path. Every run must resolve which version applied to it rather than reading a single current shape, and every diff is a computation rather than a lookup.

In exchange, questions about historical data have answers rather than reconstructions. Drift becomes a diff between two named versions, so "the partner changed something" is answerable precisely instead of approximately. Drift policy becomes meaningful at all — blocking on drift requires knowing that drift occurred, which requires a prior version to compare against. And the promotion record makes schema change an accountable act rather than an emergent property of whatever arrived most recently.

7. Quarantine and anomaly detection

7.1 Isolate, never drop

A record that cannot be normalised against the contract is quarantined. It is written to a quarantine store with a reason code, the source run, and enough diagnostic context to act on. It is never discarded.

The rule exists because the alternative is quietly corrupting. A pipeline that drops malformed records reports a high success rate, produces datasets that look complete, and destroys precisely the evidence needed to repair the partner integration that is generating the malformed records. The metric improves as the problem worsens.

Two categories of reason are currently emitted at the record level. The literal codes are internal; the distinction they draw is the publishable part:

• A structural failure — the row's field count does not match the header, or the header itself is ambiguous or contains duplicate names. The row cannot be interpreted as a record at all.
• A semantic failure — a column the contract declares as required is missing or cannot be coerced to its declared type. The row is a record, but not a valid one.

Within the semantic category, the specific violation is carried in the diagnostic detail as either a missing required value or a failed type coercion, so an operator sees which field and why rather than only this row is bad.

flowchart TD
  R["Parsed row"] --> S{"Structurally interpretable?"}
  S -->|no| Q1["Quarantine: structural failure"]
  S -->|yes| C{"Required fields present and coercible?"}
  C -->|no| Q2["Quarantine: semantic failure"]
  C -->|yes| A["Accepted into staged set"]
  Q1 --> STORE["Quarantine store: reason code, run id, diagnostic context"]
  Q2 --> STORE
  STORE --> REV["Operator review"]
  REV --> FIX["Contract corrected or partner notified"]
  FIX --> RE["Reprocess quarantined records against corrected contract"]
  RE --> A

Figure 6 — Quarantine is a holding state, not a bin. A quarantined record retains its reason code, its originating run, and its diagnostic context, which makes two things possible that a drop makes impossible: an operator can see the shape of the problem across many records rather than one, and a corrected contract can reprocess the held records rather than requiring the partner to resend. The loop back into the accepted set is the entire point of the design — quarantine is a stage in the record's life, not the end of it.

7.2 Quarantine rate is itself a signal

Because quarantined records are retained and attributed to their run, the quarantine rate becomes a measurable property of the integration rather than an invisible loss.

That measurement feeds detection. A dataset whose quarantine rate jumps between runs is telling you something changed on the partner side, usually before anyone has filed a ticket about it. A pipeline that dropped those records would show the same accepted row count and no signal at all.

7.3 Anomaly detection as a pipeline stage

Detection runs as the final stage of the pipeline rather than as an external monitor reading the results afterwards. The difference is attribution: an anomaly raised inside the run is bound to the run that produced it, with the run's schema version, row counts, and quarantine set available as context. An external monitor observing a metric dip has to reconstruct which run caused it.

Five detector categories are implemented. Their internal identifiers are not published; what each is for is:

• Delivery volume — the delivery is materially larger or smaller than the dataset's recent history.
• Field completeness — a column's empty rate departs from its established distribution, which typically means an upstream field stopped being populated.
• Normalisation-failure rate — the proportion of records failing normalisation moved.
• Inference confidence — inference is less certain about a column's type than it has been, which is often the first visible symptom of a format change.
• Repetition within a delivery — repeated records inside one file, distinct from whole-file duplication caught at landing.

Each event carries a severity drawn from an ordered scale, which is what allows an operator to triage a backlog rather than read it. Severity is a property of the event, not of the detector, so the same detector can raise a minor drift and a serious one.

7.4 What detection deliberately does not do

Detection raises events. It does not block runs, and it does not quarantine records. That separation is intentional.

Governance is deterministic and contractual: a record either satisfies the contract or it does not, and the consequence is defined in advance. Detection is statistical and contextual: a row-count drop of forty per cent is alarming for one dataset and a normal Monday for another. Wiring a statistical signal into a blocking decision means a distribution shift can halt a pipeline, and the operational response to that is invariably to raise the threshold until it stops happening — which is to say, to disable it.

Keeping detection advisory keeps it honest. It also means the signal remains useful when it is wrong, because a false positive costs an operator a glance rather than a partner a delivery window.

8. Multi-tenant isolation

8.1 Why application-level filtering fails

The conventional approach scopes queries in application code:

-- illustrative; the real table and column names are internal
SELECT * FROM <tenant_scoped_table>
 WHERE <tenant-id-column> = $1
   AND status = 'quarantined';

The correctness of the entire tenant boundary now rests on every query, written by every engineer, in every future change, remembering that first predicate.

The failure characteristics are unusually bad:

Property	Consequence
Silent	A missing predicate returns more rows, not an error
Invisible in tests	Single-tenant fixtures pass either way
Uniformly distributed	Any query in any file can introduce it
Maximum blast radius	Cross-tenant exposure

Consider each in turn, because the combination is what makes this bug class distinctive rather than merely common.

Silent means there is no error path to log, alert on, or catch. The query succeeds. The application renders the result. Nothing anywhere reports a problem.

Invisible in tests means the standard defence does not apply. A test fixture with one tenant's data produces identical results with and without the predicate, so a test suite of any size can pass while the bug is present. Only a fixture that deliberately contains a second tenant's rows — and an assertion that they are absent — detects it, and that is not the default way anyone writes tests.

Uniformly distributed means the risk does not concentrate anywhere you could guard. It is not confined to a data-access layer or a set of sensitive endpoints; it is present in every query anyone writes, including the ad-hoc one added under time pressure to fix something else.

Maximum blast radius means the consequence is the worst outcome the system can produce.

Mitigations — code review, query helpers, repository abstractions — reduce probability. None changes the category. The boundary remains a convention that application code is trusted to observe, and trust is not an enforcement mechanism.

8.2 Relocating the boundary

IntegraHive enforces isolation with PostgreSQL row-level security. A policy on each tenant-scoped table is evaluated by the database on every query, comparing the row's tenant against a context value established for the current unit of work.

The policy is applied FOR ALL commands and specifies both a USING clause, which constrains which existing rows are visible, and a WITH CHECK clause, which constrains which rows may be written. Specifying both matters: a USING-only policy prevents a tenant from reading another tenant's rows but, depending on the command, may not prevent them from writing a row attributed to another tenant. Read isolation without write isolation is half a boundary.

An application query that omits its tenant predicate now returns the rows the current tenant context permits — not everything. The application may still have bugs; it cannot have this bug.

sequenceDiagram
  participant C as Client request
  participant M as Middleware
  participant A as Application handler
  participant P as Connection pool
  participant D as PostgreSQL
  C->>M: request with session or API key
  M->>M: strip client-supplied identity headers
  M->>M: authenticate; derive tenant from verified session or token claim
  M->>A: request with server-established tenant context
  A->>P: acquire pooled client
  A->>D: BEGIN
  A->>D: set_config(tenant setting, tenant id, local = true)
  A->>D: SELECT ... (no tenant predicate required)
  D->>D: evaluate RLS policy against transaction-local setting
  D-->>A: rows permitted for this tenant only
  A->>D: COMMIT or ROLLBACK
  Note over D,P: transaction-local setting is discarded with the transaction

Figure 7 — Tenant context is established server-side and bound to the transaction. Three properties are load-bearing. First, tenant identity is derived from a verified session or token claim, never from a request header — the middleware strips client-supplied identity headers before any handler can see them. Second, the context is set as a transaction-local setting with a bound parameter, so it cannot be injected through string construction and cannot outlive its transaction. Third, the policy is evaluated by the database, so a query with no tenant predicate at all still returns only permitted rows. The ROLLBACK path is as important as the COMMIT path: a failed transaction discards the setting exactly as a successful one does.

8.3 Three decisions that carry the weight

The policy alone is insufficient. Three supporting decisions determine whether it actually holds.

A least-privilege runtime role. Row-level security is bypassed by table owners and by superusers. A platform whose application connects as the schema owner has written a policy that does not apply to itself. Enforcement therefore requires a dedicated runtime role that is explicitly not a superuser, does not have the attribute that bypasses row-level security, cannot create roles or databases, does not inherit privileges from other roles, and holds only the specific data-manipulation grants it needs rather than blanket privileges. Critically, it owns no tables and no schemas, because ownership would exempt it from the very policies it is subject to. Migrations run under a separate administrative role; the runtime role has no path to alter the structures that constrain it. This is the single most commonly skipped step, and skipping it produces a system that appears isolated under inspection and is not.

Tenant context bound to the transaction. Connection pools reuse connections. If tenant context were connection-scoped, a pooled connection could carry one request's tenant into the next — a leak that would be intermittent, load-dependent, and close to impossible to reproduce. Binding the context to the transaction removes that class of leak structurally: the setting is written with a transaction-local flag, so it is discarded on both commit and rollback, and it is written with a bound parameter rather than interpolated into a statement.

Forced row-level security. PostgreSQL exempts a table's owner from its policies by default, which means a policy is silently inert for exactly the connection most likely to be used for maintenance or a hurried fix. The tenant-scoped tables are therefore configured to force the policy, so it applies to the owner as well. The list of tables that have policies and the list that force them are derived from one shared definition rather than maintained separately, because two lists that must agree will eventually not.

8.4 Request-scoped propagation

Tenant identity established at the request boundary has to reach the database call, which may be several layers down a call stack and inside an asynchronous continuation. Threading it through every function signature is possible but fragile: any function that forgets to pass it becomes a place where a caller can accidentally supply the wrong value.

The platform instead propagates tenant identity through request-scoped asynchronous storage, so the data-access layer reads the current request's tenant rather than accepting it as an argument. The trade-off is a form of implicit context, which is generally worth being suspicious of. It is accepted here because the failure modes are asymmetric: an explicit parameter that can be passed wrongly fails silently with the wrong tenant's data, whereas request- scoped storage that is missing fails loudly with no tenant at all — and the database policy then returns nothing rather than everything.

8.5 What this does not defend against

Claiming a control without stating its boundary is how controls get trusted beyond their reach. Row-level security addresses one specific failure: an application query that omits or mis-scopes its tenant predicate. It does not address the following, each of which needs its own mechanism.

A compromised or mis-set tenant context. The policy compares against a context value. Code that sets that value from an untrusted input has moved the vulnerability rather than removed it. Tenant identity is therefore derived from the authenticated session or a validated token claim, never from a request header or parameter — and the middleware strips client-supplied identity headers before any handler sees them, precisely because a forged header would otherwise be a boundary bypass.

Privilege escalation to an exempting role. The policy does not apply to superusers, and applies to owners only because it is explicitly forced. Its guarantee is only as strong as the discipline that keeps the application off those roles, which is why the runtime role is treated as part of the control rather than as deployment detail.

Platform-level operations that must cross tenants. Some operations are legitimately cross-tenant: migrations, platform-wide reporting, and maintenance. Privileged cross-tenant administrative operations exist as a separate, process-controlled path outside the least-privilege application runtime — they do not travel through the runtime role the policy constrains, and how that path is invoked is deliberately not described here. It is an acknowledged escape hatch rather than a hidden one, and it is a process control rather than an enforced one. Section 15 lists it as such.

Aggregate and inference leakage. A query returning only permitted rows can still leak through counts, timing, or error messages that vary by whether an out-of-tenant record exists. Row-level security constrains rows returned, not information disclosed.

Anything outside the database. Object storage, log aggregation, cached responses, and message payloads are not covered by a database policy. Each needs its own tenant scoping; the landing zone is tenant-isolated by path and credential rather than by row policy, because it is not rows.

Stating these explicitly is the point. A control with an understood boundary can be layered; a control believed to be total cannot.

9. Authentication and workload identity

9.1 Three authentication models for three populations

The platform authenticates three distinct populations, and using one mechanism for all three would mean the weakest requirement setting the design.

Operators authenticate through enterprise single sign-on over OIDC, with SAML supported for organisations that require it, and hold a session for the console. This population needs interactive login, group-derived authorisation, and session revocation.

Partner systems authenticate with scoped API keys at the ingestion edge. This population is machine-to-machine, cannot perform an interactive flow, and needs credentials that can be rotated per integration rather than per person.

Partner users authenticate into the partner portal and are represented downstream by verified identity headers that the application sets after verification. The middleware unconditionally strips the client-supplied forms of those headers from every incoming request before anything else runs, so a partner cannot present the identity that downstream code trusts.

9.2 The request chain

Request handling is an ordered chain, and the order encodes the security model.

flowchart TD
  IN["Incoming request"] --> FLAG["Portal feature gate"]
  FLAG --> STRIP["Strip client-supplied identity headers"]
  STRIP --> RID["Request id: echo if well-formed, else regenerate"]
  RID --> PJWT["Verify partner token; set verified identity headers"]
  PJWT --> SVC["Internal service-key gate for service-to-service routes"]
  SVC --> CSRF["CSRF double-submit check on state-changing API calls"]
  CSRF --> RATE["Rate limit; emit limit, remaining, reset, retry-after"]
  RATE --> KEY["API-key gate for partner ingestion routes"]
  KEY --> SESS["Session check for console routes"]
  SESS --> IP["Per-tenant IP allowlist evaluated against CIDR rules"]
  IP --> CORS["CORS allowlist"]
  CORS --> H["Route handler with server-established tenant context"]

Figure 8 — The request chain, in execution order. Two orderings are deliberate. Identity headers are stripped before anything reads a header, so no later step can be tricked by a value the client supplied. And the CSRF check runs before the rate limiter, so a flood of forged state-changing requests is rejected on the cheaper check first. There is an explicit rule in the implementation that the presence of an API key does not by itself exempt a request from CSRF — an exemption that looks harmless and would allow a browser session to be leveraged against an API-key-protected route.

9.3 Rate limiting in layers

Rate limiting is applied at more than one layer, which is a response to the fact that each layer is individually insufficient.

An in-process counter is cheap and per-instance, which means it does not coordinate: an attacker spread across instances gets the limit multiplied by the instance count. A shared store coordinates correctly but costs a network round trip and introduces a dependency whose failure must not become the platform's failure. An edge limit is cheapest of all and coarsest.

Layering them means the cheap check sheds the majority of load before the expensive one is consulted, and the expensive one provides the correctness the cheap one cannot. Responses carry standard limit, remaining, and reset headers, and a retry-after on rejection, so a well-behaved partner client can back off rather than guess.

9.4 Secrets and workload identity

No credential is committed. Secrets live in a cloud-managed vault and are referenced by workload identity, so a running container acquires them from its own identity rather than from injected configuration.

The important structural consequence is the split between the connection descriptor and the credential. The database connection string carries no password: it is non-secret configuration that can live in version-controlled infrastructure code. The password is a separate reference resolved from the vault at runtime. This means the thing engineers need to read and review is reviewable, and the thing that must never be read is never in a position to be.

CI asserts this rather than trusting it. Guard tests parse the infrastructure variable files directly and fail the build if a credentialed connection string, a plaintext secret, or a versioned vault reference appears where it should not. Section 11 covers those guards in more detail; the point here is that "we do not commit secrets" is an assertion with a test behind it rather than a statement of intent.

10. Event-driven processing

10.1 Why an event spine at all

Ingestion is synchronous only up to the point where the run's evidence is committed. Everything after that — parsing large payloads, delivery routing, acknowledgement generation, model training, drift detection — is work whose latency should not be borne by the partner's HTTP request.

The event spine is NATS with JetStream persistence, deployed as an internal-only workload with no public ingress. It is reachable on the private network and from nowhere else.

10.2 Subjects and streams

Subjects follow a consistent three-part convention: domain, entity, and a past-tense event name — <domain>.<entity>.<past-tense-event>. A document arriving, a flow run completing, drift being detected, an escalation being triggered: each is one subject under its domain. The literal subject names are internal and are not published here; the convention is the part worth arguing about.

The past tense is not stylistic. A subject named for something that has happened describes a fact, and facts have exactly one correct interpretation. A subject named for something that should happen describes a command, and commands invite the publisher to know what the consumer should do — which is how an event bus degrades into a distributed function call with worse error handling.

Four streams capture the four domains — business-to-business document flow, flow execution, machine learning, and system events — each subscribing to its domain's subject wildcard. Each stream has its own retention and age limits, and the document stream additionally carries a deduplication window.

flowchart LR
  subgraph PUB["Publishers"]
    ING["Ingestion pipeline"]
    FLOW["Flow engine"]
    MLP["ML service"]
  end
  subgraph SPINE["Event spine (internal only)"]
    S1["document-flow stream"]
    S2["flow-execution stream"]
    S3["machine-learning stream"]
    S4["system-events stream"]
  end
  subgraph CONS["Durable consumers"]
    C1["document parser"]
    C2["composite trigger"]
    C3["delivery router"]
    C4["acknowledgement generator"]
    C5["flow monitor"]
  end
  ING --> S1
  FLOW --> S2
  MLP --> S3
  ING --> S4
  S1 --> C1
  S1 --> C2
  S1 --> C3
  S1 --> C4
  S2 --> C5
  C1 --> DLQ["Dead-letter store after delivery attempts exhausted"]
  C3 --> DLQ

Figure 9 — Streams, durable consumers, and the dead-letter path. Consumers are durable and use explicit acknowledgement with a bounded delivery attempt count and an acknowledgement wait window, so a consumer that crashes mid-work causes redelivery rather than loss. When attempts are exhausted the message goes to a database-backed dead-letter store rather than being dropped — the same isolate, never drop rule that governs quarantined records, applied to messages. A dead-letter store that is a database table rather than a queue is a deliberate choice: it can be queried, joined against the run that produced it, and reprocessed by an operator.

10.3 Contracts on events

Event payloads are validated against schemas rather than trusted. Each event carries an identifier, a schema version, its subject, a correlation identifier, and a metadata envelope.

The correlation identifier is what makes the asynchronous half of the system investigable. A partner asking why their delivery has not appeared downstream produces a question that spans an HTTP request, a run record, several events, and a worker execution. Without a correlation identifier that chain is reconstructed by timestamp proximity, which is guesswork. With one it is a query.

The per-event schema version is deliberately separate from the dataset schema registry described in Section 6. They version different things — the platform's own message contracts versus the partner's data contracts — and conflating them would make an internal refactor look like a partner-facing change.

Publishing uses a message identifier for deduplication at the stream level, so a publisher that retries after an ambiguous failure does not produce two events. This is the same reasoning as checksum deduplication at landing, applied one layer in: the cheapest place to handle a duplicate is before anyone acts on it.

10.4 Workers

Workers are a single process specialised by configuration rather than a set of separately-built images. One binary, selected at startup into a role — document parsing, delivery, acknowledgement, retry, flow execution, or detection.

The benefit is that a worker cannot drift from the platform's shared libraries, because it is the platform's code. The cost is that a worker image contains code for roles it will never execute, which is a modest size and attack-surface penalty. For a system with this number of roles the trade favours consistency; at an order of magnitude more roles it would not.

11. Deployment architecture

11.1 Runtime topology

The platform runs on Azure Container Apps across four workload types:

Workload	Responsibility
Web	Operator console, partner portal, API
Workers	Long-running queue consumers
ML service	Inference, profiling, anomaly detection
Jobs	One-shot tasks — migrations, probes, backfills

External ingress fronts the web tier with HTTP-concurrency autoscaling and startup, liveness, and readiness probes against a health endpoint. Container images are built remotely by the registry's build service rather than on CI runners, removing the need for a Docker daemon or a privileged runner in the delivery path.

flowchart TD
  NET["Internet"] --> ING["External ingress, managed certificate"]
  ING --> WEB["Web workload: console, portal, API"]
  subgraph APPZONE["Application zone"]
    WEB
    WRK["Worker workload"]
    MLS["ML service"]
    JOB["One-shot jobs: migrations, probes, backfills"]
  end
  subgraph DATAZONE["Data zone, private network, no public endpoint"]
    PG[("Managed PostgreSQL")]
    NATS["Event spine, internal ingress only"]
  end
  VAULT["Cloud-managed vault"]
  WEB --> PG
  WRK --> PG
  MLS --> PG
  JOB --> PG
  WEB --> NATS
  WRK --> NATS
  MLS --> NATS
  WEB -. workload identity .-> VAULT
  WRK -. workload identity .-> VAULT
  JOB -. workload identity .-> VAULT

Figure 10 — Deployment topology and trust zones. Only the web workload is reachable from the internet, and only through managed ingress. The data zone has no public endpoint: PostgreSQL is on a private virtual network and the event spine has internal ingress only. Secrets are never delivered as configuration; each workload resolves them from the vault using its own identity, shown as dashed edges because they are a control-plane relationship rather than a data path. One-shot jobs sit inside the application zone with the same database access as any other workload, which is what makes it possible to run an isolation probe under exactly the runtime role the application uses.

11.2 Health probes and what they actually assert

The health endpoint serves three distinct modes, because conflating them produces bad failure behaviour.

Liveness reports whether the process is alive, performs no dependency I/O, and always answers successfully if it answers at all. This is deliberate: a liveness probe that checks the database restarts the application when the database is briefly unavailable, which is precisely the wrong response and turns a dependency blip into a restart storm.

Readiness checks dependencies concurrently under a short timeout and reports one of three categories — healthy, impaired, or unavailable. The literal tokens are internal. The database is always treated as critical. Other dependencies use a consecutive-failure threshold before being treated as down, which prevents a single transient error from removing an instance from rotation.

The full report, with longer timeouts, covers the primary database, the read path, the secret store, object storage, event-spine connectivity, connection pool statistics, and job-queue depth including work stuck in a processing state beyond a threshold. This mode is for operators, not for the platform's own scheduling decisions.

All three modes are served with caching explicitly disabled, because a cached health response is a health response about the past.

11.3 Infrastructure as tested code

Infrastructure is Terraform, split per layer with isolated remote state so one layer's failure cannot corrupt another's. CI authenticates by OIDC federation — no long-lived cloud credentials exist in the CI provider — and applies are gated behind explicit approval rather than firing on merge.

The less conventional decision is how correctness is asserted. Plan review catches what the reviewer thinks to check and reliably misses the boring invariants nobody re-reads on the fortieth change. Those invariants are therefore unit tests that parse the infrastructure variable files directly:

• a database password is never a plain environment variable, only a vault reference
• no key appears in both the environment map and the secret map, which would emit duplicate entries
• a development identifier never appears in production configuration
• the public URL and the authentication callback URL are byte-identical, since a mismatch breaks sign-in only at runtime

These guards fail closed: a missing or malformed value trips the suite rather than passing quietly. Several were written after the corresponding incident, which is the honest origin of most good guards.

11.4 Configuration resolves where it is consumed

One deployment lesson generalises far beyond this platform and is worth stating as a principle: verify configuration at the layer that consumes it, not the layer that declares it.

The concrete failure was a public site URL configured as a runtime container environment variable and set correctly in infrastructure code, while the deployed page served a stale hostname. Three individually reasonable behaviours combined: the framework inlines that class of variable at build time rather than reading it at runtime; the image was built without the variable defined, so a fallback was compiled in; and the page was statically prerendered, freezing the result into the artefact.

Every layer was correct about its own scope. The infrastructure code accurately described a variable that was genuinely set. The variable was genuinely irrelevant. It was found by comparing the deployed HTML against the deployed configuration and noticing they disagreed — which is the only method that works, because every layer's self-report was accurate.

The remediation was not a better variable. It was to stop pretending the value was dynamic: it is now a build-time constant in one file, with a test asserting the built output carries it. A configuration mechanism that cannot be verified at the consuming layer should be replaced by one that can.

12. Observability and evidence

12.1 The distinction

Section 8 describes a control. Whether that control works in the deployed system is a separate claim requiring separate evidence.

This distinction is routinely collapsed. A team implements row-level security, reviews the migration, and records the control as complete. What exists is a design intention plus a code review — not a demonstration that the deployed database rejects a cross-tenant read.

The collapse is understandable. Reviewing a migration feels like verification: you have read the policy, you can see it is correct, and the deployment pipeline reported success. But every step in that chain is an assertion about intent. The question a probe answers is different in kind — not is the policy correct but does this database, right now, under this role, refuse this specific read.

12.2 A probe that passed and proved nothing

IntegraHive verifies isolation with a one-shot job: connect as the runtime role, attempt cross-tenant reads and writes that must fail, attempt in-tenant operations that must succeed, and emit a structured verdict.

The first implementation reported success. It also proved nothing.

The job completed and the platform reported a successful exit. But the executing pod had been garbage-collected before its result envelope could be captured. What survived was an exit status — evidence that a process ended, not evidence of what it concluded. A passing exit code from a probe whose output is gone is indistinguishable from a probe that did nothing at all.

This is a more interesting failure than a control that is simply broken, because every dashboard showed green. The verification pipeline was working as designed; the design was wrong.

12.3 What the probe actually checks

The probe is worth describing in detail, because the checks are the specific claims being made and a probe that tested less would prove less.

It connects only as the least-privilege runtime role, never as an administrative one, and runs six phases. Everything runs inside transactions that are unconditionally rolled back; nothing the probe does is ever committed.

Preflight — is this the role we think it is? The probe asserts that the current user is the runtime role, that it is not a superuser, does not hold the attribute that bypasses row-level security, cannot create roles or databases, does not replicate, and does not inherit privileges. It asserts the role has no inherited role memberships, owns zero relations and zero schemas, and holds no over-broad structural grants. It asserts the target table exists, has row-level security enabled, and is not owned by the runtime role. This phase exists because every later result is meaningless if the connection is privileged: a "pass" from a superuser connection proves only that superusers can do things.

Own-tenant operations must succeed. Insert, select, and update each affect exactly one row under the tenant's own context. A probe that only tested denial would pass on a database where nothing works at all.

Cross-tenant operations must fail. Another tenant's row is invisible. A cross-tenant insert is denied with the insufficient-privilege SQLSTATE. A cross-tenant update and delete each affect zero rows.

Empty context must deny. With no tenant context set, reads return zero rows and writes are denied.

Structural operations must be denied. Creating a table and altering one are both refused. Identifiers used here are internally generated and validated against an allowlist, so the probe cannot itself become an injection vector.

Post-rollback state must be clean. The contexts did not leak across transaction boundaries; an unscoped full-table read returns zero rows; an unscoped insert is denied; and the synthetic row did not survive the rollback. The unscoped read is treated as corroborating rather than as proof — a zero-row result could also mean an empty table — while the denied unscoped insert is the content-independent guarantee.

12.4 The redesign

Three changes address the evidence failure:

Capture before reclamation. The verdict is captured while the executing resource still exists, rather than read afterwards from whatever remains.

Bind the verdict to its run. Adjudication refuses to report a pass unless it can point at evidence originating from that specific execution. A pass inherited from an earlier run is not a pass.

Make ambiguity explicit. The probe emits four verdicts, not two: a pass, an operational failure reached before adjudication, a proven violation, and an explicit ambiguous verdict — a distinct outcome, not a pass — where evidence is absent or unreadable. Rounding missing evidence to success is how the original failure occurred; rounding it to failure would train operators to ignore the signal. A proven violation outranks an unknown, and an unknown never yields a pass.

stateDiagram-v2
  [*] --> Running
  Running --> Evidence: probe emits structured result line
  Running --> NoEvidence: process ended without capturable result
  Evidence --> Bound: result is bound to this run's identity
  Evidence --> Unbound: result cannot be attributed to this run
  Bound --> Pass: all phases passed
  Bound --> ProvenViolation: a phase proved a violation
  Bound --> OperationalFailure: failure before adjudication
  Unbound --> Ambiguous
  NoEvidence --> Ambiguous
  Pass --> [*]
  ProvenViolation --> [*]
  OperationalFailure --> [*]
  Ambiguous --> [*]

Figure 11 — Adjudication treats missing evidence as its own outcome. The two paths into ambiguous are the entire lesson. Evidence that never existed and evidence that cannot be attributed to this run are different causes with the same correct conclusion: we do not know. The original design had no such state — it had a successful exit code and an assumption — and so it rounded both of these paths into a pass. Note also that a pass is reachable only through Bound: a result the adjudicator cannot tie to the run that produced it can never be a pass, however good the result looks.

12.5 A shortcut that was rejected

The verdict could have been recovered from platform log aggregation by correlating on an execution-name field. That would have been considerably less work than redesigning the capture path.

It was rejected because the field is undocumented. A verification whose correctness depends on a vendor implementation detail is a verification with an expiry date nobody has written down. When the field is renamed — and undocumented fields are renamed, because nothing promised otherwise — the verification does not fail loudly. It reports ambiguity or, worse, silently correlates against nothing and reports whatever the fallback path reports.

The general form: a control's evidence path must not depend on behaviour the provider has not committed to. If it does, the control's expiry is set by someone who does not know it exists.

12.6 Output discipline

The probe emits a single whitelisted result line containing a status, the check results, and a pass count. It emits no tenant identifiers, no SQL, and no connection detail.

This is not incidental. A verification job runs with database access in an environment whose logs may be aggregated, retained, and read by people with different authorisation than the database itself. A probe that logged the rows it could and could not see, in the name of diagnosability, would be a cross-tenant disclosure mechanism attached to the cross-tenant control. The constrained output is a deliberate reduction in debuggability, paid for the guarantee that the evidence artefact is safe to retain.

12.7 Generalising

A control that works and a control you can demonstrate works are different deliverables. Only the second survives an audit, an incident review, or a new engineer asking "how do we know?"

Applied consistently, this reframes verification as a first-class artefact with its own failure modes, rather than a checkbox appended to implementation. The failure modes are worth naming: evidence that does not outlive its producer, evidence that cannot be bound to the run that produced it, evidence that depends on an unpromised interface, and evidence whose absence is rounded to success. All four were present in the original probe. Only the last is obvious.

12.8 Mutation testing and the accidental kill

The same distinction applies one level down, to the tests themselves.

Test suites are validated by mutation testing — deliberately breaking the code and confirming the suite notices. The metric is the kill rate, and the trap is why a mutant died. A mutant killed by the assertion targeting it is evidence. A mutant killed by a test timeout, a teardown failure, or residue from a previous mutant is not. Both increment the same counter.

Counting accidental kills as proof conceals precisely the coverage gaps the campaign was run to expose, and yields a confident number that means nothing. Worse, a mutant whose blast radius exceeds its targeting test can contaminate subsequent mutants in the same run, so one bad result silently corrupts those after it.

The resulting discipline: record the reason for each kill, treat accidental kills as survivals, and serialise database work so one suite's stray object cannot fail another's global scan. The parallel to the probe is exact — in both cases a green signal was being produced by something other than the thing it claimed to measure.

13. Failure modes

This section enumerates the ways the system is expected to fail, and what each failure is designed to degrade into. A design that has not been asked this question tends to fail in whichever way is most surprising.

13.1 A partner sends a file that does not match the contract

Expected outcome: the run reaches the parsing stage, non-conforming records are quarantined with reason codes, the schema diff is classified, and the dataset's drift policy decides whether the run proceeds, parks for review, or fails.

Degradation: partial. Conforming records may still be accepted; the quarantined ones remain available for reprocessing once the contract is corrected. No data is lost and the partner does not need to resend.

13.2 The same file arrives twice

Expected outcome: the second delivery is caught on its checksum at landing and recorded as a duplicate delivery. No processing cost, no partial application.

Degradation: none. This is a designed-for case rather than a failure.

13.3 Ingestion fails partway

Expected outcome: run status names the stage that failed. Artefacts of completed stages survive. Reprocessing re-runs the failed stage.

Degradation: graceful. The main risk is operator confusion if the status vocabulary is treated as a success flag, which is why it is a constrained enumeration rather than a boolean.

13.4 A consumer crashes mid-message

Expected outcome: the message is not acknowledged, the acknowledgement window expires, and it is redelivered up to the configured attempt limit. Exhausted messages go to the database-backed dead-letter store.

Degradation: at-least-once delivery, which means consumers must be idempotent. This is an accepted design constraint rather than a solved problem — it is stated here because "the queue handles it" is a common and wrong belief.

13.5 The event spine is unavailable

Expected outcome: synchronous ingestion continues to land, infer, register, stage, and govern, because none of those depend on the spine. Asynchronous follow-on work does not run.

Degradation: partial availability. Evidence continues to be captured; downstream propagation is delayed. This is the correct asymmetry — the platform would rather have a delayed delivery than an uncaptured one.

13.6 The database is unavailable

Expected outcome: readiness reports the unavailable category and the instance leaves rotation; liveness continues to report alive so the platform does not restart processes that are not broken. Ingestion requests fail with an error rather than succeeding without persisting.

Degradation: full outage of the write path. This is the one dependency with no meaningful degraded mode, and the design accepts that rather than pretending otherwise: a landing zone write that is not durably recorded is worse than a rejected request, because the partner believes they have delivered.

13.7 The secret store is unavailable

Expected outcome: running workloads that have already resolved their secrets continue. New instances fail to start. Readiness reflects the dependency with a consecutive-failure threshold so a single transient error does not remove healthy instances.

Degradation: the platform survives a brief outage and fails to scale during a sustained one.

13.8 A tenant context is not set

Expected outcome: the database policy returns zero rows and denies writes.

Degradation: the request fails visibly and returns nothing. This is the failure mode the whole isolation design optimises for — the unset case must be empty, never everything. Application-level filtering has the opposite default, and that asymmetry is the entire argument of Section 8.

13.9 Detection is wrong

Expected outcome: a false positive raises an advisory event with a severity; an operator dismisses it. A false negative means an anomaly is not raised.

Degradation: none to the data path, by construction, because detection cannot block. The cost of a false negative is that a real problem is found later by other means; the cost of wiring detection into blocking would be that operators raise thresholds until it never fires.

13.10 A guard test stops matching

Expected outcome: the guard suites include fixture tests asserting that each rule still matches a synthetic example of what it is supposed to catch. A rule that stops matching fails the build.

Degradation: this is the failure mode with the worst profile in the whole system, because a silently dead guard produces confidence without coverage — which is Section 12.2's failure in different clothing. It is why the fixture assertions exist at all.

14. Security boundaries

14.1 Boundaries, stated as what crosses them

The security model is easiest to state as a list of boundaries and what is permitted to cross each one.

Internet to application zone. Only HTTP to the web workload, through managed ingress with a managed certificate. Strict Content-Security-Policy including frame-ancestors 'none', HSTS, X-Content-Type-Options, Referrer-Policy, and Permissions-Policy are set on responses. Client-supplied identity headers do not cross: they are stripped.

Application zone to data zone. Only over the private network. There is no public database endpoint and the event spine has internal ingress only. The connection is made by the least-privilege runtime role.

Workload to secret store. By workload identity only. No secret crosses as configuration.

Tenant to tenant. Nothing crosses, subject to the boundary analysis in Section 8.5. This is the boundary the platform exists to maintain and the one whose end-to-end proof is still outstanding.

Private repository to published material. Covered in Section 14.3.

14.2 API surface and versioning

The API is versioned, and unversioned routes emit Deprecation and Sunset headers alongside the version header, following the standard for signalling resource sunsetting. A deprecation registry allows a specific route to carry a sunset date, a successor link relation, and a human-readable notice.

This matters for a boundary reason rather than a tidiness reason. An unversioned, undated API is a permanent compatibility obligation, and permanent obligations accumulate until the only safe change is no change. Signalling a sunset converts a compatibility question into a scheduled one.

14.3 Guarding what gets published

A platform of this kind accumulates identifiers: tenant and application identifiers, cloud resource names, database hostnames, vault names. They are not credentials, but published together they map an environment.

This becomes a live risk the moment any public material is produced — a documentation site, a conference talk, an architecture write-up, a screenshot. The usual control is proofreading, which fails for the same reason application-level tenant filtering fails: it depends on a human remembering, every time, forever.

The mechanism used here is a pattern-based guard running in continuous integration across every tree that is written to be published. It matches shapes rather than a list of known-sensitive values:

• identifier formats such as GUIDs
• cloud service hostnames — container platform, database, registry, vault, object storage
• cloud resource identifier paths
• the internal resource naming convention
• credential shapes: credentialed URLs, API-key prefixes, private-key blocks
• private repository references, CI run identifiers, and internal issue references

Shapes rather than a deny-list, for two reasons. A deny-list of real values is itself a catalogue of what to protect, and it can only catch identifiers somebody remembered to add. A shape catches the ones nobody thought of.

The guard fails closed: a missing directory, an unreadable file, or a rule that stops matching its own fixture trips the build rather than passing vacuously. Its allowlist is itself asserted — widening it fails a test, so adding an entry to make a build go green becomes a visible decision in review rather than a one-line fix under time pressure.

That last check matters more than it appears. A guard that silently stops matching is worse than no guard, because it produces confidence without coverage — the same failure described in Section 12.2, in a different costume.

14.4 A public copy of the implementation was rejected

Publishing a sanitised subset of a private repository invites two failure modes: a partial copy that drifts from reality, and a sanitisation pass that misses something. Both are permanent once published.

Public material is instead authored directly against a single reviewed facts document, with the leak guard running over everything public-bound. One review surface instead of six, and the guard is the backstop rather than the primary control.

15. Current limitations

Overstating status is the fastest way to lose a technical reader, so this section is deliberately unflattering.

15.1 Deployment status

Component	Status
Application, console, ingestion pipeline	Built and running
Container Apps runtime, private VNet, managed PostgreSQL	Deployed and live
SSO, admin authorization, tenant resolution	Built and live
Terraform across all layers, CI guards, gated apply	Built
Row-level-security tenant isolation	Implemented; end-to-end proof in progress
Staging and production environments	Defined in code, not deployed

A development environment runs the full stack. Staging and production exist as code and are not deployed. There are no customers, no production traffic, no uptime record, no benchmarks, and no certifications, and this document contains no figures for any of them because none exist.

15.2 The isolation proof is not complete

This is the honest open item and the one that matters most, because it is the platform's central claim.

The control is implemented: policies exist on the tenant-scoped tables, they specify both read and write constraints, they are forced so that ownership does not exempt, the runtime role is least-privilege and owns nothing, and tenant context is transaction-local.

The verification has been redesigned along the lines of Section 12.4. It has not yet produced a clean end-to-end pass bound to durable evidence from its own run. The first probe run returned an explicit ambiguous verdict — a distinct outcome, not a pass — after the one-shot execution environment was reclaimed before its result envelope could be captured, and the redesign has not yet produced a clean passing run.

Under the standard this paper argues for, that means isolation is implemented but not yet proven, and the correct thing to do with that status is publish it rather than round it up. Any document from this project that claims verified tenant isolation is wrong, and this sentence exists so that a reader who encounters such a claim knows it contradicts the source.

15.3 Legacy tenant-context conventions require consolidation

Historically, different parts of the schema adopted different conventions for establishing transaction-local tenant context. The surviving conventions are canonicalised behind a single module, so that no call site chooses one by hand and the probe exercises the canonical path.

This is not a vulnerability, but it is a defect: more than one convention for a single concept is a standing invitation for a future table to be created with a policy referencing the wrong one, and the failure would be a policy that never matches. Converging on one convention is a migration that has not been done.

15.4 Not every policy states its write constraint explicitly

The platform tables specify both USING and WITH CHECK. Some of the later-added tables specify only USING and rely on PostgreSQL's behaviour of applying the USING expression as the write check when none is given.

That behaviour is documented and correct today, so the isolation is real. The defect is that the write constraint is implicit: if someone later relaxes a USING clause for a legitimate read reason, the write constraint silently relaxes with it. The mitigation is explicit WITH CHECK on every policy, which one table already has and the rest do not.

15.5 The cross-tenant administrative path is a process control

As described in Section 8.5, privileged cross-tenant administrative operations — migrations, platform-wide reporting, maintenance — exist as a separate path outside the least-privilege application runtime, so that they can function at all.

What remains open is that the path is bounded by process rather than by the database: it is kept narrow by review and by keeping such paths few, which is to say it is enforced by discipline. The paper's own argument — that trust is not an enforcement mechanism — applies here and is not answered.

15.6 Schema fingerprints are order-sensitive

The fingerprint is computed over ordered column tuples, so a partner reordering columns mints a new schema version and registers as drift even when the content is identical. For positional formats this is correct. For formats where column order is not semantically meaningful it produces false drift, and the current model has no way to express that distinction per dataset.

15.7 Two API-version implementations disagree on format

Version signalling exists in two places — a global path applied to all API responses and a per-route registry with richer deprecation metadata. They emit the version header in different formats. Both are individually correct; together they are inconsistent, and a client parsing the header strictly could see either form.

15.8 The quarantine vocabulary is narrower than the schema suggests

The quarantine reason code is stored as free text with no database constraint, while the application constrains it to a small closed vocabulary through the type system. The type system is not present at the database boundary, so a future writer could store an unrecognised code without anything failing. The record-level violation detail is richer than the top-level code, which is a reasonable design, but the top-level enumeration should be constrained where the data lives.

15.9 Detection has no ground truth

The anomaly detectors raise events, and there is no labelled evaluation set to measure them against. Their precision and recall are unknown. This is stated rather than glossed because a detector with unmeasured accuracy is a hypothesis, and the honest description of a hypothesis is not "detection is in place" but "detection runs and its accuracy has not been characterised."

15.10 Single-region, single-environment

Everything above describes one environment in one region. Multi-region behaviour, failover, backup restoration timing, and disaster recovery are designed for in the infrastructure layout but have not been exercised. Nothing in this document should be read as a claim about them.

16. Roadmap

The roadmap is ordered by what would most change the truthfulness of this document, rather than by feature appeal.

First: close the isolation proof. Produce a clean probe run whose verdict is bound to durable evidence from that run, retained independently of the executing resource. Until that exists, the platform's central claim is an implementation claim rather than a verified one, and everything else is lower priority.

Second: make the proof continuous. A single passing run proves a moment. The control's value is continuous, so the evidence should be too — the probe should run on a schedule and on every change to a tenant-scoped table or policy, with an ambiguous verdict treated as a build failure rather than a warning. A control verified once is a control verified before the last forty changes.

Third: converge on a single tenant-context convention and make every write constraint explicit. Both are mechanical migrations. Both remove a class of future defect rather than fixing a current one, which is the kind of work that never becomes urgent and should therefore be scheduled.

Fourth: constrain the quarantine vocabulary at the database boundary, and extend it. Two reason codes is a small vocabulary for the range of ways partner data fails; the codes should grow as the failure taxonomy is understood, and be constrained where the data lives rather than only in the type system.

Fifth: characterise detection. Build a labelled evaluation set from historical runs and measure the detectors against it, so their events can be described with a known error profile instead of an assumed one.

Sixth: deploy staging. A second environment is the only way to test the infrastructure layer's claim to be environment-parameterised, and the only place where a promotion path can be exercised without touching the environment everything currently depends on.

Notably absent from this list: new intake channels, new detector kinds, and additional transformation capability. Each is more visible than anything above and none of them changes whether the platform's core claim is true.

17. Conclusion

Two claims, in order of confidence.

The first is narrow and well supported: moving the tenant boundary from application code into the database, with a least-privilege runtime role that owns nothing, forced policies specifying both read and write constraints, and transaction-scoped context, converts the most damaging bug class in a multi-tenant system from unlikely to unrepresentable. The cost is real and bounded — care with connection pooling, complications in administrative queries, and a role discipline that must be maintained as part of the control rather than as deployment detail.

The second is broader and drawn from the failure in Section 12: for controls that matter, verification deserves the same design scrutiny as implementation. A probe whose evidence does not outlive the process that produced it is not a weaker control — it is not a control at all, while presenting as one. The platform's dashboards were green throughout.

The uncomfortable implication is that a team can hold an accurate inventory of implemented controls and an entirely inaccurate picture of which ones are demonstrably working. Distinguishing the two is cheap to do deliberately and expensive to discover after an incident.

This paper ends where Section 15 does rather than where a conclusion would prefer to. The isolation control is implemented and its proof is outstanding. A document that argued for the difference between a control that works and evidence that a control works, and then claimed the evidence it does not yet have, would be a worked example of its own thesis.