Governance and quarantine

Governance is the stage where the platform decides whether what a partner sent is allowed to become what downstream consumers read. It runs before acceptance, not as a later sweep, and its two visible outputs are a decision about the dataset's schema and a set of records held in quarantine.

This page covers how policy is evaluated, what causes a record to be quarantined, what happens to it afterwards, how contract violations reach an operator, and why the platform never silently drops anything.

What "governance" means here

The word is used loosely across the industry, so it is worth being precise about the claim. Governance in this platform is three specific evaluations:

1. Contract conformance — does each record satisfy the registered contract's column set, types, and nullability?
2. Change classification — if the observed schema differs from the contract, what class of change is it, and does this dataset's policy permit it?
3. Data handling — which fields are masked by policy, and how long do landed and staged artefacts persist?

What governance is not is a post-hoc audit. The ordering carries most of the value: a sweep running after acceptance can tell you non-compliant data is in the warehouse, but by then a consumer has read it and the remediation is a retraction rather than a rejection. Evaluating before acceptance means the worst case is a delayed dataset, not a poisoned one.

Status: implemented. Contract conformance, change classification, and the per-dataset drift policy all run in the live development environment.

Observed schema versus contracted schema

Governance needs something to evaluate against, and that something is deliberately not "whatever arrived last time".

Each dataset has an append-only history of schema versions, over which two markers move independently:

• The current version is descriptive. It moves with every run and records what the platform most recently observed.
• The contract version is authoritative. It moves only by an explicit promotion, which records who approved it and when, and it is what governance evaluates against.

Each marker is enforced by a partial unique index, so the database itself guarantees at most one current and at most one contract version per dataset. The invariant cannot be broken by a race between concurrent runs because it is not maintained by application logic.

The consequence is the point: a partner changing their file cannot change what the platform considers correct. Drift becomes a reviewable event with a named approver rather than a silent redefinition of the agreement.

Classifying drift

When the observed schema differs from the contract, the difference is decomposed into typed change kinds — a column added, a column removed, a type changed, or nullability changed — and each carries a reason code explaining why the classifier reached its conclusion.

The reason codes exist because a coarse "type changed" flattens cases that deserve different responses. A widening that cannot lose information is not the same as a narrowing that can. A change of meaning is not the same as a change of representation. A classification reached with low confidence is not the same as a confident one. A new column that looks sensitive is not the same as one that does not, and a new required column is not the same as a new nullable one.

One rule is worth naming on its own: any multi-digit value with a leading zero forces its column to a string type and blocks numeric coercion. The values this protects are identifiers that look like numbers — zero-padded account codes, postal codes, SKUs, tracking references. Coercing them to an integer is lossless to the type system and catastrophic to the business, because 00123 and 123 are different accounts and the merge is invisible until a join produces the wrong customer.

Policy: auto, review, block

Classification answers what changed and how risky is it. Policy answers what should happen about changes of that risk. They are separate steps, so a tenant can run one dataset permissively and another strictly without needing two classifiers, and so a classification recorded against a historical run stays meaningful even if the policy is later changed.

Each dataset's drift policy resolves a change to one of three dispositions:

Disposition	Behaviour
auto	Applied without human involvement. Reserved for provably safe changes — a strictly widening type change, a new nullable column.
review	Parked for an operator. The run does not proceed to acceptance on the changed shape until someone approves or rejects it.
block	Refused. The run fails at governance and the contract stands unchanged.

Every outcome is recorded with a resolution — applied automatically, parked, approved, or rejected — so "who allowed this change?" always has an answer.

What causes a record to be quarantined

Schema governance operates on the shape of a delivery. Quarantine operates on individual records that cannot be normalised against the contract.

Two categories of reason are emitted at the record level. The literal codes are internal; what matters publicly is the distinction they draw.

A structural failure — the row's field count does not match the header, or the header itself is ambiguous or contains duplicate column names. The row cannot be interpreted as a record at all.

A semantic failure — a column the contract declares as required is missing, or its value cannot be coerced to the declared type. The row is a record, but not a valid one.

Within the second case, the diagnostic detail distinguishes a missing required value from a failed type coercion, and names the field. An operator sees which column and why, not merely this row is bad.

  parsed row
     │
     ├─ not structurally interpretable ──► quarantine: structural failure
     │
     ├─ required field missing or uncoercible ──► quarantine: semantic failure
     │
     └─ conforms ──► accepted into the staged set

What happens to a quarantined record

It is written to a quarantine store with its reason code, its originating run, and enough diagnostic context to act on. It stays there.

That enables two things a drop makes impossible. An operator can see the shape of the problem across many records rather than reasoning from one. And a corrected contract can reprocess the held records without asking the partner to resend — which matters when the delivery window has closed and the file is large.

Quarantine is a stage in a record's life, not the end of it. The loop back into the accepted set is the entire reason the design exists.

Why nothing is silently dropped

The rule is isolate, never drop, and it is a rule rather than a preference because the alternative is quietly corrupting.

A pipeline that discards malformed records reports a high success rate, produces datasets that look complete, and destroys precisely the evidence needed to repair the partner integration generating the malformed records. The metric improves as the problem worsens. There is no point at which the system reports that anything is wrong.

Retention also turns loss into signal. Because quarantined records are kept and attributed to their run, the quarantine rate becomes a measurable property of the integration. A dataset whose quarantine rate jumps between runs is telling you something changed on the partner side, usually before anyone has filed a ticket. A pipeline that dropped those records would show the same accepted row count and no signal at all.

The same rule applies one layer in, to the event spine: a message whose delivery attempts are exhausted goes to a database-backed dead-letter store rather than being discarded, where it can be queried, joined against the run that produced it, and reprocessed.

How violations surface to operators

Three channels, deliberately distinct:

Run status names the stage a run halted in. The vocabulary is a constrained set rather than free text: a run moves from receipt through landing, parsing and loading, with terminal states for a delivery recognised as a duplicate of one already accepted and for a run that failed. That set is enforced by a database check constraint and mirrored as a type in the application, so the two cannot drift apart without a type error. A run that failed while landing is an artefact problem; one that failed while parsing is a contract or data problem. Those go to different people, and the status says which.

Parked schema changes wait for an explicit approve or reject. The run does not silently proceed on an unapproved shape, and it does not silently fail either — it waits, visibly.

Anomaly events are raised by detectors as the final pipeline stage, each tagged with a severity drawn from an ordered scale so an operator can triage a backlog rather than read it. Five detector categories are implemented. They compare a delivery against the dataset's own recent history along five axes: delivery volume, field completeness, the rate at which records fail normalisation, how confident inference is about a column's type, and repetition within a single delivery.

Detection is a pipeline stage rather than an external monitor, so an anomaly is attributable to the run that produced it, with that run's schema version, row counts, and quarantine set available as context.

Status: detection is implemented and running. Its accuracy has not been characterised — there is no labelled evaluation set, so precision and recall are unknown. The honest description is "detection runs and its error profile is uncharacterised", not "anomalies are detected".

What detection deliberately does not do

Detection raises events. It does not block runs and it does not quarantine records.

Governance is deterministic and contractual: a record either satisfies the contract or it does not, and the consequence is defined in advance. Detection is statistical and contextual: a forty per cent row-count drop is alarming for one dataset and a normal Monday for another. Wiring a statistical signal into a blocking decision means a distribution shift can halt a pipeline, and the operational response to that is invariably to raise the threshold until it stops happening — which is to say, to switch it off.

Keeping detection advisory keeps it honest. A false positive costs an operator a glance rather than costing a partner a delivery window.

Known gaps

Stated here rather than left for a reader to find.

The quarantine reason code is not constrained at the database boundary. It is stored as free text; the small closed vocabulary is enforced only by the application's type system. A future writer could store an unrecognised code without anything failing. The constraint belongs where the data lives.

Two categories is a small vocabulary for the range of ways partner data fails. It should grow as the failure taxonomy is understood.

Detection has no ground truth, as above. A detector with unmeasured accuracy is a hypothesis.

Related reading: the governed ingestion pipeline for the seven stages this sits inside, and operational evidence for how the platform distinguishes a control working from evidence that it works.