The governed ingestion pipeline

Every intake channel converges on one pipeline, so behaviour does not fork by how the data arrived. A file uploaded by an operator and a payload posted to a keyed endpoint follow the same seven stages and produce the same audit trail.

land → infer → register → stage → govern → quarantine → detect

1. Land

Data is received into a tenant-isolated landing zone. A manifest and SHA-256 checksum are recorded on arrival, so a duplicate payload is caught before any processing cost is paid, and the exact bytes that arrived remain reproducible after the fact.

Nothing is parsed at this stage. Capture is deliberately separated from interpretation: if parsing fails later, the original artefact is still intact.

2. Infer

The landed file is profiled to infer its structure — columns, types, nesting — without assuming the partner sent what was agreed. This is the difference between a pipeline that reports "load failed" and one that reports which column changed and how — a date column that arrived as a string, say.

3. Register

The inferred schema is registered as a version against the dataset's contract. Every change is diffable against the previous version, so schema drift is a reviewable event with history rather than an incident discovered downstream.

4. Stage

Records are parsed and staged for loading. Run status moves from receipt through landing, parsing and loading, with row counts tracked at each transition. A run that stops partway names the stage it stopped in, which is most of the diagnostic work.

5. Govern

Policy is applied before data is accepted downstream:

• Schema-drift policy — auto, review, or block, configured per dataset (see governance)
• Field masking — sensitive columns masked by policy, not by convention
• Retention — how long staged and landed artefacts persist

Governance runs before acceptance rather than as a later sweep, so non-compliant data never becomes the thing downstream consumers already read.

6. Quarantine

Records that breach the contract or fail validation are quarantined with a reason code and diagnostic context.

The design rule is isolate, never drop. A pipeline that silently discards malformed records optimises its own success metrics while destroying the evidence needed to fix the partner integration. Quarantined records stay reviewable, and a fixed contract can reprocess them.

7. Detect

Statistical and ML detectors raise events for schema drift, value outliers, and threshold breaches, with severity surfaced to operators. Detection is a pipeline stage rather than an external monitor, so an anomaly is attributable to the run that produced it.

Why one pipeline

Per-channel pipelines drift. The SFTP path grows a quirk the REST path does not have; a governance rule gets applied in one and forgotten in the other; and the divergence is only discovered when someone asks why two partners with identical contracts produced different results.

Converging every channel on one pipeline means governance is written once and cannot be selectively skipped by choosing a different door.