How it works
deidentify(text, deps, config) runs the whole pipeline on your side and ends at one of two
outcomes: a sealed BoundaryArtifact, or a refusal with a reason. Either way the transcript
stays local and the run produces a complete local ledger.
The stages
1
Stage A: tokenize
The configured Stage-0 detectors find identifier spans: by default Presidio (with the New
Zealand, Australian, and UK recognizers and checksum checks) and the
gliner-pii-open model.
Every detector reads a copy of the text with Unicode spaces and hyphens folded, so a phone
number written with a narrow space is still found. Then the guards every detector shares
refuse spans that are not identifiers: a speaker label, a relative time (“last week”), a
pronoun tagged as a name, a word tagged as a number, a clinician role (“the GP”), and a
generic department (“A&E”). A patient’s own occupation and a named institution still count.
Each remaining span becomes a typed token such as <NAME_1>; the same original always maps
to the same token. The token map, with the originals, is the auditable layer and stays
local.2
Stage B: the judge loop
A judge ensemble reads the partly tokenized text and proposes residual identifier strings the
detectors missed. The model only proposes. Code searches for each string, whole-word and
case-insensitive, and tokenizes it only when it is present. It also accepts the same words
with a space for a hyphen (“fifty-three” for “fifty three”) and the spoken form of a number
the judge wrote as digits (“54” for “fifty-four”). A proposal the judge itself labels as
clinical content (a symptom, a condition) is ignored. The loop stops when two
consecutive rounds are dry. If it hits the round cap while still finding things, the engine
refuses; a persistent residual the matcher cannot apply can never pass as clean.
3
Stage C: fill
Each token is replaced by a realistic, type-correct surrogate, consistent per original: one
fake name for one person, a banded age, a date as an offset from the visit. A missed
identifier now sits among plausible fakes, so nothing downstream can tell which value is real.
4
Extraction
The extractor model walks the filled transcript in order and emits typed clinical beats, an
age band, and sex. The prompt forbids inventing a value for a stripped field: a fact that
cannot be stated without an identifier is omitted. The transcript is wrapped as untrusted
data so instructions inside it are ignored.
5
Beat cleaning
Every surrogate copied into a beat is replaced by a neutral phrase (a fake occupation never
crosses as a fact), each beat is re-detected and neutralized, and beats naming a condition the
data confirms is rare are dropped: the national tables, plus about 10,800 names and synonyms
of disorders Orphanet confirms are rare. An unknown condition is not treated as rare.
6
Singling-out and seal
The demographic shell (age band and sex) must pass the k-anonymity check at
boundary_k
against the base-rate provider, else the engine refuses. The check does not cover clinical
combinations, and its audit entry says so; those are left to the rarity check and the judge. A field the transcript did not
state is unspecified: it is judged against the whole population on that axis and is
never guessed. Then the artifact is sealed with an integrity digest and an opaque token is
minted.When it refuses
DeidResult.crossed is False, artifact is None, and refusal_reason is one of:
A refusal is the control working. Do not retry with a weaker configuration. Re-run only after a
change that removes the cause: a fixed transcript, a better model, or a keep entry for a value the
engine wrongly treated as an identifier. Read
runs/<id>/ledger_local.json to see every judge
round; record the reason in your own audit, because there is no artifact to carry it.
Two errors are raised before any artifact and are not refusals: TranscriptTooLargeError (over
max_transcript_chars, default 200,000 characters) and TruncatedCompletionError (the model hit
its output cap mid-answer; a cut-off structured reply is never treated as a partial result).
Coverage
Once beats exist, the engine reads how much of what the patient said they carry: for each patient utterance of the tokenized text, the best content-word overlap with any beat, and the utterances under a threshold (0.2) are reported by line index in the local ledger, with one counts-only line onrun. It is a reading of the extractor’s faithfulness, not a gate: nothing
crosses differently because of it, and no utterance text leaves the local ledger.
The ledger
Every run builds aTransformationLedger: the token map with each finder, every judge proposal
applied and rejected per round, every surrogate fill, and, when a TraceCollector is wired in,
the full request and response of every model call. local_view() is the complete record for your
Data Protection Officer (DPO) and stays local. crossing_view() is counts and types only, and is
the only view that reaches the artifact’s audit log.
Where each stage lives: detectors in detectors.py, stage0_prepass.py, gliner_detector.py;
tokens and the matcher in tokenize.py; the loop in clean_loop.py; extraction in
stage1_timeline.py; cleaning and orchestration in engine.py; singling-out in
singling_out.py; the contract and sealing in contracts.py.