De-Identification Pipeline

The pipeline must simultaneously process heterogeneous clinical data types to locate identifiers across all modalities before redaction:

• Structured Data: Scans database columns for the 18 HIPAA Safe Harbor identifiers, including names, dates, and medical record numbers.
• Unstructured Text: Applies Named Entity Recognition (NER) fine-tuned on clinical corpora to detect PHI in physician notes, discharge summaries, and radiology reports.
• Medical Imaging: Uses computer vision models to detect and redact burned-in pixel PHI in DICOM headers and image regions, such as ultrasound overlays containing patient names.
• Audio Transcripts: Transcribes and then processes clinical dictation for spoken identifiers using automatic speech recognition coupled with text NER.

Simple pattern matching is insufficient; the pipeline must apply contextual disambiguation to avoid over-redaction that destroys clinical utility:

• Negation Handling: Distinguishes 'Patient John Doe' (PHI) from 'John Doe syndrome' (a medical eponym, not PHI) using dependency parsing.
• Temporal Normalization: Converts absolute dates (e.g., 'January 5, 2023') to relative offsets (e.g., 'Day 0 + 14 days') to preserve clinical timelines while satisfying the Safe Harbor method.
• Provider vs. Patient Distinction: Classifies whether a detected name belongs to the patient, a family member, or a treating physician, redacting only the former.
• Co-reference Resolution: Tracks pronouns and anaphoric references across sentences to ensure 'he' or 'the patient' linked to a redacted name is also consistently handled.

Beyond heuristic redaction, the pipeline must provide formal privacy guarantees to satisfy institutional review boards and regulators:

• HIPAA Expert Determination: Applies statistical models to certify that the risk of re-identification is 'very small,' as defined by §164.514(b), using methods like k-anonymity or k-map estimation.
• Differential Privacy Integration: Injects calibrated noise into the de-identified output to bound the influence of any single individual's record, quantified by the epsilon privacy budget.
• Re-Identification Risk Scoring: Assigns a quantitative score to each output record based on population uniqueness metrics, flagging high-risk records for manual review before release.
• Adversarial Validation: Periodically tests the pipeline with simulated re-identification attacks using external datasets to empirically measure residual risk.

The pipeline acts as a mandatory computational gate that executes locally at each institution before any data or gradient leaves the trusted environment:

• Local-Only Execution: All de-identification computation runs within the hospital's firewall, ensuring raw PHI never traverses the network.
• Schema Validation: Enforces a strict output schema that rejects any record containing unredacted identifier patterns, acting as a circuit breaker for data leaks.
• Differential Privacy Budget Tracking: Integrates with the federated aggregator to deduct from the global privacy budget, halting training when the epsilon threshold is exhausted.
• Audit Log Generation: Produces a cryptographically signed manifest of all redaction actions, including the specific PHI types detected and the transformation applied, for downstream chain-of-custody verification.

A robust pipeline quantifies the trade-off between privacy and data utility to prevent rendering clinical data useless for downstream model training:

• Information Loss Scoring: Measures the percentage of clinically relevant tokens removed versus PHI tokens, alerting operators if over-redaction exceeds a configurable threshold.
• Concept Retention Analysis: Verifies that key clinical concepts (diagnoses, medications, procedures) remain intact post-redaction using a pre-trained medical ontology matcher.
• Downstream Task Validation: Runs a lightweight proxy model on the de-identified output to confirm that predictive performance on a held-out task (e.g., mortality prediction) does not degrade beyond an acceptable margin.
• Synthetic Surrogate Generation: Where redaction would destroy critical structure, replaces PHI with realistic but fictitious surrogates (e.g., synthetic names) that maintain linguistic and statistical properties.

The pipeline must dynamically adapt its redaction policy based on granular patient consent directives managed by the consent orchestration layer:

• Attribute-Level Control: Applies different redaction rules per data field based on individual patient permissions—one patient may allow age sharing while another requires full date-of-birth suppression.
• Purpose-Based Filtering: Tags each data element with its allowed processing purposes (e.g., 'research only,' 'treatment excluded') and filters the output stream accordingly.
• Right-to-Erasure Propagation: When a patient invokes the right to erasure, the pipeline flags all records associated with that identity for exclusion from future federated rounds and logs the action immutably.
• Real-Time Consent Revocation: Integrates with the institutional consent registry to immediately halt processing of a specific patient's data if consent is withdrawn mid-pipeline execution.