FHIR Bulk Data Access

Unlike real-time FHIR searches, Bulk Data Access uses a kick-off and poll architecture. A client requests an export, and the server initiates a long-running job. The client then polls a status endpoint until the job is complete, at which point it receives a manifest of download URLs. This prevents timeouts when exporting millions of records.

Exported data is formatted as Newline Delimited JSON (NDJSON) , a flat file format where each line is a valid JSON object. This format is ideal for streaming large datasets because it is:

• Splittable: Files can be partitioned for parallel processing.
• Appendable: New data can be added without rewriting the entire file.
• Line-Processable: Tools can read one record at a time without loading the full dataset into memory.

The specification defines three distinct scopes for data extraction, each serving a different analytical need:

• System-Level Export: Exports data for all patients on a server. Requires powerful authorization.
• Patient-Level Export: Exports all data for a specific, pre-defined list of patients.
• Group-Level Export: Exports data for all members of a defined Group resource, such as a cohort of diabetic patients. This is the most common mechanism for population health analytics.

To simplify implementation, the specification defines a Flat FHIR profile. Instead of complex, nested resource graphs, servers export a pre-defined set of resource types as flat, independent files. The standard types include:

• Patient: Demographics.
• Observation: Vital signs and lab results.
• Condition: Diagnoses and problems.
• MedicationRequest: Prescribed medications.
• AllergyIntolerance: Documented allergies. This flat structure is optimized for direct loading into analytical databases.

Bulk data access is secured using the SMART Backend Services authorization profile, not user-facing OAuth. This machine-to-machine flow relies on:

• JSON Web Keys (JWK) : For client authentication.
• Signed JWT Assertions: To request access tokens.
• OAuth 2.0 Client Credentials Grant: To obtain a token scoped to the requested export level. This ensures that only registered, non-interactive systems can initiate large-scale data transfers.

The specification includes robust error handling for long-running jobs. If an export fails, the server returns an OperationOutcome resource detailing the error. Files are generated transactionally; a download URL is only provided for a file that was fully written. Clients must implement exponential backoff when polling the status endpoint and handle partial failures gracefully by re-requesting only the failed files.

Unlike the standard RESTful API that returns interconnected FHIR Bundles, Bulk Data Access generates flat, denormalized NDJSON files. Each line is a single resource (e.g., a Patient or Observation), optimized for direct ingestion into analytics tools like Spark or BigQuery without recursive parsing. This design sacrifices relational context for linear scan performance on massive datasets.

Bulk Data Access abandons user-centric OAuth flows in favor of SMART Backend Services Authorization. This machine-to-machine profile uses asymmetric cryptography (JWKS) to establish a trusted client identity without a human present. The client signs a JWT assertion to obtain an access token, enabling unattended, automated batch jobs that run during off-peak hours.

The specification defines two distinct scopes:

• Patient-Level Export: Targets all resources linked to a specific Group of patients, ensuring a longitudinal view of a cohort.
• System-Level Export: Exports all data for a given resource type (e.g., all MedicationRequests) from the entire server, regardless of patient linkage. This is critical for data warehousing and cross-entity analytics but requires stricter governance.

To maintain data integrity, the export manifest includes an attestation mechanism. The server can provide a checksum or digital signature for each output file, allowing the consuming system to verify that the data has not been corrupted or tampered with during transfer. This ties directly to the FHIR Provenance resource to establish a complete chain of custody for the exported dataset.