The 5-Tier Medallion Architecture for Clinical AI

High-throughput, low-latency event streaming from edge devices and HL7 FHIR API sources. Partitioning by department enables parallel consumption. Modern deployments use KRaft mode, eliminating ZooKeeper for simplified operations.

Micro-batch ingestion jobs consume from Kafka topics and append raw JSON/HL7 payloads as immutable Parquet/Iceberg files on the object store, with exactly-once processing semantics.

Acts as the underlying distributed storage layer. All raw files are append-only. Portable across AWS S3, Azure ADLS Gen2, GCS, and on-premises MinIO/Ceph via the S3A connector.

Provides ACID-compliant snapshot isolation, time-travel queries, and schema evolution without rewriting historical data.

Implemented via Iceberg's transaction model to handle thousands of simultaneous writer streams without lock contention.

A Gemma or lightweight open-weight LLM runs as a sidecar to the Flink/Spark ingestion consumers, so redaction happens inline before each record is committed to Iceberg — no raw text field is ever written to durable storage unredacted. See Data Protection Enforcement below for what the redaction step itself covers.

A Gemma or lightweight open-weight LLM sidecar container performs permanent, irreversible PII Redaction on raw text fields before any payload leaves the hospital network perimeter. This is the first line of defense — direct identifiers (names, SSNs, MRNs) are stripped at ingestion time using NER-based detection.

Masking is applied to device telemetry metadata fields (serial numbers, location codes) that could serve as quasi-identifiers. Format-preserving placeholders maintain structural consistency for downstream parsing without exposing device-to-patient mappings.

Parse raw JSON/HL7/FHIR payloads into structured Parquet tables with Iceberg's schema enforcement. Delta detection via MERGE INTO operations deduplicates late-arriving events.

Centralized schema registry for all Silver tables, consumed by Spark, Flink, Trino, and ML pipelines uniformly across clouds.

Row-level and column-level access policies enforce multi-tenant isolation using Attribute-Based Access Control (ABAC). Partition key (/TenantID/DepartmentID) ensures strict isolation.

Tracks full data lineage from Bronze ingest through every Spark transformation. PHI classification tags propagate automatically to all derived columns.

The Silver layer operates as the Command (Write) side. Long-term clinical states are written to Iceberg tables; operational real-time states are written to a low-latency distributed database (Cassandra or CockroachDB/YugabyteDB).

PII Redaction — permanent, irreversible removal of all direct identifiers — is enforced at the Silver boundary via NER-based detection (Microsoft Presidio, spaCy, or Google Sensitive Data Protection). Every record entering Silver passes through the redaction pipeline; no raw patient names, SSNs, emails, or phone numbers survive.

Anonymization via generalization (age→age bands, zip→region prefix) and suppression (removing quasi-identifiers) is applied to structured fields, then checked against the re-identification risk test set out in GDPR Recital 26 — whether re-identification is achievable by "means reasonably likely to be used." The bar for that test is deliberately conservative: research on quasi-identifiers has shown that age, zip code, and gender alone can uniquely re-identify a large majority of the US population, which is why all three are generalized or suppressed rather than just one.

Pseudonymization replaces patient identifiers with HMAC-based artificial aliases under strict cryptographic governance. Keys are stored separately in a dedicated secrets manager — never in the same database as pseudonymized data. This enables longitudinal tracking across encounters without exposing identity.

Executes dimensional modeling transformations — building Star Schemas, Fact tables (patient encounters, device events, claims) and Dimension tables.

Automated Spark jobs compact small files and expire old snapshots on a 15-minute schedule, maintaining fast OLAP query performance.

Federated SQL query engine deployed on Kubernetes, enabling sub-second analytical queries across Gold Iceberg tables without moving data.

Open-source BI dashboarding deployed on Kubernetes, connecting to Trino for interactive dashboards — a cloud-agnostic alternative to proprietary tools.

Orchestrates Gold transformation DAGs, SLA monitoring, and automated data quality checks using Great Expectations.

Masking applies format-preserving placeholders to any residual analytics fields that need structural consistency. Insurance member IDs become ****-****-1234, partial dates preserve month granularity. Static masking enables consistent downstream lookups in star-schema rollups.

Tokenization swaps any residual identifiers with opaque, vault-backed tokens. The vault (HashiCorp Vault / AWS Secrets Manager) is the only authoritative mapping — tokens carry no structural resemblance to originals. Every detokenize operation is logged for SOC 2 accountability.

Pseudonymization via deterministic HMAC enables cross-record joins in dimensional models without exposing identity. The same patient across encounters resolves to the same pseudonym — enabling ALOS, readmission, and cost-per-case analytics — while the re-identification key stays locked in a separate secrets manager.

Extract semantic chunks from clinical text at scale. Named Entity Recognition (NER) tags clinical entities (diagnosis, medication, procedure) before embedding.

High-performance in-memory columnar format used for zero-copy transfer of feature vectors between Spark feature computation and the vector store ingestion pipeline.

Manages clinical ML features — patient risk scores, device health indices, lab trend vectors — with versioned, point-in-time-correct retrieval.

Vector databases store high-density embeddings. Hybrid Search combines dense vector similarity with BM25 sparse lexical matching, merged via Reciprocal Rank Fusion (RRF).

Deployed as Kubernetes services using vLLM or Triton Inference Server, supporting BioBERT, Clinical-BERT, or BioMedLM models.

Maps absolute relational concepts. Nodes represent patients, specimens, or facilities; edges represent deterministic medical relationships and clinical timelines.

Every ingested chunk is deterministically tagged against SNOMED-CT (clinical terms), LOINC (laboratory observations), ICD-11 (disease classification), RxNorm (pharmaceuticals), and HGNC (genomic nomenclature).

Maintains a versioned history of all feature values for offline ML training, enabling time-travel audits.

PII Redaction via production-grade NER models (BioBERT, Clinical-BERT, Microsoft Presidio) strips all direct identifiers from clinical notes, radiology reports, and pathology narratives before they enter the chunking and embedding pipeline. Regex alone cannot reliably detect names, organizations, or context-sensitive PII — NER is required.

Anonymization via generalization and suppression is applied to structured metadata accompanying embedded chunks — patient demographics, encounter dates, and location fields are generalized (age→decade bands, dates→month precision, locations→region level) before they become vector-store metadata.

Synthetic Data generation (via Faker, SDV/CTGAN, or LLM-based schema synthesis) produces statistically faithful test datasets with zero real PII for prompt engineering, RAG evaluation, embedding quality testing, and QA against the Diamond index. If the purpose is testing — don't use real data at all.

All agent tool calls, LLM responses, Chain-of-Thought (CoT) thinking traces, reflection loops, and state transitions are published as structured events to a dedicated Kafka topic.

Every Kafka agent event is micro-batched into an append-only Iceberg table. Compliance teams and clinicians can query execution trajectories with full time-travel capability.

Produces a structured MEP containing: (1) trace ID, (2) memory reads with similarity scores, (3) extracted CoT steps, (4) tool call I/O, and (5) post-generation critique results.

Complex Event Processing on the agent event stream detects anomalous behavior (excessive calls, goal drift, policy violations) and triggers automated circuit breakers.

A versioned registry of all MCP tool contracts (OpenAPI schemas). Agents reference tool versions by contract ID, ensuring deterministic behavior and auditability.

NATS JetStream is the underlying message transport — the durable queue that carries supervisor/worker handoffs and distributed task assignments between agent pods. A2A (below) sits above it as the protocol layer: agent discovery, capability negotiation, and task semantics. NATS moves the bytes; A2A defines what they mean.

Traces every agent interaction across microservices — LLM calls, tool invocations, memory accesses — exporting to Tempo for visualization in Grafana.

NeMo GuardRails runs as a sidecar container, enforcing content safety, PHI detection, prompt injection protection, and hallucination detection before output delivery.

Serves the read side of agent state for rapid HITL checkpoint querying, session replay UI, and operational dashboards.

All patient references in the Platinum audit ledger use vault-backed opaque tokens — no raw patient identifiers appear in execution traces, MEPs, or CoT logs. The token vault is isolated from the ledger itself, with every detokenize operation independently logged for SOC 2 accountability.

Differential Privacy adds mathematically calibrated noise to aggregated metrics exported from the Platinum layer (agent performance statistics, population-level outcome correlations). This provides formal privacy guarantees that satisfy GDPR Recital 26's "all means reasonably likely" standard.

Agent Gateway is the open-source gateway for AI traffic — LLM, MCP, A2A, and HTTP in one data plane. In the healthcare lakehouse, it serves as the unified connectivity layer that routes all agentic traffic through centralized policy enforcement, rate limiting, and observability. Deployed as a Kubernetes service, Agent Gateway provides a single control point for managing which agents can call which LLMs, invoke which MCP tools, and communicate with which other agents — with full audit logging to the Platinum Iceberg ledger.

MCP provides standardized tool connectivity for agents — wrapping SMART on FHIR endpoints, internal APIs, and external services into a uniform tool interface. Agent Gateway provides MCP server management and routing, enabling clinical agents to discover and invoke tools across departments through a single gateway. The Tool Contract Registry (Platinum) versions every MCP schema, ensuring deterministic behavior and auditability.

Google's open Agent-to-Agent (A2A) protocol enables structured inter-agent communication with agent discovery, capability negotiation, and task delegation. In the Platinum layer, A2A enables cross-department agent collaboration — a pharmacy agent delegating a drug-interaction check to a lab agent, or a triage agent escalating a STEMI case to the cardiology supervisor agent — with Agent Gateway managing discovery, routing, and access control.