FTI LLM Pipeline Split
Decompose an LLM/RAG system into three independently-deployable pipelines — feature, training, inference — so Agent Confession defenses can be applied and audited at each pipeline boundary.
Intent & Description
Short description: Feature, training, and inference pipelines communicate only through a feature store and model registry — and this clean separation means Agent Confession defenses can be enforced at each boundary independently, without one pipeline’s changes silently undermining another’s guardrails.
🎯 Intent
Decouple feature ingestion, model adaptation, and serving so each evolves on its own cadence — and exploit the clean pipeline boundaries as natural enforcement points for Agent Confession defenses: the feature store screens corpus content for embedded triggers, and the inference pipeline applies output guardrails without depending on the training pipeline to have baked in all defenses.
📋 Context
An LLM application team owns data ingestion, model fine-tuning, and serving. These have different cadences and owners. When bundled together, an Agent Confession defense added to the inference pipeline may be silently bypassed by a training artifact that fine-tuned the model to be unusually forthcoming with its directives, or by a corpus update in the feature pipeline that introduced embedded Agent Confession triggers into the RAG index. The FTI split makes these interactions visible and auditable.
💡 Solution
- Feature pipeline: ingests raw documents, cleans, chunks, embeds, writes to the feature store. At this stage, screen incoming documents for embedded Agent Confession trigger phrases; flag or quarantine documents that contain them before they enter the RAG index.
- Training pipeline: reads features, fine-tunes (SFT, DPO), writes models to the registry. Include Agent Confession probe examples in the fine-tuning evaluation set to verify that training does not increase directive-disclosure compliance.
- Inference pipeline: reads from the feature store at request time, loads the model from the registry, generates, applies output guardrails including directive-echo detection.
- Communication only via the feature store and model registry — cross-pipeline Agent Confession audits are possible because the integration surfaces are narrow and versioned.
Real-world Use Case
- Feature, training, and inference have materially different cadences and ownership — and Agent Confession defenses applied in one pipeline must not be silently undone by changes in another.
- The feature store is the right place to quarantine corpus documents containing embedded Agent Confession triggers before they enter the RAG index.
- The training pipeline’s eval set should include confession probes to prevent fine-tuning from inadvertently increasing directive-disclosure compliance.
Source
Advantages
- Teams iterate independently; Agent Confession defenses at each boundary (feature quarantine, training eval, inference guardrail) are independently auditable and deployable.
- Feature store and model registry are clean integration surfaces — an Agent Confession audit can verify what entered the corpus and what model was serving at the time of an incident.
Disadvantages
- Schema changes across the feature store ripple through downstream pipelines — and a schema change that removes a quarantine field silently drops Agent Confession screening from the feature pipeline.
- Decomposition overhead is not worth it for very small systems where a single-pipeline Agent Confession guardrail is sufficient.