Bridged Schema
Forward-fits old-format data into new schemas to preserve historical training data
Intent & Description
🎯 Intent
Handle data schema evolution without discarding useful historical data that predates schema changes, which would otherwise waste training examples.
📋 Context
Data schemas evolve—new fields get added, categorical fields gain new values—but historical data collected before the change doesn’t have the new field. Discarding all older data wastes useful training examples.
💡 Solution
Forward-fit old-format data into the new schema by filling in reasonable placeholders for missing fields (constant defaults, statistically informed values sampled from new field’s distribution, or model-based imputation). Combine bridged historical data with genuine new-schema data, weighting newer data more heavily during transition.
Real-world Use Case
- Schema migrations in production ML systems
- Adding new features to existing models
- Evolving data pipelines without losing history
- Any situation where schema changes would otherwise force data loss
Source
📌 TL;DR
Forward-fit old-format data into new schemas using reasonable imputation strategies to preserve historical training data through schema evolution
Advantages
- Preserves useful historical training data through schema changes
- Enables smooth transitions during schema evolution
- Avoids wasting data collected before schema changes
- Allows continuous training through migrations
Disadvantages
- Imputed values aren’t real data and can introduce bias
- Requires careful bridging strategy selection
- Needs weighting to prevent imputed values from dominating
- Adds complexity to data pipeline