Multimodal Input
Combines different data types (images, text, tabular) into a unified model
Intent & Description
🎯 Intent
Handle prediction problems that mix genuinely different data types where useful signal is spread across all modalities, not contained in any single one.
📋 Context
Real-world problems often involve images, free text, structured/tabular fields, and categorical metadata simultaneously. The predictive signal is distributed across these different modalities.
💡 Solution
Each modality gets its own suitable representation (embedding for text/categorical, convolutional or pretrained backbone for images, normalization for numeric fields). These representations are then fused at some point: early (concatenated near input), late (combined near output), or hybrid. Also handles metadata about the data itself (device, process) as additional input features.
Real-world Use Case
- Product recommendation using images, descriptions, and metadata
- Medical diagnosis combining imaging, clinical notes, and lab results
- Content moderation using text, images, and user metadata
- Any task where predictive signal spans multiple data types
Source
📌 TL;DR
Combine different data types with appropriate encodings and fusion strategies to leverage predictive signal across multiple modalities
Advantages
- Leverages signal from multiple data types
- More accurate than single-modality approaches
- Flexible fusion strategies (early, late, or hybrid)
- Can handle missing modalities with learned defaults
Disadvantages
- Adds architectural complexity
- Needs more data to train well
- Harder to determine which modality drives predictions
- Risk of one modality numerically dominating others