Back to Catalog
Agentic AI
Planning & Control Flow
MapReduce for Agents
Split an oversize task into independent chunks, process each in parallel, then aggregate.
Intent & Description
🎯 Intent
Split an oversize task into independent chunks, process each in parallel, then aggregate.
📋 Context
A team needs to apply a language model to an input that is too large for a single call — twelve hundred pages of vendor contracts, a million-row table, hundreds of documents to summarise — or to a task that decomposes naturally into independent pieces (per row, per document, per section). Per-piece work is short; what is hard is the scale.
💡 Solution
- Map: split the input into chunks; process each independently with a per-chunk LLM call. - Reduce: aggregate intermediate answers via a structured information protocol that surfaces cross-chunk dependencies, plus a confidence-calibration step to resolve conflicting answers between chunks.
Real-world Use Case
- Input is too large for any single context window to handle well.
- Chunks are mostly independent and a structured reducer can resolve cross-chunk dependencies.
- A confidence-calibration step can reconcile conflicting per-chunk answers.
Source
Advantages
- Scales to inputs orders of magnitude larger than the context window.
- Embarrassingly parallel; latency scales with longest chunk, not total input size.
Disadvantages
- Cross-chunk dependencies must be modelled explicitly in the reduce step.
- Reduce stage can become the new bottleneck at very high chunk counts.