Hallucinated Tools
Trusting the model to only call tools you gave it — then watching it invoke functions that don't exist.
Intent & Description
🎯 Intent
Assuming the model will only call tools it was shown in the prompt — without validating the name at dispatch time.
📋 Context
An agent is configured with a registered tool palette. The host accepts whatever name the model emits and dispatches it without checking against the palette. The model confidently calls things that don’’t exist — and the host silently fails or mis-routes to a similarly named tool that does.
💡 Solution
Validate every tool call against the registered palette before dispatch. Reject unknown names with a typed error the agent loop can react to. See tool-use, structured-output.
Real-world Use Case
- Never use this; treat any model-emitted tool name as untrusted input.
- Validate every tool call against the registered tool palette before dispatch.
- Reject unknown tool names with a typed error the agent loop can handle.
Source
📌 TL;DR
Validate every model-emitted tool name against your registered palette before dispatching — treat it as untrusted input.
Disadvantages
- Silent failures when unrecognized tool names just disappear
- Mis-dispatches to similarly named tools that do exist — wrong actions run silently
- Debugging is confusing — the model looks correct, the host is at fault