Frozen Rubric Reflection
Constrain the reviewer to a fixed hand-authored rubric — no invented criteria, consistent verdicts across runs, auditable checks every time.
Intent & Description
🎯 Intent
Make review criteria stable and auditable by fixing them in a rubric the reviewer cannot deviate from.
📋 Context
You use a model to review another model’s output before shipping. Without constraints, the reviewer invents different criteria each run and produces inconsistent verdicts. Auditors and downstream consumers need to know exactly what checks were performed.
💡 Solution
A fixed rubric file lists exactly the categories the reviewer may flag. The reviewer prompt includes the rubric and a JSON Schema enforcing it. Temperature is zero. Output validates against the schema; finding categories not in the rubric are rejected.
Real-world Use Case
- Review criteria should be stable across runs so verdicts can be compared.
- Auditors need an explicit list of categories the model checked.
- Reflection drift across calls is producing inconsistent reviews.
Source
📌 TL;DR
Fixed rubric. Zero temperature. Schema-enforced output. The reviewer can’t invent new criteria. Consistent, auditable verdicts every run.
Advantages
- Consistent reviews across runs and users — verdicts are comparable.
- Rubric is the single load-bearing artifact; iteration happens in one place.
Disadvantages
- Hard ceiling on what the reviewer can catch — anything not in the rubric is invisible.
- Rubric authorship is its own engineering discipline; bad rubrics produce bad reviews.