Multi-Principal Welfare Aggregation
When an agent serves multiple principals with conflicting preferences, declare the aggregation rule explicitly — make it a config choice, not an invisible prompt accident.
Intent & Description
🎯 Intent
When an agent serves multiple humans with conflicting preferences, declare the aggregation rule explicitly rather than letting it be implicit in the prompt or fine-tune.
📋 Context
An agent serves a team, household, customer cohort, or entire user base. Principals have conflicting preferences — different summary styles, different escalation defaults, different behaviors in a shared workspace. Some preferences are zero-sum. Without an explicit rule, the agent’s behavior reflects an implicit choice that no one made deliberately.
💡 Solution
When the agent’s action space affects multiple principals, route the decision through an explicit aggregation function. Options: sum-of-utilities (utilitarian); weighted welfare (declared per-principal weights); collegial mechanism (each principal must be receiving “enough” reward for their preferences to count); role-priority (some principals have veto). Surface the active rule in traces and documentation. Make it a configuration change, not a prompt change.
Real-world Use Case
- An agent serves multiple principals whose preferences can conflict.
- Actions are zero-sum or rivalrous across principals.
- Operators or users need to understand and adjust how preference aggregation works.
Source
📌 TL;DR
When your agent serves multiple principals with conflicting wants, make the “whose preferences win?” rule explicit in config — don’t let it be an accident of prompt wording.
Advantages
- Aggregation becomes a deliberate policy, not an implicit accident.
- Disputes over agent behavior have a vocabulary — stakeholders argue about the rule, not the vibe.
- Operators can switch aggregation rules without retraining or re-prompting.
Disadvantages
- Explicit rules invite strategic gaming — principals misreport preferences to manipulate outcomes (Gibbard’s theorem).
- Some rules require principal-weight assignment that itself becomes contested.
- Computational cost of welfare aggregation scales with the principal count.