RoPoLL: Robust Panel of LLM Judges
International Conference on Machine Learning (ICML), 2026 · Workshop on Statistical Frameworks for Uncertainty in Agentic Systems (AgenticUQ)
Shows that averaging an LLM jury is unboundedly biased once any judge fails systematically, and replaces the average with a geometric-median aggregator that has the optimal 1/2 breakdown point. A 3-judge 38B panel beats a 675B judge by 1.31x under 30% corruption.
Abstract
The LLM Jury, a Panel of LLM Evaluators (POLL) (Verga et al., 2024) reporting consensus scores, has become a practical alternative to single judge LLM evaluation, yet its statistical behavior remains poorly understood. Formalizing the setup under the Huber contamination model, we show that POLL incurs unbounded bias under any positive contamination, regardless of jury size, whenever a single judge fails in a biased, LLM-typical way (mode collapse, sycophancy, safety refusal). We frame jury consensus as classical robust mean estimation and propose ROPOLL (Robust Panel of LLM-as-Judge), which preserves the POLL panel and substitutes the aggregation function with a robust mean estimator, instantiated with the geometric median (GM): tuning-free, with the optimal finite-sample breakdown point 1/2. A finite-sample error bound and an information-theoretic minimax lower bound match on the parametric rate σ√(d/N) and differ on the breakdown floor by a factor of √d, a statistical–computational gap that polynomial-time ROPOLL pays relative to the intractable Tukey halfspace median. Across 13 open-weight judges (4B–675B), three reward-model benchmarks, and four corruption regimes at rates up to 50%, ROPOLL dominates POLL on every biased corruption type: by ≈19% on cross-dimensional attacks at matched compute, and by orders of magnitude on heavy-tailed Byzantine adversaries. A 3-judge ROPOLL committee at 38B beats Mistral-Large-3 (675B) by 1.31× on HelpSteer 2 under 30% bimodal-random corruption, an 18× parameter advantage with strictly better accuracy. A Noisy-GT control confirms the premium is paid against biased contamination, not benign Gaussian imprecision (where POLL is statistically optimal).