Optimization of Scoring Rules

Scoring rules are everywhere. Any decision problem where an agent has beliefs about an unknown state and takes an action and realizes payoffs according to the action and the realized state is a scoring rule. Behavioral subjects in experiments are evaluated and rewarded according to scoring rules. Machine learning algorithms are trained and evaluated according to scoring rules.  Students' coursework is graded according to scoring rules.

This talk will introduce an objective for optimizing proper scoring rules. The objective is to maximize the increase in payoff of a forecaster who exerts a binary level of effort to refine a posterior belief from a prior belief. In this framework we characterize optimal scoring rules in simple settings, give efficient algorithms for computing optimal scoring rules in complex settings, and identify simple scoring rules that are approximately optimal. In comparison, standard scoring rules in theory and practice -- for example the quadratic rule, scoring rules for the expectation, and scoring rules for multiple tasks that are averages of single-task scoring rules -- can be very far from optimal.

The talk will briefly survey some recent applications to measuring AI reliance (Guo, Wu, Hartline, Hullman 2024), quantifying calibration error of online learning algorithms (Lu, Wu 2024), and evaluating textual peer reviews (Wu, Hartline 2024).  Joint work with Yingkai Li, Liren Shan, and Yifan Wu.