Lawma: The Power of Specialization for Legal Tasks | Social Foundations of Computation – Max Planck Institute for Intelligent Systems

Institute Homepage

Institute Homepage Sign In

Research Overview

Social Prediction

Performative Prediction: Past and Future

Difficult Lessons on Social Prediction from Wisconsin Public Schools

Allocation Requires Prediction Only if Inequality Is Low

Digital Platforms, Power and Work

Performative Power

An Engine Not a Camera: Measuring Performative Power of Online Search

Causal Inference from Competing Treatments

Contesting Algorithmic Systems

Algorithmic Collective Action in Machine Learning

Decline Now: A Combinatorial Model for Algorithmic Collective Action

Algorithmic Collective Action in Recommender Systems: Promoting Songs by Reordering Playlists

Algorithmic Fairness

Fairness and Machine Learning: Limitations and Opportunities

Unprocessing Seven Years of Algorithmic Fairness

Science of Machine Learning Benchmarks

ImageNot: A Contrast with ImageNet Preserves Model Rankings

Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget

A Theory of Dynamic Benchmarks

Evaluating Large Language Models

Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks

Evaluating Language Models as Risk Scores

Fine-tuning Large Language Models

Training on the Test Task Confounds Evaluation and Emergence

Lawma: The Power of Specialization for Legal Tasks

Social Foundations of Computation Members Publications

Lawma: The Power of Specialization for Legal Tasks

Lawma — Caption: Distribution of accuracy across all tasks for GPT4 and Lawma. Lawma achieves mean accuracy of 82%, whereas GPT-4 achieves mean accuracy 62%.

A lightly finetuned Llama-3 model vastly outperforms GPT-4 on more than 100 annotation tasks take from the U.S. Supreme Court and U.S. Appeals Courts database.

Members

Thumb ticker sm 5bc8e8c7 1c24 4a28 b899 720dca90de2f 1 201 a

Empirical Inference, Social Foundations of Computation

Ricardo Dominguez-Olmedo

Doctoral Researcher

Thumb ticker sm 20241104 hardt moritz 12 cleaned kleiner

Social Foundations of Computation

Director

Publications

Social Foundations of Computation Conference Paper Lawma: The Power of Specialization for Legal Tasks Dominguez-Olmedo, R., Nanda, V., Abebe, R., Bechtold, S., Engel, C., Frankenreiter, J., Gummadi, K., Hardt, M., Livermore, M. The Thirteenth International Conference on Learning Representations (ICLR 2025), January 2025, Accepted (Accepted) ArXiv Code BibTeX