The ability to declassify private thoughts on demand is convenient, allowing for targeted analysis without cluttering the public view with sealed reasoning.
Questions about DeceptionBench
What is DeceptionBench?
DeceptionBench is a benchmark for evaluating large language models' abilities to deceive and detect deception within a minimal social-deduction game. It runs cross-play matches across an M×M matrix of models, where each agent produces a private thought and a public statement each turn. A separate monitor model reviews the private reasoning to assess intent and determine whether hidden reasoning leaks the deception. The project publishes results and methodology in a preprint-style dossier, tracking metrics such as monitor AUC and total compute logged across recorded games.
How does the benchmark separate deception and detection capabilities?
The benchmark operates by having every agent write two things each turn—a private thought and a public statement—inside a minimal social-deduction game. The science lives in the gap between them, measuring how a model's hidden reasoning either gives away the lie or covers it. This approach allows for a controlled deception lab where a lone impostor knows the truth while a panel of villagers must reason it out from public statements and votes across a few day/night rounds.
What specific methodology does the benchmark use to rank models?
The system uses cross-play rather than self-play to disentangle the two skills, putting model A in the impostor's chair against a panel of model B. By filling the whole M×M grid, reading across a row ranks a model's deception, while reading down a column ranks its detection. This method ensures that the row scores A's deception and the column scores B's detection cleanly.
How does the monitor model evaluate the private reasoning?
Each agent emits a hidden thought plus a public line during the game, creating a gap that the study measures. A monitor model tries to catch the impostor from the private reasoning alone, quantifying how much intent leaks. The public statement is what all other players ever hear, while the private reasoning is sealed and never shown to other players.
What is the publication history and citation information for this work?
The benchmark is part of a larger study called Among LLMs, which is a cross-play study of deception and detection in language models. The study is published on Zenodo with a DOI of 10.5281/zenodo.21209429. The work is attributed to Eburu, Evans, and was published in 2026.
What are the key features of the benchmark's design?
The benchmark is described as a declassified benchmark of machine deception and detection. It is a controlled deception lab where a lone impostor knows the truth and a panel of villagers must reason it out. The study is designed to be short with one clean win signal.
What is the official website of DeceptionBench?
The official website of DeceptionBench is deceptionbench.xyz.
How is DeceptionBench rated on Trustburn?
On Trustburn, DeceptionBench has a rating of 4.2 out of 5 based on 5 reviews.
AI-generated from reviews on this page. Verify anything important with the company directly.
The benchmark structure is logically sound, using a social-deduction game to isolate deception from detection. The cross-play methodology effectively disentangles these capabilities, which is a notable improvement over traditional self-play evaluations. The monitor model’s ability to score intent from private reasoning adds a layer of depth to the analysis. However, the fact that private thoughts are sealed until declassified introduces a layer of opacity. For those needing to verify the exact reasoning behind a specific turn, the need to drag the cursor suggests that full transparency is not automatic. This might slow down the iterative process of analyzing specific failure modes compared to fully open datasets.
The dashboard displays compute logs and game counts efficiently. Having access to the full interrogation logs and live heatmap makes tracking model behavior straightforward for ongoing research.
From the available documentation, DeceptionBench presents a compelling framework for assessing machine deception through its cross-play benchmark structure. The distinction between the impostor’s private reasoning and public statements offers a clear metric for monitorability, which is increasingly critical as models gain agency. The matrix layout, displaying impostor win rates against various villager panels, provides a granular view of how different models perform in deceptive versus detection roles. This separation of skills is a significant advantage over self-play benchmarks that often conflate the two. While the snapshot indicates robust data logging, the requirement to declassify private thoughts suggests that real-time transparency might vary, which could be a consideration for researchers needing immediate access to the full reasoning traces during live evaluations. Overall, the scientific approach seems well-calibrated.
The cross-play design effectively separates deception from detection, though full confidence intervals remain locked behind the main matrix view.
News
No news yet. This is where the business shares its own updates and press.
Photos
No photos yet. Photos shared by reviewers will show up here.
Widgets
Get the widget for your site →About DeceptionBench
DeceptionBench is a benchmark for evaluating large language models' abilities to deceive and detect deception within a minimal social-deduction game. It runs cross-play matches across an M×M matrix of models, where each agent produces a private thought and a public statement each turn. A separate monitor model reviews the private reasoning to assess intent and determine whether hidden reasoning leaks the deception. The project publishes results and methodology in a preprint-style dossier, tracking metrics such as monitor AUC and total compute logged across recorded games.
- Website
- deceptionbench.xyz