BenchPress - Causal Reasoning

About the causal benchmark

A standalone benchmark for causal-inference reasoning. 100 multiple-choice questions arranged as 20 concept bundles × 5 variants. Each bundle covers one causal-inference pitfall (e.g. confounding, M-bias, Simpson's paradox); the 5 variants stress-test the same concept different ways:

• Base. Narrative scenario combining 2 to 3 interacting causal issues.
• Trap. Looks like the base concept applies but the obvious answer is wrong.
• Transfer. Same reasoning translated into a formal DAG with short elimination-style options.
• Numeric. Multi-step calculation with tables and conditional probabilities.
• Analyst. Two analysts debate the scenario; pick the most accurate assessment.

Scoring is purely deterministic (multiple choice, no LLM judges needed). Accuracy = correct ÷ valid responses. Errors are API failures; Invalid is the model returning an empty or unextractable response. Both are reported separately.

Top performer per variant

Each variant tests the same causal-inference concept from a different angle. Best score on each is shown alongside what the variant tests.

Per-variant accuracy (Top 10)

For the top 10 overall models, accuracy on each of the five variants. Bars that drop sharply on Trap or Transfer reveal where a model relies on pattern-matching over structural reasoning.

Variant heatmap (Top 15)

Composite shading per variant for the top 15 overall models. Green ≥80, yellow ≥60, red below.

Accuracy leaderboard

All models with causal data. Score is correct out of 100. Errors and Invalid columns separate API failures from empty/unextractable responses.