What is HealthBench Hard?
HealthBench Hard is the hardest slice of HealthBench, the health benchmark OpenAI released in May 2025. From the parent set of 5,000 realistic health conversations, it keeps the 1,000 that frontier models handled worst, so it measures exactly the part of health conversation that models have not solved. The paper is at arxiv.org/abs/2505.08775.
How the subset was built
| Parent benchmark | OpenAI's HealthBench, released May 2025: 5,000 realistic multi-turn health conversations with laypeople and professionals, each graded against its own physician-written rubric. |
|---|---|
| Rubrics | 48,562 rubric criteria across the parent set, written by 262 physicians from 60 countries. Each criterion states what an ideal response includes or avoids and carries a physician-assigned point weight; negative criteria subtract points for harmful content. |
| Selection | Every conversation was scored by five frontier models spanning providers: o3, Grok 3, Gemini 2.5 Pro, Claude 3.7 Sonnet, and Llama 4 Maverick. Conversations no model could score on were removed, about 1.5 percent, and the 1,000 with the lowest average score across the five became the Hard set. Averaging keeps the selection adversarial to the frontier generally rather than to any one vendor. |
| Difficulty | At release the best score on the subset was o3's 0.320, against about 0.60 for the same model on the full benchmark, and three then-current models scored exactly zero. The current leader on this site scores 0.428. |
| Distribution | The conversations, rubrics, and grading code ship in OpenAI's simple-evals repository, so the subset is fixed, public, and re-runnable. |
How grading works
A model grader (GPT-4.1, chosen after meta-evaluation against physician grades) judges each rubric criterion as met or not met. Criteria carry physician-assigned weights from -10 to +10, so a response loses points for harmful content regardless of what else it gets right, and a single conversation's score can go negative. The overall score is earned points divided by the maximum positive points, averaged across conversations and clipped to 0 to 1. The median conversation is graded on eleven criteria.
Why the scores are so low
The subset is defined by failure: every conversation in it is one the frontier could not handle well at release. The best launch-era score was o3's 0.320, the current leader sits at 0.428, and the bottom of the table runs close to zero. A low absolute number here is not a defect in the evaluation; it is the reason the benchmark still has headroom while scores on the parent set run far higher.
The HealthBench family
| HealthBench May 2025 | The parent set: 5,000 multi-turn health conversations spanning laypeople and professionals, scored against 48,562 rubric criteria that 262 physicians in 60 countries wrote. |
|---|---|
| HealthBench Hard May 2025 | The 1,000 conversations frontier models found hardest. The benchmark this site tracks. |
| HealthBench Consensus May 2025 | A near-zero-error baseline: 3,671 examples judged on 34 behavioral criteria that passed physician-consensus validation. |
| HealthBench Professional April 2026 | OpenAI's 2026 follow-up: 525 physician-authored tasks focused on what clinicians bring to a model at work. |
The current ranking of 9 models is on the leaderboard. How the numbers on this site are produced is on the methodology page.