Methodology
No number on this site is taken from a vendor's marketing page: the scores come from running the published benchmark itself. Because every model answers the same conversation set under one grader configuration, two rows can be read against each other directly. This page covers the run protocol; what the benchmark measures and how the subset was chosen is on the benchmark page.
Run protocol
| Conversation set | The released 1,000-conversation HealthBench Hard set, unmodified. How the subset was selected is on the benchmark page. |
|---|---|
| Candidates | Each model is run under default sampling, one response per conversation, no tools, no retrieval, no system-prompt scaffolding. API models run through their vendor's public endpoint; open-weights models run from the released weights. |
| Grader | The benchmark's default grader configuration: GPT-4.1. Criterion judgments are binary. |
| Score | Rubric points earned over maximum positive points, clipped to the 0 to 1 range and shown to three decimals. |
| Pricing and specs | The context and price columns are read from vendor documentation and current price sheets on each update date; the base rates shown were checked on August 16, 2026. |
Comparability
Rubric scores are model-graded, so they are comparable only within an identical grader configuration. Published HealthBench Hard numbers vary between sources because grader versions, sampling settings, and product scaffolding vary; vendor system-card results in particular are graded under configurations a raw run does not reproduce. Comparing a score from this table against a score graded elsewhere is not meaningful; comparing two rows of this table is.
Limitations
One response per conversation measures typical output, not best-of-k. The grader is a model too; sticking to the benchmark's default grader makes results reproducible without making them infallible, since grader bias remains an open problem for rubric evaluations. The subset itself was frozen against the May 2025 frontier, so nothing released since had any hand in the selection. This site does not yet break scores out by theme. Above all, a benchmark number certifies nothing clinically: it records rubric adherence across 1,000 hard conversations and stops there.
Updates
A new frontier release, or a material update to an existing one, triggers a re-run. The updates page dates every change down to price corrections, and the data page carries each release of the results in full. As of August 16, 2026 the table covers 9 models.