Renal dose check: a ward chart with ten ways to be wrong
Sample benchmark task · AI evaluation
What an agent benchmark is
An agent benchmark hands an AI model a job in a real environment, here a Linux terminal with a few files in it, and grades what it produces with automated tests. The model can run commands, write code and check its own work. What it can't do is talk its way past the tests: they read the output file and nothing else.
This one gives the agent a small ward chart: 15 synthetic adults, their creatinine results and their medication orders, recorded in mixed units. For each patient it has to work out kidney function as creatinine clearance (CrCl, by the Cockcroft-Gault formula), then flag every active order whose daily dose is over the limit for that kidney function.
The instruction states the formula, the constants and the dosing rules. So the task doesn't test medical knowledge. It tests whether the agent reads the chart the way a careful pharmacist would: which unit a number is in, whether an order is still running, how often a dose is given, and which result is the latest.
The traps
Each trap changes one patient's answer, or a small group built for the same trap, and no patient carries two. So when a patient fails, it points at one mistake. Open one to see the patient's rows and both readings of them.
PT002 · creatinine unit
Creatinine in µmol/L read as mg/dL
Kidney function looks 88 times worse than it is, so a drug the patient needs gets withheld.
CrCl 32.5 → 0.4
PT005 · weight unit
Weight in lb read as kg
Clearance is inflated 2.2 times, so the patient keeps a full-function dose their kidneys can't clear.
CrCl 47.1 → 103.9
PT008 · order status
A stopped order counted as active
The flag lands on a drug nobody is taking. False alarms wear clinicians down and invite a fix to a dose that doesn't exist.
flags RX5633 wrongly
PT009 · frequency
q8h read as eight doses a day
Every eight hours is three doses, not eight. The misreading raises alarms on orders that are fine.
flags RX3742, RX4864 wrongly
PT010 · frequency
BD read as once a day
Twice daily counted once halves the daily dose on paper and hides a real overdose.
misses RX5724
PT011 · frequency
TDS read as twice a day
Three doses counted as two drops the daily total under the limit and hides an overdose.
misses RX7981
PT012 · latest lab
The last row, not the latest result
A creatinine from a kidney injury that has since recovered makes the patient look sicker than they are, and fine orders get flagged.
CrCl 100 → 42.3
PT013 · limit boundary
A dose exactly at the limit
A daily dose equal to the maximum is within it. Flagging it is a false alarm on a correct order.
flags RX9491 wrongly
Valid look-alikes
These two patients are legitimate. They're there to punish shortcuts, the blanket rules that guess a unit from the size of a number instead of reading the unit column.
PT004 · creatinine unit
A low creatinine that really is µmol/L
A frail 90-year-old with little muscle has a low creatinine. Guess the unit from the number and she looks like she's in kidney failure.
calculated CrCl 123.9 → 1.4
PT007 · weight unit
A genuine 150 kg patient
A rule that big numbers must be pounds halves the weight of exactly the patients actual-weight dosing matters most for.
CrCl 152.2 → 69.1
The evidence
A benchmark is only as good as its grading. Two things have to hold: the answer key is right, and the tests pass right answers and fail wrong ones.
The reference passes
The answer key comes from the reference solution, so the first thing to prove is that the key is right. A second implementation, written in SQL against the instruction alone and sharing no code with the reference, agrees on all 15 patients: the same CrCl at 0.1 mL/min and the same flagged orders.
$ pytest tests/test_outputs.py # solve.py's output
32 passedA grader can also be too strict. These defensible choices a careful agent might make all pass, which shows the 1% tolerance and the set comparison leave room for honest variation:
- simulator with no mistakes · passes
- µmol/L→mg/dL factor 88.42 instead of 88.4 · passes
- lb→kg as ÷2.2 · passes
- lb→kg as ÷2.2046 · passes
- round kg to 0.1 and SCr to 0.01 mg/dL before CG · passes
- choose band from CrCl rounded to 0.1 · passes
- flagged ids emitted in file order, not sorted · passes
Which mistake fails which test
One simulated agent runs with each mistake switched on alone, and the real test file grades it. Every mistake fails, and only on the patients built for it.
| Simulated mistake | Result | Tests that fail |
|---|---|---|
| Creatinine in µmol/L read as mg/dLcr_umol_as_mgdl | fails8 of 32 |
|
| Weight in lb read as kgweight_lb_as_kg | fails6 of 32 |
|
| A stopped order counted as activecount_discontinued | fails1 of 32 |
|
| q8h read as eight doses a dayfreq_qNh_as_count | fails1 of 32 |
|
| BD read as once a dayfreq_bd_as_once | fails1 of 32 |
|
| TDS read as twice a dayfreq_tds_as_twice | fails1 of 32 |
|
| The last row, not the latest resultlab_last_row_wins | fails2 of 32 |
|
| A dose exactly at the limitflag_at_limit | fails1 of 32 |
|
| A low creatinine that really is µmol/Lcr_unit_by_magnitude | fails2 of 32 |
|
| A genuine 150 kg patientweight_unit_by_magnitude | fails2 of 32 |
|
| Output file never writtenno_output_file · malformed output | failsall 32 |
|
| Output file exists but is emptyempty_output · malformed output | fails31 of 32 |
|
| One patient's line missingdrop_one_patient · malformed output | fails3 of 32 |
|
| One patient written twiceduplicate_patient · malformed output | failsall 32 |
|
| A patient id that is not in the chartextra_patient · malformed output | fails1 of 32 |
|
| CrCl written with full float precisioncrcl_unrounded · malformed output | fails1 of 32 |
|
| CrCl written as a JSON stringcrcl_as_string · malformed output | fails2 of 32 |
|
Graded with the task's own pytest suite when this page was built from commit ba39b9c (2026-11-07). The run matched the evidence committed in the repository, which reports no problems.