PowerWorld Bench
30 real PowerWorld study requests · a fresh Claude every time · graded against a key written before the run
Claude Opus 5.5 · the same 30 tasks for every setup · whiskers show the 95% interval
| Rank | Setup | Solved | Solved, as a bar | Min / task | Tokens / task | Cost / task |
|---|
Earlier tests
Before the 30 tasks: 23 PowerWorld questions, answered with the kit and by the same model searching the web, marked blind. Then the case-auditor agent's own blind-judged test.
| Test | With the kit | Without |
|---|---|---|
| Calls that report success and return something wrong | 15 of 16 | 7 of 16web search |
| Looking up a command name | 7 of 7 | 6 of 7web search |
| All 23 questions | 22 of 23 | 13 of 23web search |
| The case-auditor agent described the case correctlyblind judge; build after v0.4.0, Sonnet | 21 of 24 | 10 of 24plain model |
Web search finds what a command does. It rarely finds that the call reports success and does nothing: on those 16 questions the web got 7 and the kit got 15. The kit also needed 3.2x fewer searches per right answer.
The case-auditor agent and a plain model got the same audit findings for 8 cases, 3 runs each. A blind judge found the agent described the case correctly in 21 of 24 runs, the plain model in 10. The plain model hardly ever invents anything: on one combined check (no invented cost data, only the three severity labels, no pasted raw findings) it passed 50 of 51, the agent 51 of 51. The difference is in the call: what actually stops the study.
Why the agent searches first: on 16 test questions, searching the pages found the right one 10 times, against 8 for the routing table and 7 for starting at the index. The full search rules
For exact argument syntax, open PowerWorld's manual instead. The kit deliberately does not publish it, so it sometimes names the right command without a signature. Every question, both verdicts, and what each setup spent
Wanna try it yourself?
Install HiveMind >_One sentence pasted into Claude Code or Codex, then try it on your own case.
The 30 tasks
Each one is a request typed into a fresh Claude with HiveMind installed, with the case file on disk and nothing else. Click a task for the request as typed and the answer.