Bench

FreshBench asks coding assistants about changes in the versions a project installed, and a judge grades each answer against a key checked against primary sources. Every count comes with its 95% Wilson interval, and each part of this page says what it counts.

What each model knows, lesson by lesson

One square per lesson (57). Filled: the model answered the change right both times, with no tools. Empty: it didn't. Dotted: not measured yet.

Measured on Oct 11, 2026, 2 days ago. The run's questions, answers, grades, and logs

Claude Haiku 5.5

Claude Sonnet 5.5

Claude Opus 5.5

In a project, with and without ludion sync

FreshBench B questions (45, one per change), asked to Claude Code in a project with the new versions installed. Each cell: answers graded correct out of questions. With sync, the project got the 31 seed lessons frozen at commit 7d61ed1.

Measured on Oct 11, 2026, 2 days ago. The run's questions, answers, grades, and logs

Claude Opus 5.5

Without ludion sync24/45

With ludion sync36/45

+27points

Not measured in a project: Claude Haiku 5.5, Claude Sonnet 5.5.

By subject
ModelConditionagentsvitestwranglerAll
Claude Opus 5.5Claude Code's read-only toolsweb search onludion sync off11/156/157/1524/45
Claude Opus 5.5Claude Code's read-only toolsweb search onludion sync on14/1511/1511/1536/45
Claude Haiku 5.5Not measured
Claude Sonnet 5.5Not measured

Knowledge dates

Each dot is a lesson, placed on the day npm published the first version it is about. Filled: the model knows it. Hollow: it doesn't. Dotted: not measured. The gray band is a model's reliable knowledge cutoff, where its maker publishes one. A break in the axis is a stretch with nothing in it, shortened.

Publish dates from the npm registry · Cutoffs as Anthropic's model overview publishes them · 1 lesson isn't about an npm package and has no date here.

Every lesson

  1. agents/>=0.20.0behaviorsilentDrafted by an agentagents >=0.20.0 changed what `createMcpHandler` does.
  2. agents/>=0.20.0behaviorsilentDrafted by an agentagents >=0.20.0 changed what `MCPClientConnection` does.
  3. agents/>=0.21.0behaviorsilentDrafted by an agentagents >=0.21.0 changed what `useAgentChat` does.
  4. agents/>=0.22.0defaultsilentDrafted by an agentagents >=0.22.0 changed the default of `throttle`.
  5. agents/>=0.22.0addedDrafted by an agentagents >=0.22.0 added `Scheduler`.
  6. agents/>=0.22.0behaviorsilentDrafted by an agentagents >=0.22.0 changed what `Agent` does.
  7. agents/>=0.23.0deprecatedsilentDrafted by an agentagents >=0.23.0 deprecated `subAgent()`; use `this.dynamicAgents.get()` instead.
  8. agents/>=0.23.0behaviorsilentDrafted by an agentagents >=0.23.0 changed what `browser_execute` does.
  9. agents/>=0.24.0behaviorsilentDrafted by an agentagents >=0.24.0 changed what `queue()` does.
  10. agents/>=0.24.0addedDrafted by an agentagents >=0.24.0 added `transport`.
  11. agents/>=0.25.0deprecatedsilentDrafted by an agentagents >=0.25.0 deprecated `CdpSession`; use `CdpConnection` instead.
  12. agents/>=0.25.0behaviorsilentDrafted by an agentagents >=0.25.0 changed what `loadCdpSpec` does.
  13. agents/>=0.27.0addedDrafted by an agentagents >=0.27.0 added `npx agents tui`.
  14. agents/>=0.27.0behaviorsilentDrafted by an agentagents >=0.27.0 changed what `browserTool` does.
  15. agents/>=0.27.0behaviorsilentDrafted by an agentagents >=0.27.0 changed what `MCPClientManager.removeServer` does.
  16. python/>=3.12Python 3.12 removed the distutils module from the standard library (PEP 632); use setuptools or packaging instead.
  17. vitest/>=4.0.0behaviorsilentDrafted by an agentvitest >=4.0.0 changed what `vi.restoreAllMocks` does.
  18. vitest/>=5.0.0behaviorsilentDrafted by an agentvitest >=5.0.0 changed what `-t` does.
  19. vitest/>=5.0.0defaultsilentDrafted by an agentvitest >=5.0.0 changed the default of `extends`.
  20. vitest/>=5.0.0addedDrafted by an agentvitest >=5.0.0 added `vi.when`.
  21. vitest/>=5.0.0addedDrafted by an agentvitest >=5.0.0 added `-p`.
  22. vitest/>=5.0.0behaviorsilentDrafted by an agentvitest >=5.0.0 changed what `vitest list` does.
  23. vitest/>=5.0.0behaviorDrafted by an agentvitest >=5.0.0 changed what `toHaveTextContent` does.
  24. vitest/>=5.0.0defaultsilentDrafted by an agentvitest >=5.0.0 changed the default of `--merge-reports`.
  25. vitest/>=5.0.0addedDrafted by an agentvitest >=5.0.0 added `--repeats`.
  26. vitest/>=5.0.0behaviorsilentDrafted by an agentvitest >=5.0.0 changed what `vitest.config` does.
  27. vitest/>=5.0.0defaultsilentDrafted by an agentvitest >=5.0.0 changed the default of `clearMocks`.
  28. vitest/>=5.0.0behaviorsilentDrafted by an agentvitest >=5.0.0 changed what `vi.fn` does.
  29. vitest/>=5.0.0behaviorsilentDrafted by an agentvitest >=5.0.0 changed what `Duration` does.
  30. vitest/>=5.0.0renamedsilentDrafted by an agentvitest >=5.0.0 renamed `experimental_parseSpecifications` to `parseSpecifications`.
  31. vitest/>=5.0.0behaviorsilentDrafted by an agentvitest >=5.0.0 changed what `attachmentsDir` does.
  32. vitest/>=5.0.0behaviorsilentDrafted by an agentvitest >=5.0.0 changed what `projects` does.
  33. vitest/>=5.0.0defaultDrafted by an agentvitest >=5.0.0 changed the default of `browser.locators.exact`.
  34. vitest/>=5.0.0defaultsilentDrafted by an agentvitest >=5.0.0 changed the default of `HTML reporter`.
  35. wrangler/>=4.113.0behaviorsilentDrafted by an agentwrangler >=4.113.0 changed what `wrangler dev` does.
  36. wrangler/>=4.113.0behaviorsilentDrafted by an agentwrangler >=4.113.0 changed what `deploy` does.
  37. wrangler/>=4.116.0behaviorsilentDrafted by an agentwrangler >=4.116.0 changed what `wrangler check startup` does.
  38. wrangler/>=4.116.0behaviorsilentDrafted by an agentwrangler >=4.116.0 changed what `deploy` does.
  39. wrangler/>=4.117.0renamedsilentDrafted by an agentwrangler >=4.117.0 renamed `/cdn-cgi/handler/scheduled` to `/cdn-cgi/local/scheduled`.
  40. wrangler/>=4.118.0defaultsilentDrafted by an agentwrangler >=4.118.0 changed the default of `X_LOCAL_OBSERVABILITY`.
  41. wrangler/>=4.121.0addedDrafted by an agentwrangler >=4.121.0 added `--jurisdiction us`.
  42. wrangler/>=4.121.0behaviorsilentDrafted by an agentwrangler >=4.121.0 changed what `DO_NOT_TRACK` does.
  43. wrangler/>=4.121.0behaviorsilentDrafted by an agentwrangler >=4.121.0 changed what `wrangler preview` does.
  44. wrangler/>=4.122.0defaultsilentDrafted by an agentwrangler >=4.122.0 changed the default of `nodejs_compat`.
  45. wrangler/>=4.124.0behaviorsilentDrafted by an agentwrangler >=4.124.0 changed what `--latest` does.
  46. wrangler/>=4.125.0addedDrafted by an agentwrangler >=4.125.0 added `connect`.
  47. wrangler/>=4.126.0behaviorsilentDrafted by an agentwrangler >=4.126.0 changed what `.cloudflare/output/v0/config.json` does.
  48. wrangler/>=4.127.0addedDrafted by an agentwrangler >=4.127.0 added `workflows[].concurrency.limit`.
  49. wrangler/>=4.129.1behaviorsilentDrafted by an agentwrangler >=4.129.1 changed what `bundle size reporter` does.
  50. wrangler/>=4.131.1behaviorsilentDrafted by an agentwrangler >=4.131.1 changed what `deployed trigger output` does.
  51. wrangler/>=4.136.0behaviorsilentDrafted by an agentwrangler >=4.136.0 changed what `wrangler preview --json` does.
  52. wrangler/>=4.141.0defaultsilentDrafted by an agentwrangler >=4.141.0 changed the default of `durable_objects.code_update_strategy`.
  53. wrangler/>=4.145.0renamedDrafted by an agentwrangler >=4.145.0 renamed `WRANGLER_R2_SQL_AUTH_TOKEN` to `WRANGLER_BASIN_SQL_AUTH_TOKEN`.
  54. wrangler/>=4.145.0renamedsilentDrafted by an agentwrangler >=4.145.0 renamed `wrangler r2 sql` to `wrangler basin sql`.
  55. wrangler/>=4.148.0behaviorsilentDrafted by an agentwrangler >=4.148.0 changed what `wrangler versions secret bulk` does.
  56. wrangler/>=4.148.0behaviorsilentDrafted by an agentwrangler >=4.148.0 changed what `--temporary` does.
  57. wrangler/>=4.148.0behaviorsilentDrafted by an agentwrangler >=4.148.0 changed what `wrangler types` does.
How this was measured, and how to run it again

How it was measuredThis table as JSON

Run it again

node tools/bench/src/projects.ts <projects> && [LUDION_SITE=<site built at 7d61ed1, served locally>] node tools/bench/src/freshbench.ts run docs/bench/2026-10-11-done3/questions.json <out> B-project[-sync] --set B --condition project --projects <projects> --model claude-opus-5-5 [--sync <cli built at 7d61ed1> --ludion-home <empty dir>] && node tools/bench/src/freshbench.ts judge docs/bench/2026-10-11-done3/questions.json <out> B-project[-sync]

The parts in brackets are the run with sync. Its CLI and lessons are the ones frozen for the run, served from the repository at that commit, not today's ludion.ai. Setup: docs/bench/2026-10-11-done3/README.md.

Without tools

FreshBench A questions for the 29 measured lessons, each asked twice with no tools. A cell counts the lessons whose change the model answered right both times; ludion sync leaves those lessons out for that model.

ModelConditionagentsvitestwranglerAll
Claude Haiku 5.5No toolsweb search offludion sync off1/91/90/112/29
Claude Sonnet 5.5No toolsweb search offludion sync off1/90/90/111/29
Claude Opus 5.5No toolsweb search offludion sync off1/90/90/111/29