vitest-5-0-0-24-b

Question

How does vitest list find tests without executing them?

Answer key

A correct answer says

`vitest list` collects tests by static parsing by default instead of running files.

The stale answer

`vitest list` executes test files to collect test names.

The change's lesson

Answer

In the vitest version installed here (5.0.3), `vitest list` collects tests by **static parsing**. It reads each test file's source and picks out the `describe`/`test`/`it` calls, without importing or running the file. I confirmed this in your `node_modules`: the CLI defines a `--static-parse` option that defaults to `true` (`cac.DfDGTQ9W.js:1340`). When it's on, `collect()` calls `parseSpecifications(files, …)` instead of running the files (`index.DpLw24bj.js:20943`). You can control how many files are parsed at once with `--static-parse-concurrency`.

Because nothing runs, tests created at runtime (for example, inside loops or with computed names) may be missed or shown differently. Pass `--no-static-parse` to go back to the older behaviour, where the files are actually run to collect the tests.

Answering cost $0.11.

Grade

correct

The judge's reason: It states that `vitest list` uses static parsing by default without running files, and mentions running files only as the opt-out older behavior.

The judge's log

Session log

What the agent wrote, the tools it called, and what they returned, from docs/bench/2026-10-11-done3/runs/B-project-sync/vitest-5-0-0-24-b.jsonl. Thinking and the session's settings are left out. The raw log

1. Tool call: Grep

   {
     "pattern": "static-parse|staticParse",
     "path": "C:\\Users\\haya0\\AppData\\Local\\Temp\\ludion-bench-Bq85BG\\work\\node_modules\\vitest",
     "output_mode": "files_with_matches",
     "head_limit": 5
   }

2. Tool output

   Found 3 files
   node_modules\vitest\dist\chunks\plugin.d.BsjqSb4-.d.ts
   node_modules\vitest\dist\chunks\index.DpLw24bj.js
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js

3. Tool call: Grep

   {
     "pattern": "static-parse|staticParse",
     "path": "C:\\Users\\haya0\\AppData\\Local\\Temp\\ludion-bench-Bq85BG\\work\\node_modules\\vitest\\dist\\chunks",
     "output_mode": "content",
     "-C": 2,
     "head_limit": 30
   }

4. Tool output

   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1323-	provide: null,
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1324-	filesOnly: null,
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js:1325:	staticParse: null,
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js:1326:	staticParseConcurrency: null,
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1327-	projects: null,
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1328-	watchTriggerPatterns: null,
   --
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1338-	},
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1339-	filesOnly: { description: "Print only test files with out the test cases" },
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js:1340:	staticParse: {
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1341-		description: "Parse files statically instead of running them to collect tests (default: true)",
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1342-		default: true
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1343-	},
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js:1344:	staticParseConcurrency: {
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1345-		description: "How many tests to process at the same time (default: os.availableParallelism())",
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-1346-		argument: "<limit>"
   --
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-2411-		if (!options.filesOnly) {
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-2412-			const { testModules: tests, unhandledErrors: errors } = await ctx.collect(cliFilters.map(normalize), {
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js:2413:				staticParse: options.staticParse,
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js:2414:				staticParseConcurrency: options.staticParseConcurrency
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-2415-			});
   node_modules\vitest\dist\chunks\cac.DfDGTQ9W.js-2416-			if (errors.length) {
   --
   node_modules\vitest\dist\chunks\index.DpLw24bj.js-20941-				unhandledErrors: []
   node_modules\vitest\dist\chunks\index.DpLw24bj.js-20942-			};
   node_modules\vitest\dist\chunks\index.DpLw24bj.js:20943:			if (options.staticParse !== false) {
   node_modules\vitest\dist\chunks\index.DpLw24bj.js:20944:				const testModules = await this.parseSpecifications(files, { concurrency: options.staticParseConcurrency });
   node_modules\vitest\dist\chunks\index.DpLw24bj.js-20945-				if (hasFailed(testModules.map((testModule) => testModule.task))) process.exitCode = 1;
   node_modules\vitest\dist\chunks\index.DpLw24bj.js-20946-				return {
   
   [Showing results with pagination = limit: 30]

5. Agent

   In the vitest version installed here (5.0.3), `vitest list` collects tests by **static parsing**. It reads each test file's source and picks out the `describe`/`test`/`it` calls, without importing or running the file. I confirmed this in your `node_modules`: the CLI defines a `--static-parse` option that defaults to `true` (`cac.DfDGTQ9W.js:1340`). When it's on, `collect()` calls `parseSpecifications(files, …)` instead of running the files (`index.DpLw24bj.js:20943`). You can control how many files are parsed at once with `--static-parse-concurrency`.
   
   Because nothing runs, tests created at runtime (for example, inside loops or with computed names) may be missed or shown differently. Pass `--no-static-parse` to go back to the older behaviour, where the files are actually run to collect the tests.