Testing and release

Run from the repository root:

pnpm install
pnpm run typecheck
pnpm run test
pnpm run build
pnpm --dir docs run build

Node.js 22 or later is required. Do not treat a successful build as sufficient for changes involving persistence, agent sequencing, or browser UI.

Test by risk

Change	Minimum additional verification
Canonical model or world format	parser problems, round-trip fixtures, session population, authoring tools
SQLite schema	fresh database, migration from prior schema, snapshot/restore, branch
Record wire format	JSON Schema, parser, atomic application, read-back, keeper verification
Recall	character isolation, no-vector fallback, vector invalidation, story-time fading
Orchestration	protocol order, retries, cancellation, stable child prefix, background settling
DSH integration	contract test, hot reload, chat open/reopen, archive/delete
Browser UI	typecheck, production build, and live verification in a dedicated profile
Documentation	English and Chinese production builds with broken links treated as errors

tests/support/architecture.spec.ts in the core package and tests/architecture.spec.ts in the state package read the source imports and fail on a broken layering: shared code imports only itself and the model; the browser half never imports host code, Node or the state package; the host never imports the browser half; packages are used only through their root exports; SQLite is reached through the driver alone; the tools, records and memory never see the driver; the model package imports nothing but itself. When a new rule is wanted, add it there rather than describing it.

Two cross-cutting tests guard what no single feature test would notice. tests/host/prompts/rules-coverage.spec.ts lists, for each rule that prompts of a kind must carry (work only from what is given; write in the language the chat's prompts name), the prompts that carry it and, with a reason, every one that need not, in each shipped language; a prompt added later is in neither list and fails until someone decides. A loop test reads every request a character was sent and fails on a key or on the number of a turn or round: the story has names and times, the numbers belong to the code.

Use deterministic fakes for agent-loop tests. Assert tool calls and state transitions, not model prose. Concurrency tests should force overlapping turns/jobs and verify serialization, cancellation, and cleanup. Failure tests must confirm that a partially completed operation either rolls back or remains explicitly resumable.

Quality gate

Pull requests and pushes to main run type-checking, the tests with coverage, the release build and the release check, then a SonarQube analysis whose quality gate must pass. Publishing waits for both. Run the same coverage locally:

pnpm test:coverage

Each package writes packages/<name>/coverage/lcov.info; scripts/merge-coverage.mjs joins them into coverage/lcov.info with paths relative to the repository root, which is what sonar-project.properties reads. Every source file of a package is listed, so a file no test loads counts as uncovered rather than missing. Tests, built output, starter worlds, shipped packs and configuration files are left out of the analysis or the coverage figures. The scan needs the SONAR_TOKEN repository secret and does not run for pull requests from forks. Fix what the gate reports instead of excluding the file; add an exclusion only for code that cannot be run by a test, and say why in the pull request.

Live evaluation

Scripted-model tests cannot show what a real model does with the prompts. eval/ plays scenarios (look around and talk, walk through an exit, try something the world's rules forbid, wait for hours, say something a character must remember) into a running dev DSH with the model you want to judge, then checks the story's database, the chat log and the helpers' record. Use a throwaway roleplay root, never your own:

node eval/run.mjs "<dsh url with token>" --label qwen-35b
node eval/run.mjs "<dsh url with token>" --label qwen-35b-2 --baseline eval/results/qwen-35b.json

Scenarios that a model plays (soak: it pursues a goal in its own words for a few turns, and the story must stay sound) need a model for the player, any OpenAI-compatible endpoint, and are skipped without one:

node eval/run.mjs "<dsh url with token>" --only soak --player-model <name> [--player-url http://localhost:8081/v1] [--player-key-env VAR] [--player-reasoning none|omit|low|high]

The player is bounded on every side: at most --player-turns turns (default 6, never above 30), --player-tokens tokens spent by the player's own model (default 30000), --player-minutes of wall time (default 40) and none that would pass it by the typical turn time; one reply is at most 160 tokens and an action 300 characters; it is sent only the latest narration and its last five actions, never the whole story. It stops when it says it is done, repeats itself, is given nothing to read twice, writes something that is not an action (a / command), or a turn fails or ends without delivering anything, and the report says which (stop: done, turns, tokens, time, stuck, invalid, failed). eval/player.test.mjs tests these limits against a stub (node --test eval/player.test.mjs).

A finished story can also be audited afterwards, read-only: node eval/audit.mjs <session folder> --url <openai base url> --model <name> [--scope story|character:<name>|turn:<n>] [--out report.json]. A judge model reads the story's records (for a character, only what it took part in or perceived) and names errors from eval/audit-criteria.json (contradiction, fabrication, timeline, knowledge leak, memory lapse, persona drift, speaking for the player, repetition), each with the record it rests on. Every dimension starts at 100 and the script, not the model, subtracts the penalty of each valid error; an error with no record number or not on the list is not counted, and when the records cannot show a dimension the judge abstains and the score is null. The criteria list errors only, never a model answer, and are an editable file; narrative quality is not scored. eval/audit.test.mjs tests the scoring and evidence against a stub.

A report is written to eval/results/<label>.json. With --baseline the run says which scenarios got more than twice as slow and which checks passed before and fail now, and exits non-zero. Scenarios are small files in eval/scenarios/; add one when a live session finds a bug.

The check benchmark

The world checks have a labelled benchmark in packages/dsh-agentic-roleplay-core/tests/benchmark/. Its base is Salt Harbor, an invented harbor town written in English and in Chinese with the same structure, that uses every part of the world format: a cast with sheets, voices, feelings, goals, relationships, knowledge and routines; places with features and ways of every state, and a lock; things of every kind, hidden, locked, worn and in containers; organisations with members and goals; background of both scopes with dates and links; tracked values of every kind and subject; triggers of the main condition and effect types; dice, a plot, a calendar, rules, prompts and openings. Each case is the base with one defect planted (or none, or an ambiguous pair that is not a defect) and names the findings the checks must make and nothing else.

• Always run (pnpm test): coverage.spec.ts starts a story from the base world in both languages and asserts that every part of it reached the story, by feature, so a part that stops arriving is named. cases.spec.ts runs the cases the program decides, the consistency, quality, first-scene and card-comparison checks, in both languages, with the exact findings expected.
• Live, against the development machine's models, scored for precision, recall, false alarms, retries, fallbacks and time: BENCH_LIVE=1 BENCH_LABEL=<name> BENCH_MODE=llm|decision|hybrid npx vitest run tests/benchmark/live.spec.ts runs the cases a model judges (meaning-level duplicates of values and names, contradictory background, thin sheets, flat openings, gaps, and the comparison with the card, including a paraphrase and a translation that must not be flagged). The result is written to eval/results/checks-<label>.json. BENCH_ONLY, BENCH_LANGS and BENCH_REPEAT narrow or repeat it; run a case several times before trusting a difference, because a model's answer is chance per call.
• Reranker trial: BENCH_LIVE=1 npx vitest run tests/benchmark/rerank.spec.ts ranks every pair of three labelled sets (values, things, background) by embedding closeness and by a reranker, and counts how many of the real pairs land among the twelve a check asks a model about.

Do not tune a threshold from an unlabelled collection of worlds: add a case here first.

Persisted compatibility

Never edit an existing migration after release. Add a new ordered migration and keep row mapping tolerant only where the schema explicitly permits it. Opening a session that is behind the latest schema first checks its integrity and keeps a copy of it as it was in migration-backups/schema-<version>.sqlite beside the database; a migration that damages data can be undone by putting that copy back as session.sqlite. Keep world files human-editable and report all validation problems in one pass. Stable ids and media paths must not be silently rewritten.

Release review

Before publishing:

• inspect generated JavaScript and declarations for private-package imports;
• confirm bundle dependencies and cordis.patch.yml include every required plugin once conceptually;
• check package files include prompts, client output, and public entry points;
• verify English and Chinese prompt sets have identical names and variables;
• test without each optional AI capability;
• check configuration is represented by Schemastery and documented;
• update both documentation locales and user-visible release notes;
• review git diff for generated caches, local profiles, test worlds, credentials, and other machine-specific files.

Use a dedicated development profile and disposable session/world fixtures. Never validate by modifying a user's normal DSH profile.