# Eval dashboard (Claude Agent SDK) Each version of an agent runs a fixed suite of cases checked in code; an agent reads and compares the runs and explains regressions from recorded results, and a version goes to production only after approval, refused when its latest run fails a case production passes. - Registry name: `eval-dashboard-claude` (`@candor/eval-dashboard-claude`) - Tier: Premium. Installing it needs a Premium or Team licence key: the person runs `npx candor-ui key ` themselves; never ask for the key in chat or write it to a file. - Docs: https://candor.themethodic.com/docs/eval-dashboard-claude ## Install ```bash npx candor-ui add eval-dashboard-claude ``` Run `npx candor-ui@latest init` once first (https://candor.themethodic.com/get-started.md). Without the CLI: `npx shadcn@latest add @candor/eval-dashboard-claude`. It brings in: badge, button, card-registry, chat, claude-agent-server, comparison-view, composer, eval-scorecard, failure-clusters, message, ops, prompt-suggestions, quality-trend, thread, use-claude-agent. npm packages: @anthropic-ai/claude-agent-sdk@^0.3.318, drizzle-orm@^0.45.4, lucide-react@^1.67.0, postgres@^4.0.0, zod@^4.6.7. It installs under your app's `app/` folder and runs with a server and an agent: set `ANTHROPIC_API_KEY`, and read the comment at the top of its `store.ts` for storage and auth. Its files: - `app/eval-dashboard/api/approve/route.ts` - `app/eval-dashboard/api/route.ts` - `app/eval-dashboard/api/runs/route.ts` - `app/eval-dashboard/checks.ts` - `app/eval-dashboard/compare/page.tsx` - `app/eval-dashboard/db/connect.ts` - `app/eval-dashboard/db/drizzle-repo.ts` - `app/eval-dashboard/db/schema.ts` - `app/eval-dashboard/db/seed.ts` - `app/eval-dashboard/page.tsx` - `app/eval-dashboard/repo.ts` - `app/eval-dashboard/store.ts` - `app/eval-dashboard/ui.tsx` ## Two runs side by side, case by case: which cases regressed and which were fixed, and one case's input, both outputs and every check. Without `?a=…&b=…` it compares the newest run with production's latest run on the same suite. No props. ## No props. ## Two runs case by case: the totals, then each case's result in both runs and whether it regressed or was fixed. `caseHref` links each case to its detail; the cases and runs are the app's own, so the links are too. Props: - `before: RunLabel` - `after: RunLabel` - `note: string` - `summary: ComparisonRow[]` - `cases: CaseRow[]` - `caseHref?: ((caseId: string) => string) | undefined` - `only?: CaseChange[] | undefined` ## One case in two runs, side by side. Outputs are shown as plain text, never as markup or links. Props: - `detail: CaseDetailData` ## The eval dashboard: the suite's pass rate over runs, a run's scorecard against production's latest run, its failures, and regressions and fixes case by case, beside a chat with an agent that reads and compares runs, starts them, and promotes a version only after the person approves and the gate passes. Props: - `chat: EvalDashChat`: The chat state from useCandorChat (AI SDK) or useClaudeAgent (Claude Agent SDK); the block works with either. - `api: string`: The runs API route the dashboard reads, e.g. "/eval-dashboard/api/runs". - `compareHref: string`: The page that compares two runs case by case, e.g. "/eval-dashboard/compare". ## Functions ### connectEvalDashDb ```ts connectEvalDashDb(url: string) ``` Open the block's database from a Postgres URL with the postgres-js driver. Swap the driver here (node-postgres, Neon's websocket Pool, …) if your app already uses another; it must support interactive transactions. ### drizzleEvalDashRepo ```ts drizzleEvalDashRepo(db: EvalDashDb) ``` ### seedEvalDash ```ts seedEvalDash(db: EvalDashDb) ``` Load the demo's versions, suite revisions, runs and production version into the database, leaving existing rows alone: versions and revisions that exist are kept, production is set only when nothing is in production, and the runs and promotions load only into empty tables. Run it once after migrating, e.g. from a script: `await seedEvalDash(connectEvalDashDb(process.env.DATABASE_URL!))`. ### resultsOf ```ts resultsOf(suite: SuiteRecord, outputs: Record) ``` A finished run's results from recorded outputs, the way the store records them. ### demoEvalDashData ```ts demoEvalDashData() ``` The demo's versions, suite revisions, runs and promotions, for the in-memory store and for seeding a database (./db/seed.ts). ### memoryEvalDashRepo ```ts memoryEvalDashRepo() ``` In this server process's memory: gone on restart and not shared between instances. Transactions run one at a time, and a transaction that throws is rolled back to the state it started from. ### setEvalDashRepo ```ts setEvalDashRepo(repo: EvalDashRepo) ``` Use your own storage (or a database you opened yourself). Call it once, before the routes run. ### evalDashRepo ```ts evalDashRepo() ``` The storage in use: whatever setEvalDashRepo() was given, else Postgres when DATABASE_URL is set, else memory. ### setCaseRunner ```ts setCaseRunner(run: CaseRunner, label: string) ``` Call your agent for real: `setCaseRunner(async ({ version, case }) => myAgent(version, case.input), "Support agent, live")`. `label` is stored on each run, so a run always says where its outputs came from. Call it once at startup. ### viewerOf ```ts viewerOf(request: Request) ``` Who is asking. Replace with your auth; the demo reads a cookie. ### overview ```ts overview() ``` The production version, the current suite, every version with whether it may be promoted, and every run. ### qualityTrend ```ts qualityTrend() ``` Cases passed per finished run, oldest first, with what changed alongside: a version's first run, a promotion, a new suite revision. A run where no case could run has no score. ### scorecard ```ts scorecard(runId?: string) ``` One run's checks by kind against production's latest run on the same suite. A check kind production never ran has no baseline. ### failureClusters ```ts failureClusters(runId?: string) ``` A run's failing cases grouped by the kind of check that failed, and the cases that couldn't run, with the change against the baseline run. ### compareRuns ```ts compareRuns(beforeId: string, afterId: string) ``` Two finished runs case by case: which cases regressed (passed before, not now), which were fixed, and the totals. A case one run didn't have, or couldn't run, is a change nobody can call; it's "unknown", except that a case that passed before and couldn't run now counts as a regression, as the gate counts it. ### caseDetail ```ts caseDetail(beforeId: string, afterId: string, caseId: string) ``` One case in two runs: its input, each output as recorded, and each check's result. ### startRun ```ts startRun(version: number, by: string) ``` Run the current suite against one version and record every case's output and check results. Refused while the version already has a run going. Cases run one at a time through the runner set with setCaseRunner(); a case whose runner call fails is recorded as couldn't run, not as a fail. Run big suites from a background job rather than in the agent's request: the run record works the same either way. ### promoteVersion ```ts promoteVersion(input: { version: number; replaces: number }, by: string) ``` Put a version in production, replacing `replaces`, after a person approved it. Refused when production is no longer `replaces`, the version has no finished run on the current suite, production has none to check against, or the version's latest run fails a case production's latest run passes. ### addVersion ```ts addVersion(input: { model: string; instructions: string; note: string }, by: string) ``` Add a version of your agent (from your instructions editor, say). It gets the next number. ### addSuiteRevision ```ts addSuiteRevision(input: { cases: SuiteCase[]; note: string }, by: string) ``` Replace the suite with a new revision. Runs on earlier revisions are kept, but the gate counts only runs on this one. ### resetStore ```ts resetStore() ``` For tests: start from the seed data again. On a database this deletes the block's tables' rows. ### evalDashToolTitle ```ts evalDashToolTitle(tool: string, input: unknown) ``` Plain-words titles for the block's tools, the same for both agent SDKs. ### useEvalDashData ```ts useEvalDashData(url: string) ``` Load the dashboard's data from `url` now and every few seconds: runs finish and versions get promoted from the chat. ## Types ### CheckKind ```ts keyof typeof checkKinds ``` ### CheckSpec One check a case asks for: its kind, and the text it looks for when the kind needs one. ```ts { kind: CheckKind; arg?: string | undefined } ``` ### CheckResult One check's result on one output, as a run stores it. ```ts { kind: CheckKind; label: string; passed: boolean } ``` ### EvalDashDb ```ts PgDatabase ``` ### VersionRecord One version of the support agent: its instructions and model. Versions are never changed or deleted. ```ts { version: number; model: string; instructions: string; note: string; createdAt: string; by: string } ``` ### SuiteCase One test case: the input the agent gets and the checks run in code on what it writes back. ```ts { id: string; title: string; input: string; checks: CheckSpec[] } ``` ### SuiteRecord One revision of the suite, every case in full. Revisions are never changed; runs say which one they ran. ```ts { revision: number; cases: SuiteCase[]; note: string; createdAt: string; by: string } ``` ### CaseResult One case's result in a run. `passed` is true when every check passed, and null when the case couldn't run (the runner failed), since then nothing is known about it. ```ts { caseId: string; output: string | null; checks: CheckResult[]; passed: boolean | null; error?: string | undefined } ``` ### RunStatus ```ts "running" | "finished" | "abandoned" ``` ### RunRecord One run of the suite against one version. Results are stored when it finishes, all at once. ```ts { id: string version: number /** The suite revision it ran. */ suite: number status: RunStatus /** Where the outputs came from, in words: the runner the app set, or the demo's recordings. */ runner: string startedBy: string startedAt: string finishedAt?: string | undefined results: CaseResult[] } ``` ### ProductionRecord The version in production now. ```ts { version: number; since: string; by: string } ``` ### PromotionRecord One promotion, kept for the record. `replaced` is null for the first. ```ts { version: number; replaced: number | null; by: string; at: string } ``` ### NewRunRecord ```ts Pick ``` ### RunPatch ```ts { status: RunStatus; finishedAt?: string | undefined } ``` ### CaseRunner Runs one case against one version of your agent and returns what it wrote. Throw when it can't. ```ts (input: { version: VersionRecord; case: SuiteCase }) => Promise ``` ### RunSummary ```ts ReturnType ``` ### CaseStatus ```ts "pass" | "fail" | "error" | "absent" ``` ### CaseChange ```ts "regressed" | "fixed" | "unchanged" | "unknown" ``` ### CaseRow ```ts { id: string; title: string; before: CaseStatus; after: CaseStatus; change: CaseChange; failing: string[] } ``` ### Comparison ```ts Awaited> ``` ### CaseDetailData What GET /api/runs returns for one case: its input, and each run's recorded output and checks. ```ts { id: string title: string input: string runs: { run: string; version: number; runner: string; status: CaseStatus; output: string | null; checks: { label: string; passed: boolean }[]; error?: string | undefined }[] } ``` ### EvalDashData What GET /api/runs returns. ```ts { production: { version: number; since: string; by: string } suite: { revision: number; cases: number; note: string } runner: string versions: VersionRow[] runs: RunSummary[] trend: { title: string; subtitle: string; points: QualityPoint[] } scorecard: { run: string; version: number; runner: string; baseline: string | null; title: string; cases: number; couldNotRun: number; criteria: Criterion[] } ``` ### EvalDashChat What either adapter hook returns (useCandorChat or useClaudeAgent): the block works with both. ```ts { messages: MessageData[] composer: React.ComponentProps approvals: { id: string; tool: string; input?: unknown; title?: string | undefined; reason?: string | undefined }[] respond: (id: string, approved: boolean, reason?: string) => unknown error?: string | undefined } ``` ## Rules for using Candor UI - Components take plain typed data. Never pass model output as HTML or render it as a link; messages are typed blocks. - A control that can't be used says why: wrap it in ``, don't just disable it. - Don't show a value you don't have: leave it out (it renders as —) rather than guess. An estimate is labelled as one. - Nothing is pre-selected in a decision a person makes, and the agent is named from your app's data, never from model output. - Your app's words go in `LocaleProvider` (fixed strings) and `TermsProvider` (nouns); entity kinds and agent roles in `EntityKindsProvider` and `AgentRolesProvider`.