Eval dashboard (Claude Agent SDK)

Premium

The person's answer to the agent's permission request: { id, approved, reason? }.

Preview

It runs with a server and an agent, so there’s no preview here: install it to run it in your app.

Install

With the Candor CLI

First time? Get started sets up the registry and theme. Or with shadcn directly: npx shadcn@latest add @candor/eval-dashboard-claude

Brings in: badge, button, card-registry, chat, claude-agent-server, comparison-view, composer, eval-scorecard, failure-clusters, message, ops, prompt-suggestions, quality-trend, thread, use-claude-agent · npm: @anthropic-ai/claude-agent-sdk@^0.3.318, drizzle-orm@^0.45.4, lucide-react@^1.67.0, postgres@^4.0.0, zod@^4.6.7

<POST>

No props.

Also takes the props of Request.

<POST>

The agent, on the Claude Agent SDK. Set ANTHROPIC_API_KEY. Runs on one Node server, not spread over instances.

No props.

Also takes the props of Request.

<GET>

The dashboard's data as of now: production, the versions and runs, the trend, one run's scorecard and failure clusters (?run=R-004, else the newest finished run), and a case-by-case comparison of two runs (?a=R-003&b=R-004, else that run against its production baseline), with one case's detail when &case=C-04 is given.

No props.

Also takes the props of Request.

<ComparePage>

Two runs side by side, case by case: which cases regressed and which were fixed, and one case's input, both outputs and every check. Without ?a=…&b=… it compares the newest run with production's latest run on the same suite.

No props.

<Page>

No props.

<RunComparison>

Two runs case by case: the totals, then each case's result in both runs and whether it regressed or was fixed. caseHref links each case to its detail; the cases and runs are the app's own, so the links are too.

Prop	Type	Default	Description
beforerequired	`RunLabel`	—	—
afterrequired	`RunLabel`	—	—
noterequired	`string`	—	—
summaryrequired	`ComparisonRow[]`	—	—
casesrequired	`CaseRow[]`	—	—
caseHref	`((caseId: string) => string) \| undefined`	—	—
only	`CaseChange[] \| undefined`	—	—

<CaseDetail>

One case in two runs, side by side. Outputs are shown as plain text, never as markup or links.

Prop	Type	Default	Description
detailrequired	`CaseDetailData`	—	—

<EvalDashboard>

The eval dashboard: the suite's pass rate over runs, a run's scorecard against production's latest run, its failures, and regressions and fixes case by case, beside a chat with an agent that reads and compares runs, starts them, and promotes a version only after the person approves and the gate passes.

Prop	Type	Default	Description
chatrequired	`EvalDashChat`	—	The chat state from useCandorChat (AI SDK) or useClaudeAgent (Claude Agent SDK); the block works with either.
apirequired	`string`	—	The runs API route the dashboard reads, e.g. "/eval-dashboard/api/runs".
compareHrefrequired	`string`	—	The page that compares two runs case by case, e.g. "/eval-dashboard/compare".

Functions

connectEvalDashDb(url: string)

Open the block's database from a Postgres URL with the postgres-js driver. Swap the driver here (node-postgres, Neon's websocket Pool, …) if your app already uses another; it must support interactive transactions.

drizzleEvalDashRepo(db: EvalDashDb)

seedEvalDash(db: EvalDashDb)

Load the demo's versions, suite revisions, runs and production version into the database, leaving existing rows alone: versions and revisions that exist are kept, production is set only when nothing is in production, and the runs and promotions load only into empty tables. Run it once after migrating, e.g. from a script: await seedEvalDash(connectEvalDashDb(process.env.DATABASE_URL!)).

resultsOf(suite: SuiteRecord, outputs: Record<string, string>)

A finished run's results from recorded outputs, the way the store records them.

demoEvalDashData()

The demo's versions, suite revisions, runs and promotions, for the in-memory store and for seeding a database (./db/seed.ts).

memoryEvalDashRepo()

In this server process's memory: gone on restart and not shared between instances. Transactions run one at a time, and a transaction that throws is rolled back to the state it started from.

setEvalDashRepo(repo: EvalDashRepo)

Use your own storage (or a database you opened yourself). Call it once, before the routes run.

evalDashRepo()

The storage in use: whatever setEvalDashRepo() was given, else Postgres when DATABASE_URL is set, else memory.

setCaseRunner(run: CaseRunner, label: string)

Call your agent for real: setCaseRunner(async ({ version, case }) => myAgent(version, case.input), "Support agent, live"). label is stored on each run, so a run always says where its outputs came from. Call it once at startup.

viewerOf(request: Request)

Who is asking. Replace with your auth; the demo reads a cookie.

overview()

The production version, the current suite, every version with whether it may be promoted, and every run.

qualityTrend()

Cases passed per finished run, oldest first, with what changed alongside: a version's first run, a promotion, a new suite revision. A run where no case could run has no score.

scorecard(runId?: string)

One run's checks by kind against production's latest run on the same suite. A check kind production never ran has no baseline.

failureClusters(runId?: string)

A run's failing cases grouped by the kind of check that failed, and the cases that couldn't run, with the change against the baseline run.

compareRuns(beforeId: string, afterId: string)

Two finished runs case by case: which cases regressed (passed before, not now), which were fixed, and the totals. A case one run didn't have, or couldn't run, is a change nobody can call; it's "unknown", except that a case that passed before and couldn't run now counts as a regression, as the gate counts it.

caseDetail(beforeId: string, afterId: string, caseId: string)

One case in two runs: its input, each output as recorded, and each check's result.

startRun(version: number, by: string)

Run the current suite against one version and record every case's output and check results. Refused while the version already has a run going. Cases run one at a time through the runner set with setCaseRunner(); a case whose runner call fails is recorded as couldn't run, not as a fail. Run big suites from a background job rather than in the agent's request: the run record works the same either way.

promoteVersion(input: { version: number; replaces: number }, by: string)

Put a version in production, replacing replaces, after a person approved it. Refused when production is no longer replaces, the version has no finished run on the current suite, production has none to check against, or the version's latest run fails a case production's latest run passes.

addVersion(input: { model: string; instructions: string; note: string }, by: string)

Add a version of your agent (from your instructions editor, say). It gets the next number.

addSuiteRevision(input: { cases: SuiteCase[]; note: string }, by: string)

Replace the suite with a new revision. Runs on earlier revisions are kept, but the gate counts only runs on this one.

resetStore()

For tests: start from the seed data again. On a database this deletes the block's tables' rows.

evalDashToolTitle(tool: string, input: unknown)

Plain-words titles for the block's tools, the same for both agent SDKs.

useEvalDashData(url: string)

Load the dashboard's data from url now and every few seconds: runs finish and versions get promoted from the chat.

Types

CheckKind: keyof typeof checkKinds
CheckSpec: One check a case asks for: its kind, and the text it looks for when the kind needs one.; { kind: CheckKind; arg?: string | undefined }
CheckResult: One check's result on one output, as a run stores it.; { kind: CheckKind; label: string; passed: boolean }
EvalDashDb: PgDatabase<PgQueryResultHKT, any>
VersionRecord: One version of the support agent: its instructions and model. Versions are never changed or deleted.; { version: number; model: string; instructions: string; note: string; createdAt: string; by: string }
SuiteCase: One test case: the input the agent gets and the checks run in code on what it writes back.; { id: string; title: string; input: string; checks: CheckSpec[] }
SuiteRecord: One revision of the suite, every case in full. Revisions are never changed; runs say which one they ran.; { revision: number; cases: SuiteCase[]; note: string; createdAt: string; by: string }
CaseResult: One case's result in a run. `passed` is true when every check passed, and null when the case couldn't run (the runner failed), since then nothing is known about it.; { caseId: string; output: string | null; checks: CheckResult[]; passed: boolean | null; error?: string | undefined }
RunStatus: "running" | "finished" | "abandoned"
RunRecord: One run of the suite against one version. Results are stored when it finishes, all at once.; { id: string version: number /** The suite revision it ran. */ suite: number status: RunStatus /** Where the outputs came from, in words: the runner the app set, or the demo's recordings. */ runner: string startedBy: string startedAt: string finishedAt?: string | undefined results: CaseResult[] }
ProductionRecord: The version in production now.; { version: number; since: string; by: string }
PromotionRecord: One promotion, kept for the record. `replaced` is null for the first.; { version: number; replaced: number | null; by: string; at: string }
NewRunRecord: Pick<RunRecord, "version" | "suite" | "runner" | "startedBy" | "startedAt">
RunPatch: { status: RunStatus; finishedAt?: string | undefined }
CaseRunner: Runs one case against one version of your agent and returns what it wrote. Throw when it can't.; (input: { version: VersionRecord; case: SuiteCase }) => Promise<string>
RunSummary: ReturnType<typeof runSummary>
CaseStatus: "pass" | "fail" | "error" | "absent"
CaseChange: "regressed" | "fixed" | "unchanged" | "unknown"
CaseRow: { id: string; title: string; before: CaseStatus; after: CaseStatus; change: CaseChange; failing: string[] }
Comparison: Awaited<ReturnType<typeof compareRuns>>
CaseDetailData: What GET /api/runs returns for one case: its input, and each run's recorded output and checks.; { id: string title: string input: string runs: { run: string; version: number; runner: string; status: CaseStatus; output: string | null; checks: { label: string; passed: boolean }[]; error?: string | undefined }[] }
EvalDashData: What GET /api/runs returns.; { production: { version: number; since: string; by: string } suite: { revision: number; cases: number; note: string } runner: string versions: VersionRow[] runs: RunSummary[] trend: { title: string; subtitle: string; points: QualityPoint[] } scorecard: { run: string; version: number; runner: string; baseline: string | null; title: string; cases: number; couldNotRun: number; criteria: Criterion[] }
EvalDashChat: What either adapter hook returns (useCandorChat or useClaudeAgent): the block works with both.; { messages: MessageData[] composer: React.ComponentProps<typeof Composer> approvals: { id: string; tool: string; input?: unknown; title?: string | undefined; reason?: string | undefined }[] respond: (id: string, approved: boolean, reason?: string) => unknown error?: string | undefined }