Eval review loop

Premium

The scores, the failure clusters and the replies to grade, as of now.

Preview

It runs with a server and an agent, so there’s no preview here: install it to run it in your app.

Install

With the Candor CLI

First time? Get started sets up the registry and theme. Or with shadcn directly: npx shadcn@latest add @candor/eval-review-loop

Brings in: button, card-registry, chat, composer, diff, eval-scorecard, failure-clusters, message, ops, prompt-suggestions, review-queue, thread, use-candor-chat · npm: @ai-sdk/anthropic@^4.0.96, @ai-sdk/react@^4.0.166, ai@^7.0.162, drizzle-orm@^0.45.4, lucide-react@^1.67.0, postgres@^4.0.0, zod@^4.6.7

<GET>

No props.

Also takes the props of Request.

<POST>

A person's grade on one reply: { id, grade: "pass" | "fail", note? }.

No props.

Also takes the props of Request.

<POST>

The agent, on the Vercel AI SDK. Set ANTHROPIC_API_KEY, and CANDOR_APPROVAL_SECRET so approvals can't be forged.

No props.

Also takes the props of Request.

<Page>

No props.

<ReviewPage>

The reviewer's side: grade the support agent's replies, pass or fail with an optional note. Nothing is pre-graded.

No props.

<EvalReviewLoop>

The eval loop: scores from people's grades and recorded checks beside a chat with an agent that reads them, summarises the failures and proposes an instructions change. The change applies as a new version, only after the person approves it.

Prop	Type	Default	Description
chatrequired	`EvalChat`	—	The chat state from useCandorChat (AI SDK) or useClaudeAgent (Claude Agent SDK); the block works with either.
apirequired	`string`	—	The evals API route the scores are read from, e.g. "/eval-review-loop/api/evals".
reviewHrefrequired	`string`	—	Where people grade replies, linked when some aren’t graded yet.

Functions

connectEvalDb(url: string)

Open the block's database from a Postgres URL with the postgres-js driver. Swap the driver here (node-postgres, Neon's websocket Pool, …) if your app already uses another; it must support interactive transactions.

drizzleEvalRepo(db: EvalDb)

seedEval(db: EvalDb)

Load the demo's instruction versions and graded replies into the database, leaving existing rows alone: versions that exist are kept, and the replies load only into an empty table. Run it once after migrating, e.g. from a script: await seedEval(connectEvalDb(process.env.DATABASE_URL!)).

demoEvalData()

The demo's instruction versions and graded replies, for the in-memory store and for seeding a database (./db/seed.ts).

memoryEvalRepo()

In this server process's memory: gone on restart and not shared between instances. Transactions run one at a time, and a transaction that throws is rolled back to the state it started from.

setEvalRepo(repo: EvalRepo)

Use your own storage (or a database you opened yourself). Call it once, before the routes run.

evalRepo()

The storage in use: whatever setEvalRepo() was given, else Postgres when DATABASE_URL is set, else memory.

viewerOf(request: Request)

Who is asking. Replace with your auth; the demo reads a cookie.

readInstructions()

recordOutput(input: { ask: string; answer: string })

Record a reply the support agent wrote, under the instructions current now. Call it from your support agent; the checks run here, on the text, once.

scorecard(version?: number)

Pass rates for one version (by default the scored one) against the version before it. A criterion with nothing recorded before has no baseline.

failureClusters(version?: number)

Failing replies grouped by what failed: each check, and people's fail grades. The change is against the version before.

reviewItems(version?: number)

What the review queue shows: the scored version's replies (or version's), in a stable order (the queue opens on the first ungraded one). A reply's grade is only ever a person's: nothing is pre-filled.

grade(id: string, value: "pass" | "fail", note: string | undefined, by: string)

A person's grade on one reply. Grading again replaces their earlier grade.

proposeInstructions(input: { text: string; reason: string }, by: string)

A proposed instructions change. Nothing changes yet.

applyInstructions(id: string, by: string)

Apply a proposal as a new version, if the instructions haven't changed since it was written. The old version is kept.

resetStore()

For tests: start from the seed data again. On a database this deletes the block's tables' rows.

evalToolTitle(tool: string, input: unknown)

Plain-words titles for the block's tools, the same for both agent SDKs.

Types

CheckId: keyof typeof checks
EvalDb: PgDatabase<PgQueryResultHKT, any>
VersionRecord: One version of the support agent's instructions. Versions are never changed or deleted.; { version: number; text: string; createdAt: string; by: string; note: string }
GradeRecord: { grade: "pass" | "fail"; note?: string | undefined; by: string; at: string }
OutputRecord: One reply the support agent wrote, with the checks run on it when it was recorded and a person's grade.; { id: string version: number ask: string answer: string /** Run in code when the reply was recorded: true is a pass. */ checks: Record<CheckId, boolean> grade?: GradeRecord | undefined }
ProposalRecord: { id: string /** The version the change was written against; it applies only while that is still current. */ baseVersion: number before: string after: string reason: string by: string createdAt: string state: "proposed" | "applied" | "superseded" appliedAs?: number | undefined }
NewOutputRecord: Omit<OutputRecord, "id">
NewProposalRecord: Omit<ProposalRecord, "id" | "state" | "appliedAs">
ProposalPatch: Partial<Pick<ProposalRecord, "state" | "appliedAs">>
Proposal: ProposalRecord
EvalSnapshot: What GET /api/evals returns.; { instructions: { version: number; by: string } scorecard: { version: number; current: boolean; title: string; replies: number; ungraded: number; criteria: Criterion[] } clusters: FailureCluster[] }
EvalChat: What either adapter hook returns (useCandorChat or useClaudeAgent): the block works with both.; { messages: MessageData[] composer: React.ComponentProps<typeof Composer> approvals: { id: string; tool: string; input?: unknown; title?: string | undefined; reason?: string | undefined }[] respond: (id: string, approved: boolean, reason?: string) => unknown error?: string | undefined }