# Eval review loop (Claude Agent SDK) People grade a support agent's replies, recorded checks score them, and an agent summarises the failures and proposes an instructions change that applies, as a new version, only after approval. - Registry name: `eval-review-loop-claude` (`@candor/eval-review-loop-claude`) - Tier: Premium. Installing it needs a Premium or Team licence key: the person runs `npx candor-ui key ` themselves; never ask for the key in chat or write it to a file. - Docs: https://candor.themethodic.com/docs/eval-review-loop-claude ## Install ```bash npx candor-ui add eval-review-loop-claude ``` Run `npx candor-ui@latest init` once first (https://candor.themethodic.com/get-started.md). Without the CLI: `npx shadcn@latest add @candor/eval-review-loop-claude`. It brings in: button, card-registry, chat, claude-agent-server, composer, diff, eval-scorecard, failure-clusters, message, ops, prompt-suggestions, review-queue, thread, use-claude-agent. npm packages: @anthropic-ai/claude-agent-sdk@^0.3.318, drizzle-orm@^0.45.4, lucide-react@^1.67.0, postgres@^4.0.0, zod@^4.6.7. It installs under your app's `app/` folder and runs with a server and an agent: set `ANTHROPIC_API_KEY`, and read the comment at the top of its `store.ts` for storage and auth. Its files: - `app/eval-review-loop/api/approve/route.ts` - `app/eval-review-loop/api/evals/route.ts` - `app/eval-review-loop/api/route.ts` - `app/eval-review-loop/checks.ts` - `app/eval-review-loop/db/connect.ts` - `app/eval-review-loop/db/drizzle-repo.ts` - `app/eval-review-loop/db/schema.ts` - `app/eval-review-loop/db/seed.ts` - `app/eval-review-loop/page.tsx` - `app/eval-review-loop/repo.ts` - `app/eval-review-loop/review/page.tsx` - `app/eval-review-loop/store.ts` - `app/eval-review-loop/ui.tsx` ## No props. ## The reviewer's side: grade the support agent's replies, pass or fail with an optional note. Nothing is pre-graded. No props. ## The eval loop: scores from people's grades and recorded checks beside a chat with an agent that reads them, summarises the failures and proposes an instructions change. The change applies as a new version, only after the person approves it. Props: - `chat: EvalChat`: The chat state from useCandorChat (AI SDK) or useClaudeAgent (Claude Agent SDK); the block works with either. - `api: string`: The evals API route the scores are read from, e.g. "/eval-review-loop/api/evals". - `reviewHref: string`: Where people grade replies, linked when some aren’t graded yet. ## Functions ### connectEvalDb ```ts connectEvalDb(url: string) ``` Open the block's database from a Postgres URL with the postgres-js driver. Swap the driver here (node-postgres, Neon's websocket Pool, …) if your app already uses another; it must support interactive transactions. ### drizzleEvalRepo ```ts drizzleEvalRepo(db: EvalDb) ``` ### seedEval ```ts seedEval(db: EvalDb) ``` Load the demo's instruction versions and graded replies into the database, leaving existing rows alone: versions that exist are kept, and the replies load only into an empty table. Run it once after migrating, e.g. from a script: `await seedEval(connectEvalDb(process.env.DATABASE_URL!))`. ### demoEvalData ```ts demoEvalData() ``` The demo's instruction versions and graded replies, for the in-memory store and for seeding a database (./db/seed.ts). ### memoryEvalRepo ```ts memoryEvalRepo() ``` In this server process's memory: gone on restart and not shared between instances. Transactions run one at a time, and a transaction that throws is rolled back to the state it started from. ### setEvalRepo ```ts setEvalRepo(repo: EvalRepo) ``` Use your own storage (or a database you opened yourself). Call it once, before the routes run. ### evalRepo ```ts evalRepo() ``` The storage in use: whatever setEvalRepo() was given, else Postgres when DATABASE_URL is set, else memory. ### viewerOf ```ts viewerOf(request: Request) ``` Who is asking. Replace with your auth; the demo reads a cookie. ### readInstructions ```ts readInstructions() ``` ### recordOutput ```ts recordOutput(input: { ask: string; answer: string }) ``` Record a reply the support agent wrote, under the instructions current now. Call it from your support agent; the checks run here, on the text, once. ### scorecard ```ts scorecard(version?: number) ``` Pass rates for one version (by default the scored one) against the version before it. A criterion with nothing recorded before has no baseline. ### failureClusters ```ts failureClusters(version?: number) ``` Failing replies grouped by what failed: each check, and people's fail grades. The change is against the version before. ### reviewItems ```ts reviewItems(version?: number) ``` What the review queue shows: the scored version's replies (or `version`'s), in a stable order (the queue opens on the first ungraded one). A reply's grade is only ever a person's: nothing is pre-filled. ### grade ```ts grade(id: string, value: "pass" | "fail", note: string | undefined, by: string) ``` A person's grade on one reply. Grading again replaces their earlier grade. ### proposeInstructions ```ts proposeInstructions(input: { text: string; reason: string }, by: string) ``` A proposed instructions change. Nothing changes yet. ### applyInstructions ```ts applyInstructions(id: string, by: string) ``` Apply a proposal as a new version, if the instructions haven't changed since it was written. The old version is kept. ### resetStore ```ts resetStore() ``` For tests: start from the seed data again. On a database this deletes the block's tables' rows. ### evalToolTitle ```ts evalToolTitle(tool: string, input: unknown) ``` Plain-words titles for the block's tools, the same for both agent SDKs. ## Types ### CheckId ```ts keyof typeof checks ``` ### EvalDb ```ts PgDatabase ``` ### VersionRecord One version of the support agent's instructions. Versions are never changed or deleted. ```ts { version: number; text: string; createdAt: string; by: string; note: string } ``` ### GradeRecord ```ts { grade: "pass" | "fail"; note?: string | undefined; by: string; at: string } ``` ### OutputRecord One reply the support agent wrote, with the checks run on it when it was recorded and a person's grade. ```ts { id: string version: number ask: string answer: string /** Run in code when the reply was recorded: true is a pass. */ checks: Record grade?: GradeRecord | undefined } ``` ### ProposalRecord ```ts { id: string /** The version the change was written against; it applies only while that is still current. */ baseVersion: number before: string after: string reason: string by: string createdAt: string state: "proposed" | "applied" | "superseded" appliedAs?: number | undefined } ``` ### NewOutputRecord ```ts Omit ``` ### NewProposalRecord ```ts Omit ``` ### ProposalPatch ```ts Partial> ``` ### Proposal ```ts ProposalRecord ``` ### EvalSnapshot What GET /api/evals returns. ```ts { instructions: { version: number; by: string } scorecard: { version: number; current: boolean; title: string; replies: number; ungraded: number; criteria: Criterion[] } clusters: FailureCluster[] } ``` ### EvalChat What either adapter hook returns (useCandorChat or useClaudeAgent): the block works with both. ```ts { messages: MessageData[] composer: React.ComponentProps approvals: { id: string; tool: string; input?: unknown; title?: string | undefined; reason?: string | undefined }[] respond: (id: string, approved: boolean, reason?: string) => unknown error?: string | undefined } ``` ## Rules for using Candor UI - Components take plain typed data. Never pass model output as HTML or render it as a link; messages are typed blocks. - A control that can't be used says why: wrap it in ``, don't just disable it. - Don't show a value you don't have: leave it out (it renders as —) rather than guess. An estimate is labelled as one. - Nothing is pre-selected in a decision a person makes, and the agent is named from your app's data, never from model output. - Your app's words go in `LocaleProvider` (fixed strings) and `TermsProvider` (nouns); entity kinds and agent roles in `EntityKindsProvider` and `AgentRolesProvider`.