Screens / Judging the output

Eval review queue

Let people grade real agent answers one at a time, fast, and turn the failures into work.

For: Reviewers who grade, and the people who fix what they find.

Markdown

3 ways to lay it out

1. One at a time, from the keyboard

Pick it when: The default: a reviewer grades dozens of answers in a sitting.

The question and the agent's answer in the middle with Pass and Fail beneath, and a progress count.

Review queueProGrade real answers one at a time: P pass, F fail, J and K to move.
ThreadFreeThe conversation the answer came from.
Tool callFreeWhat the agent did to get it.

Install one at a time, from the keyboard

Pro and Premium items need a licence key; free ones install as they are.

2. Queue with the evidence beside it

Pick it when: Reviewers need the sources or the run behind an answer to grade it fairly.

The answer on the left and its sources and trace on the right.

Review queueProThe grading.
Retrieval inspectorProThe passages behind an answer, in the order the retriever ranked them.
CitationFreeThe sources cited.

Install queue with the evidence beside it

Pro and Premium items need a licence key; free ones install as they are.

3. Failures by cluster

Pick it when: After grading: what went wrong most, and what to do about it.

Failure clusters as a list with shares, and the examples of the selected one.

Failure clustersProFailing runs grouped by what went wrong, largest first, each with its share.
Report replyProFlag a reply for a person to review.

Install failures by cluster

Pro and Premium items need a licence key; free ones install as they are.

Rules that matter most here

Nothing is pre-selected in a decision
Every answer starts ungraded; the queue never shows a suggested grade as a grade.
Unknown stays unknown
A skipped answer is counted as skipped, not as passed.
Reasoning is believed, not verified
An automated check's reasons are labelled as the check's, next to the person's grade.

Already covered elsewhere

Builds the whole screen with an agent route and a store: Eval review loop (Premium), with a -claude twin for the Claude Agent SDK.

Brief for a coding agent

Paste this into the agent that builds the screen. The docs MCP server returns the same text from get_screen.

Build an eval review queue with Candor UI. Start from the eval-review-loop block (eval-review-loop-claude for the Claude Agent SDK): people grade a support agent's replies, recorded checks score them, and an agent summarises the failures.
ReviewQueue shows one answer at a time and takes P for pass, F for fail, J and K to move; every answer starts ungraded.
Beside the answer, show what it came from (Thread, ToolCall) and, for knowledge answers, RetrievalInspector.
Count skipped answers as skipped, not as passed.
FailureClusters shows what failed most, with each cluster's share.
Label an automated check's reasons as the check's, not as verified.

Start from the "One at a time, from the keyboard" layout: The question and the agent's answer in the middle with Pass and Fail beneath, and a progress count.
Candor items: review-queue (https://candor.themethodic.com/docs/review-queue.md), thread (https://candor.themethodic.com/docs/thread.md), tool-call (https://candor.themethodic.com/docs/tool-call.md).
Install: npx candor-ui add review-queue thread tool-call
Read each item's docs before using it and take its props from there; don't guess them. Pro and Premium items need the person's licence key, which they add with `npx candor-ui key <key>`; don't ask for it.