# Eval review queue

Let people grade real agent answers one at a time, fast, and turn the failures into work.

For: Reviewers who grade, and the people who fix what they find.

Web page: https://candor.themethodic.com/screens/eval-review-queue

## Layouts

### One at a time, from the keyboard

Pick it when: The default: a reviewer grades dozens of answers in a sitting.

Layout: The question and the agent's answer in the middle with Pass and Fail beneath, and a progress count.

Items:

- **Review queue** (`review-queue`, Pro; https://candor.themethodic.com/docs/review-queue.md): Grade real answers one at a time: P pass, F fail, J and K to move.
- **Thread** (`thread`, free; https://candor.themethodic.com/docs/thread.md): The conversation the answer came from.
- **Tool call** (`tool-call`, free; https://candor.themethodic.com/docs/tool-call.md): What the agent did to get it.

```bash
npx candor-ui add review-queue thread tool-call
```

### Queue with the evidence beside it

Pick it when: Reviewers need the sources or the run behind an answer to grade it fairly.

Layout: The answer on the left and its sources and trace on the right.

Items:

- **Review queue** (`review-queue`, Pro; https://candor.themethodic.com/docs/review-queue.md): The grading.
- **Retrieval inspector** (`retrieval-inspector`, Pro; https://candor.themethodic.com/docs/retrieval-inspector.md): The passages behind an answer, in the order the retriever ranked them.
- **Citation** (`citation`, free; https://candor.themethodic.com/docs/citation.md): The sources cited.

```bash
npx candor-ui add review-queue retrieval-inspector citation
```

### Failures by cluster

Pick it when: After grading: what went wrong most, and what to do about it.

Layout: Failure clusters as a list with shares, and the examples of the selected one.

Items:

- **Failure clusters** (`failure-clusters`, Pro; https://candor.themethodic.com/docs/failure-clusters.md): Failing runs grouped by what went wrong, largest first, each with its share.
- **Report reply** (`report-reply`, Pro; https://candor.themethodic.com/docs/report-reply.md): Flag a reply for a person to review.

```bash
npx candor-ui add failure-clusters report-reply
```

## Rules that matter most here

- **Nothing is pre-selected in a decision** (https://candor.themethodic.com/principles#nothing-preselected): Every answer starts ungraded; the queue never shows a suggested grade as a grade.
- **Unknown stays unknown** (https://candor.themethodic.com/principles#unknown-stays-unknown): A skipped answer is counted as skipped, not as passed.
- **Reasoning is believed, not verified** (https://candor.themethodic.com/principles#reasoning-is-believed): An automated check's reasons are labelled as the check's, next to the person's grade.

A full-stack block builds this with an agent route and a store: `eval-review-loop` (https://candor.themethodic.com/docs/eval-review-loop.md). Each block has a `-claude` twin for the Claude Agent SDK.

## Brief for a coding agent

Paste this to the agent that builds the screen.

```text
Build an eval review queue with Candor UI. Start from the eval-review-loop block (eval-review-loop-claude for the Claude Agent SDK): people grade a support agent's replies, recorded checks score them, and an agent summarises the failures.
ReviewQueue shows one answer at a time and takes P for pass, F for fail, J and K to move; every answer starts ungraded.
Beside the answer, show what it came from (Thread, ToolCall) and, for knowledge answers, RetrievalInspector.
Count skipped answers as skipped, not as passed.
FailureClusters shows what failed most, with each cluster's share.
Label an automated check's reasons as the check's, not as verified.

Start from the "One at a time, from the keyboard" layout: The question and the agent's answer in the middle with Pass and Fail beneath, and a progress count.
Candor items: review-queue (https://candor.themethodic.com/docs/review-queue.md), thread (https://candor.themethodic.com/docs/thread.md), tool-call (https://candor.themethodic.com/docs/tool-call.md).
Install: npx candor-ui add review-queue thread tool-call
Read each item's docs before using it and take its props from there; don't guess them. Pro and Premium items need the person's licence key, which they add with `npx candor-ui key <key>`; don't ask for it.
```