Eval scorecard

Pro

An eval run against its baseline: overall pass rate, then each criterion's rate, its change from the baseline, and a regression flag where it fell by more than the threshold. A criterion with no baseline shows its rate and no change rather than comparing against nothing.

Markdown

Preview

Install

With the Candor CLI

First time? Get started sets up the registry and theme. Or with shadcn directly: npx shadcn@latest add @candor/eval-scorecard

Brings in: locale, ops, utils · npm: lucide-react@^1.67.0

Usage

TSX · The example above

"use client"

import { EvalScorecard } from "@/components/eval-scorecard"
import type { Criterion } from "@/lib/ops"

const criteria: Criterion[] = [
  { id: "c1", label: "Cites a file or table for every fact about the code", passed: 92, failed: 8, baseline: 0.9 },
  { id: "c2", label: "Asks before acting when the target environment is unclear", passed: 71, failed: 29, baseline: 0.84 },
  { id: "c3", label: "Uses the code’s own names (tables, files, flags)", passed: 97, failed: 3, baseline: 0.95 },
  { id: "c4", label: "Marks assumptions as assumptions", passed: 64, failed: 16, baseline: null },
]

export default function Example() {
  return <EvalScorecard className="max-w-2xl" title="Dev agent · Instructions v4" criteria={criteria} />
}

<EvalScorecard>

Prop	Type	Default	Description
titlerequired	`string`	—	The run's name, shown over the overall pass rate.
criteriarequired	`Criterion[]`	—	Each criterion's passed and failed counts and its baseline pass rate (0–1, null when there is none).
threshold	`number \| undefined`	0.05	A drop larger than this (0–1) counts as a regression.
unit	`string \| undefined`	—	What each counted item is, plural, as in "12 of 14 checks passed". Defaults to "checks".
criterionLabel	`string \| undefined`	—	The heading over the criteria column. Defaults to "Criterion".
baselineLabel	`string \| undefined`	—	What the comparison is against, as in "vs baseline" and `No ${baselineLabel}`. Defaults to "baseline".
className	`string \| undefined`	—	Extra classes for the outer element.

Strings

The fixed words, in English by default. Change any of them under evalScorecard in <LocaleProvider strings={{ … }}>; the table is exported as evalScorecardStrings.

Key	English
unit	checks
criterion	Criterion
baseline	baseline
percentPass	% pass
passedOf	{passed} of {total} {unit} passed
regressions	{ one: "{count} regression", other: "{count} regressions" }
passRate	Pass rate
vs	vs {baseline}
passedFailed	{passed} passed · {failed} failed
noBaseline	No {baseline}
points	{points} pts