Skip to content
Candor UI
Start free
Tier

Screen guide

Pro

A question or a walk-through over AgentScreen's numbered markers, for the person watching the screen. A question ("Which of these, 1 or 2?") has one answer button per marker, named by its number and label, none chosen; a walk-through ("Step 2 of 4: Click Approve") has Back, Next and the list of steps, the current one marked.

It takes the markers, and the app passes the same list to AgentScreen: the question's, or the current step's. Both number a list the same way (numberedMarkers: from 1, in order, a repeated id once), and the picture draws them in one place only, so a marker can't be drawn twice or read with a different number here than on the screen. (Were the guide to tell the app which markers to show, through a callback, there'd be a render with the screen still on the last step's markers while the guide named the next step's numbers.)

The walk-through's step is the app's (step, onStepChange). It moves only on the person's Back or Next, or the app's own report that the step is done: never on a guess that the person clicked. A new step is announced. When the picture is stale, lost or ended, pass frameStale (AgentScreen's onFrameStaleChange gives it) and the guide says its markers may be out of date, and locks the question's answers with that reason: a marker on an old picture may point at something that has moved.

Markdown
Version 5.4.0· other versions

This page is for 5.4.0, the current release. An earlier release’s page is the Markdown that release published, so the docs match the version you have installed (candor-lock.json records it).

Not in the docs of 1.3.0, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.5.1, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.8.0, 2.9.0, 2.10.0, 3.0.0, 4.0.0, 4.0.1, 4.0.2, 4.1.0, 4.2.0, 4.3.0, 4.4.0, 4.5.0, 5.0.0, 5.1.0 and 5.2.0: the page came later, or had another name then.

Docs pages weren’t kept for 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0 and 1.2.1: they are kept from 1.3.0 on, and before that only llms.txt was published.

Preview

Loading the preview

Install

With the Candor CLI

First time? Get started sets up the registry and theme. Or with shadcn directly: npx shadcn@latest add @candor/screen-guide

Brings in: agent-screen, bidi, button, heading-level, heading-scope, locale, locked, tokens, use-announcer, use-focus-within, utils · npm: lucide-react@^1.67.0

Usage

TSX · The example above
"use client"

import * as React from "react"
import type { AgentIdentity } from "@/lib/candor/chat"
import { AgentScreen, type AgentScreenMarker } from "@/components/candor/agent-screen"
import { ScreenGuide, type ScreenGuideStep } from "@/components/candor/screen-guide"

const agent: AgentIdentity = { id: "a-web", name: "Browser agent", role: "dev" }

// A drawn stand-in for the screenshot your server sends: a request with two buttons, 640 by 400.
const frame =
  "data:image/svg+xml," +
  encodeURIComponent(
    `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 400"><rect width="640" height="400" fill="#f6f6f4"/><rect width="640" height="44" fill="#e4e4e0"/><text x="24" y="28" font-family="sans-serif" font-size="15" fill="#333">Settings</text><text x="40" y="110" font-family="sans-serif" font-size="20" fill="#333">Access request</text><rect x="40" y="300" width="130" height="40" rx="4" fill="#2f8f5b"/><text x="76" y="325" font-family="sans-serif" font-size="15" fill="#fff">Approve</text><rect x="190" y="300" width="130" height="40" rx="4" fill="#fff" stroke="#ccc"/><text x="232" y="325" font-family="sans-serif" font-size="15" fill="#333">Reject</text></svg>`
  )

const steps: (ScreenGuideStep & { markers: AgentScreenMarker[] })[] = [
  { id: "open", instruction: "Open Settings", markers: [{ id: "settings", x: 50, y: 22, label: "Settings" }] },
  { id: "decide", instruction: "Click Approve or Reject", markers: [
    { id: "approve", x: 105, y: 320, label: "Approve" },
    { id: "reject", x: 255, y: 320, label: "Reject" },
  ] },
]

export default function Example() {
  const [mode, setMode] = React.useState<"question" | "walkthrough">("walkthrough")
  const [step, setStep] = React.useState("open")
  const [answer, setAnswer] = React.useState<string>()
  // AgentScreen's verdict on the picture, passed to the guide: stale, lost or ended locks its answers.
  const [frameStale, setFrameStale] = React.useState(false)
  // One list, held by the app and passed to both: the screen and the guide number it the same way.
  const markers = mode === "question" ? steps[1]!.markers : (steps.find((s) => s.id === step)?.markers ?? [])

  return (
    <div className="flex max-w-3xl flex-col gap-4">
      <div className="flex gap-2 text-sm">
        <button type="button" className="underline" onClick={() => setMode(mode === "question" ? "walkthrough" : "question")}>
          {mode === "question" ? "Show the walk-through" : "Show the question"}
        </button>
      </div>
      <ScreenGuide
        mode={mode}
        markers={markers}
        onAnswer={setAnswer}
        steps={steps}
        step={step}
        // The step moves when the person presses Back or Next, or when your app sees the click on its own screen.
        onStepChange={setStep}
        frameStale={frameStale}
      />
      {answer && <p className="text-sm text-muted-foreground">You answered: {answer}</p>}
      <AgentScreen agent={agent} connection="live" source={{ kind: "image", src: frame }} action="Waiting for your choice" control="agent" markers={markers} onFrameStaleChange={setFrameStale} />
    </div>
  )
}

<ScreenGuide>

PropTypeDefaultDescription
moderequired"question" | "walkthrough"—"question": which of the markers the person means. "walkthrough": steps, with Back and Next.
markersrequiredreadonly AgentScreenMarker[]—The markers on the screen now, the same list passed to AgentScreen's `markers`, so the numbers match.
questionstring | undefined—The question, in the app's or the agent's words. Left out, it reads "Which of these, 1 or 2?". Shown as text.
onAnswer((id: string) => void) | undefined—Question mode: called with the marker's id when the person answers. Nothing is chosen until they do. Left out, the answers are drawn locked, saying the app doesn't take an answer here.
stepsreadonly ScreenGuideStep[] | undefined—Walk-through mode: the steps, in order.
stepstring | undefined—Walk-through mode: the id of the current step, held by the app.
onStepChange((id: string) => void) | undefined—Walk-through mode: called with the step's id when the person presses Back or Next. Change `step` from here, or when the app itself sees the step done; the guide never moves on its own. Left out, Back and Next are drawn locked, saying the app moves to each step itself.
frameStaleboolean | undefined—The screen's picture is stale, lost or ended, as AgentScreen stamps it (its `onFrameStaleChange` says so): the guide says its markers may be out of date, and the question's answers are locked with those words.
classNamestring | undefined—Extra classes for the outer element.
headingLevelHeadingLevel | undefined—The level of its title. Defaults to one below the nearest `HeadingScope`, else 2.

Strings

The fixed words, in English by default. Change any of them under screenGuide in <LocaleProvider strings={{ … }}>; the table is exported as screenGuideStrings.

KeyEnglish
questionWhich of these, {options}?
noMarkersNothing is marked on the screen yet.
stepStep {n} of {total}: {instruction}
stepsSteps
backBack
nextNext
firstStepThis is the first step.
lastStepThis is the last step.
answerNotTakenThe app doesn’t take an answer here.
stepsByAppThe app moves to each step itself.
stepUnknownThe app’s current step isn’t one of these steps.
staleThe screen’s picture may be out of date, so these markers may be too.

Types

ScreenGuideStep
One step of a walk-through: its id and what the person does, in the app's words ("Click Approve").
{ id: string; /** What to do on this step, in the app's or the agent's words. Shown as text. */ instruction: string; }

Types from other items

The props above take these. Each is defined by another item; import it from the path shown.

HeadingLevel from @/lib/candor/heading-level
A heading's level in the page's outline: `1` is `<h1>`, `6` is `<h6>`. Every Candor item that draws a heading takes its `headingLevel` as this type; the level is where the heading sits in the outline, never how large it looks.
1 | 2 | 3 | 4 | 5 | 6