Skip to content
Tier

Agent screen

Pro

A live view of a computer-use or browser agent's screen, from frames or a stream the app passes. It says who has control: the agent, or a person after Take over, with Hand back to return it. Pause and Stop each say what happens to the step the agent is part-way through, in the app's words (pauseNote, stopNote), or that the app hasn't said. The agent's current action is shown as text beside the frame and is the frame's text alternative.

Nothing is drawn as a frame until one arrives: while connection is "connecting" it says so. With frameAt, a frame older than staleAfter (15 seconds unless set) is stamped "No new frame since 10:42", as RunHeartbeat does for a run; without it, Candor doesn't judge freshness. A lost connection or an ended session keeps the last frame, labelled as the last one.

Recording says so before it starts (beside Record, recordingNote or "Nothing is recorded until you press Record"), while it is about to start, and while it runs, in both views. Collapsed, it is one row with a thumbnail, the action and the badges; expanded, the frame with the action and controls beside it. Candor sends no input to the remote screen: a person who takes over uses whatever control surface the app provides.

Markdown
Version 5.2.0· other versions

This page is for 5.2.0, the current release. An earlier release’s page is the Markdown that release published, so the docs match the version you have installed (candor-lock.json records it).

Not in the docs of 1.3.0, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.5.1, 2.6.0, 2.6.1, 2.6.2, 2.7.0 and 2.8.0: the page came later, or had another name then.

Docs pages weren’t kept for 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0 and 1.2.1: they are kept from 1.3.0 on, and before that only llms.txt was published.

Preview

Loading the preview

Install

With the Candor CLI

First time? Get started sets up the registry and theme. Or with shadcn directly: npx shadcn@latest add @candor/agent-screen

Brings in: agent-avatar, bidi, button, chat, heading-level, heading-scope, locale, locked, relative-time, tokens, utils · npm: lucide-react@^1.67.0

Usage

TSX · The example above
"use client"

import * as React from "react"
import type { AgentIdentity } from "@/lib/candor/chat"
import { AgentScreen, type AgentScreenRecording, type AgentScreenRun } from "@/components/candor/agent-screen"

const agent: AgentIdentity = { id: "a-web", name: "Browser agent", role: "dev" }

// A drawn stand-in for the screenshot your server sends after each action.
const frame = (field: number) =>
  "data:image/svg+xml," +
  encodeURIComponent(
    `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 400"><rect width="640" height="400" fill="#f6f6f4"/><rect width="640" height="36" fill="#e4e4e0"/><circle cx="20" cy="18" r="6" fill="#c9c9c4"/><circle cx="40" cy="18" r="6" fill="#c9c9c4"/><rect x="70" y="9" width="420" height="18" rx="9" fill="#fff"/><text x="40" y="82" font-family="sans-serif" font-size="20" fill="#333">New invoice</text>${[0, 1, 2]
      .map((i) => `<rect x="40" y="${110 + i * 60}" width="360" height="36" rx="4" fill="#fff" stroke="${i === field ? "#3b6fd8" : "#ccc"}" stroke-width="${i === field ? 2 : 1}"/>`)
      .join("")}<rect x="40" y="300" width="120" height="36" rx="4" fill="#3b6fd8"/><text x="70" y="324" font-family="sans-serif" font-size="15" fill="#fff">Submit</text></svg>`
  )

const actions = ["Typing the customer's name", "Typing the billing address", "Choosing the due date"]

export default function Example() {
  const [step, setStep] = React.useState(0)
  const [frameAt, setFrameAt] = React.useState(() => new Date().toISOString())
  const [control, setControl] = React.useState<"agent" | "person">("agent")
  const [run, setRun] = React.useState<AgentScreenRun>("running")
  const [recording, setRecording] = React.useState<AgentScreenRecording>("off")
  const [recordingSince, setRecordingSince] = React.useState<string>()

  // A new frame every four seconds while the agent works. Paused, stopped or taken over, none arrives, and after 15
  // seconds the frame is stamped as not new.
  React.useEffect(() => {
    if (run !== "running" || control !== "agent") return
    const id = setInterval(() => {
      setStep((s) => (s + 1) % actions.length)
      setFrameAt(new Date().toISOString())
    }, 4000)
    return () => clearInterval(id)
  }, [run, control])

  return (
    <div className="flex max-w-3xl flex-col gap-6">
      <AgentScreen
        agent={agent}
        connection="live"
        source={{ kind: "image", src: frame(step) }}
        frameAt={frameAt}
        action={run === "stopped" ? undefined : actions[step]}
        control={control}
        onTakeOver={() => setControl("person")}
        onHandBack={() => setControl("agent")}
        run={run}
        onPause={() => setRun("paused")}
        onResume={() => setRun("running")}
        onStop={() => setRun("stopped")}
        pauseNote="It finishes typing into the field it's in, then waits. Nothing is submitted."
        stopNote="It stops now. The form stays open with what's been typed; nothing is submitted."
        recording={recording}
        recordingSince={recordingSince}
        recordingNote="The screen is recorded, not your microphone. Recordings are kept for 30 days and your team can watch them."
        onRecord={() => {
          // Your app says "starting" until the first recorded frame, then "on".
          setRecording("starting")
          setTimeout(() => {
            setRecording("on")
            setRecordingSince(new Date().toISOString())
          }, 1200)
        }}
        onStopRecording={() => setRecording("off")}
      />
      {/* Collapsed in a side panel, before the first frame: it says it's connecting and draws no picture. */}
      <AgentScreen className="max-w-sm" agent={{ id: "a-research", name: "Research agent", role: "analyst" }} connection="connecting" defaultView="collapsed" action="Opening the supplier's site" />
    </div>
  )
}

<AgentScreen>

PropTypeDefaultDescription
agentrequiredAgentIdentity—The agent whose screen this is, named by the app.
sourceAgentScreenSource | undefined—What to draw: an image URL for the latest frame, a video URL, or a MediaStream. Leave it out until there is one.
connectionrequiredAgentScreenConnection—The connection to the screen: "connecting" (no frame is drawn), "live", "lost" or "ended".
frameAtDate | string | undefined—When the latest frame was captured, from the app or the stream. Without it, freshness isn't judged.
staleAfternumber | undefinedFRAME_STALE_AFTERHow old a live frame may be before it's stamped as not new, in milliseconds. Defaults to 15 seconds.
actionstring | undefined—What the agent is doing now, in the app's words: "Clicking Submit on the invoice form". Shown as text, never markup.
control"agent" | "person" | undefined"agent"Who has control of the screen: "agent" (the default) or "person" after Take over.
personstring | undefined—The person in control, as the app names them, when it isn't the viewer. Left out, it reads "You have control".
onTakeOver(() => void) | undefined—Shows Take over while the agent has control; called when it's pressed.
takeOverLockedstring | null | undefined—Why the viewer can't take over, if they can't. The button stays visible and says so. `null` or unset means they can.
onHandBack(() => void) | undefined—Shows Hand back while a person has control; called when it's pressed.
runAgentScreenRun | undefined"running"Where the agent's run is, as the app reports it. Defaults to "running".
onPause(() => void) | undefined—Shows Pause while running; called when it's pressed.
onResume(() => void) | undefined—Shows Resume while paused; called when it's pressed.
onStop(() => void) | undefined—Shows Stop until the run has stopped or finished; called when it's pressed.
pauseNotestring | undefined—What pausing does to the step in flight, in the app's words: "It finishes typing the address, then waits."
stopNotestring | undefined—What stopping does to the step in flight, in the app's words: "It stops now; the form may be half filled in."
recordingAgentScreenRecording | undefined"off"Recording, as the app reports it: "off" (the default), "starting" or "on".
recordingNotestring | undefined—What is recorded and who can see it, in the app's words. Shown beside Record before it starts, and while it runs.
recordingSinceDate | string | undefined—When recording started, from the app.
onRecord(() => void) | undefined—Shows Record while nothing is recorded; called when it's pressed. Report the change through `recording`.
onStopRecording(() => void) | undefined—Shows Stop recording while recording; called when it's pressed.
view"collapsed" | "expanded" | undefined—"collapsed" or "expanded", when the app controls the view.
defaultView"collapsed" | "expanded" | undefined"expanded"The view to start in when the app doesn't control it. Defaults to "expanded".
onViewChange((view: "collapsed" | "expanded") => void) | undefined—Called with the view the person asked for.
aspectRationumber | undefinedSCREEN_ASPECTThe screen's width over its height, so the frame keeps its shape while connecting. Defaults to 16 / 10.
classNamestring | undefined—Extra classes for the outer element.
headingLevelHeadingLevel | undefined—The level of the title shown when expanded. Defaults to one below the nearest `HeadingScope`, else 2.

Strings

The fixed words, in English by default. Change any of them under agentScreen in <LocaleProvider strings={{ … }}>; the table is exported as agentScreenStrings.

KeyEnglish
label{agent}'s screen
frameAlt{agent}'s screen. Now: {action}
frameAltNoAction{agent}'s screen
nowNow
noActionNo current action reported.
connectingConnecting to the screen. Nothing is shown until the first frame arrives.
connectingShortConnecting
noFrameNo frame has arrived yet.
staleNo new frame since {time}
staleUnreadableThis frame's time can't be read, so it may be out of date.
lostConnection lost. Last frame {time}
lostNoTimeConnection lost. This is the last frame received.
lostNoFrameConnection lost before any frame arrived.
endedThe session has ended. Last frame {time}
endedNoTimeThe session has ended. This is the last frame received.
endedNoFrameThe session has ended.
agentControl{agent} has control
youControlYou have control
personControl{person} has control
takeOverTake over
handBackHand back to {agent}
pausePause
resumeResume
stopStop
ifPauseIf you pause
ifStopIf you stop
inFlightUnknownThe app hasn't said what happens to the step in flight.
pausingPausing
pausedPaused
stoppingStopping
stoppedStopped
finishedFinished
recordRecord
stopRecordingStop recording
recordNoteNothing is recorded until you press Record.
recordingStartingRecording is about to start
recordingRecording
recordingSinceRecording since {time}
recordedWhat's recorded
expandExpand the screen
collapseCollapse the screen

Types

AgentScreenSource
What the screen is drawn from: an image URL per frame (a screenshot after each action), a video URL the browser can play, or a `MediaStream` (from WebRTC, say).
| { kind: "image"; src: string } | { kind: "video"; src: string } | { kind: "stream"; stream: MediaStream }
AgentScreenConnection
The connection to the screen, as the app knows it.
"connecting" | "live" | "lost" | "ended"
AgentScreenRun
Where the agent's run is, as the app reports it. "pausing" and "stopping" are asked for and not yet done.
"running" | "pausing" | "paused" | "stopping" | "stopped" | "finished"
AgentScreenRecording
Recording, as the app reports it: "starting" is the moment before the first recorded frame.
"off" | "starting" | "on"

Types from other items

The props above take these. Each is defined by another item; import it from the path shown.

AgentIdentity from @/lib/candor/chat
{ id: string; /** The displayed name, e.g. "Architect agent". Never derived from model output. */ name: string; role: AgentRole; specialty?: string | undefined; }
HeadingLevel from @/lib/candor/heading-level
A heading's level in the page's outline: `1` is `<h1>`, `6` is `<h6>`. Every Candor item that draws a heading takes its `headingLevel` as this type; the level is where the heading sits in the outline, never how large it looks.
1 | 2 | 3 | 4 | 5 | 6
AgentRole from @/lib/candor/chat
What an agent does: "assistant", "coder", "reviewer"… The app registers each role's glyph and hue (AgentRolesProvider).
string