# Agent screen

A live view of a computer-use or browser agent's screen, from frames or a stream the app passes. It says who has control: the agent, or a person after Take over, with Hand back to return it. Pause and Stop each say what happens to the step the agent is part-way through, in the app's words (`pauseNote`, `stopNote`), or that the app hasn't said. The agent's current action is shown as text beside the frame and is the frame's text alternative.

Nothing is drawn as a frame until one arrives: while `connection` is "connecting" it says so. With `frameAt`, a frame older than `staleAfter` (15 seconds unless set) is stamped "No new frame since 10:42", as RunHeartbeat does for a run; without it, Candor doesn't judge freshness. A lost connection or an ended session keeps the last frame, labelled as the last one.

Recording says so before it starts (beside Record, `recordingNote` or "Nothing is recorded until you press Record"), while it is about to start, and while it runs, in both views. Collapsed, it is one row with a thumbnail, the action and the badges; expanded, the frame with the action and controls beside it. Candor sends no input to the remote screen: a person who takes over uses whatever control surface the app provides.

- Registry name: `agent-screen` (`@candor/agent-screen`)
- Tier: Pro. Installing it needs a Pro, Premium or Team licence key: the person runs `npx candor-ui key <key>` themselves; never ask for the key in chat or write it to a file.
- Docs: https://candor.themethodic.com/docs/agent-screen

## Install

```bash
npx candor-ui add agent-screen
```

Run `npx candor-ui@latest init` once first (https://candor.themethodic.com/get-started.md). Without the CLI: `npx shadcn@latest add @candor/agent-screen`.
It brings in: agent-avatar, button, chat, locale, locked, relative-time, tokens, utils.
npm packages: lucide-react@^1.67.0.

## Example

```tsx
"use client"

import * as React from "react"
import type { AgentIdentity } from "@/lib/chat"
import { AgentScreen, type AgentScreenRecording, type AgentScreenRun } from "@/components/agent-screen"

const agent: AgentIdentity = { id: "a-web", name: "Browser agent", role: "dev" }

// A drawn stand-in for the screenshot your server sends after each action.
const frame = (field: number) =>
  "data:image/svg+xml," +
  encodeURIComponent(
    `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 400"><rect width="640" height="400" fill="#f6f6f4"/><rect width="640" height="36" fill="#e4e4e0"/><circle cx="20" cy="18" r="6" fill="#c9c9c4"/><circle cx="40" cy="18" r="6" fill="#c9c9c4"/><rect x="70" y="9" width="420" height="18" rx="9" fill="#fff"/><text x="40" y="82" font-family="sans-serif" font-size="20" fill="#333">New invoice</text>${[0, 1, 2]
      .map((i) => `<rect x="40" y="${110 + i * 60}" width="360" height="36" rx="4" fill="#fff" stroke="${i === field ? "#3b6fd8" : "#ccc"}" stroke-width="${i === field ? 2 : 1}"/>`)
      .join("")}<rect x="40" y="300" width="120" height="36" rx="4" fill="#3b6fd8"/><text x="70" y="324" font-family="sans-serif" font-size="15" fill="#fff">Submit</text></svg>`
  )

const actions = ["Typing the customer's name", "Typing the billing address", "Choosing the due date"]

export default function Example() {
  const [step, setStep] = React.useState(0)
  const [frameAt, setFrameAt] = React.useState(() => new Date().toISOString())
  const [control, setControl] = React.useState<"agent" | "person">("agent")
  const [run, setRun] = React.useState<AgentScreenRun>("running")
  const [recording, setRecording] = React.useState<AgentScreenRecording>("off")
  const [recordingSince, setRecordingSince] = React.useState<string>()

  // A new frame every four seconds while the agent works. Paused, stopped or taken over, none arrives, and after 15
  // seconds the frame is stamped as not new.
  React.useEffect(() => {
    if (run !== "running" || control !== "agent") return
    const id = setInterval(() => {
      setStep((s) => (s + 1) % actions.length)
      setFrameAt(new Date().toISOString())
    }, 4000)
    return () => clearInterval(id)
  }, [run, control])

  return (
    <div className="flex max-w-3xl flex-col gap-6">
      <AgentScreen
        agent={agent}
        connection="live"
        source={{ kind: "image", src: frame(step) }}
        frameAt={frameAt}
        action={run === "stopped" ? undefined : actions[step]}
        control={control}
        onTakeOver={() => setControl("person")}
        onHandBack={() => setControl("agent")}
        run={run}
        onPause={() => setRun("paused")}
        onResume={() => setRun("running")}
        onStop={() => setRun("stopped")}
        pauseNote="It finishes typing into the field it's in, then waits. Nothing is submitted."
        stopNote="It stops now. The form stays open with what's been typed; nothing is submitted."
        recording={recording}
        recordingSince={recordingSince}
        recordingNote="The screen is recorded, not your microphone. Recordings are kept for 30 days and your team can watch them."
        onRecord={() => {
          // Your app says "starting" until the first recorded frame, then "on".
          setRecording("starting")
          setTimeout(() => {
            setRecording("on")
            setRecordingSince(new Date().toISOString())
          }, 1200)
        }}
        onStopRecording={() => setRecording("off")}
      />
      {/* Collapsed in a side panel, before the first frame: it says it's connecting and draws no picture. */}
      <AgentScreen className="max-w-sm" agent={{ id: "a-research", name: "Research agent", role: "analyst" }} connection="connecting" defaultView="collapsed" action="Opening the supplier's site" />
    </div>
  )
}
```

## <AgentScreen>

Props:

- `agent: AgentIdentity`: The agent whose screen this is, named by the app.
- `source?: AgentScreenSource | undefined`: What to draw: an image URL for the latest frame, a video URL, or a MediaStream. Leave it out until there is one.
- `connection: AgentScreenConnection`: The connection to the screen: "connecting" (no frame is drawn), "live", "lost" or "ended".
- `frameAt?: Date | string | undefined`: When the latest frame was captured, from the app or the stream. Without it, freshness isn't judged.
- `staleAfter?: number | undefined` (default `FRAME_STALE_AFTER`): How old a live frame may be before it's stamped as not new, in milliseconds. Defaults to 15 seconds.
- `action?: string | undefined`: What the agent is doing now, in the app's words: "Clicking Submit on the invoice form". Shown as text, never markup.
- `control?: "agent" | "person" | undefined` (default `"agent"`): Who has control of the screen: "agent" (the default) or "person" after Take over.
- `person?: string | undefined`: The person in control, as the app names them, when it isn't the viewer. Left out, it reads "You have control".
- `onTakeOver?: (() => void) | undefined`: Shows Take over while the agent has control; called when it's pressed.
- `takeOverLocked?: string | undefined`: Why the viewer can't take over, if they can't. The button stays visible and says so.
- `onHandBack?: (() => void) | undefined`: Shows Hand back while a person has control; called when it's pressed.
- `run?: AgentScreenRun | undefined` (default `"running"`): Where the agent's run is, as the app reports it. Defaults to "running".
- `onPause?: (() => void) | undefined`: Shows Pause while running; called when it's pressed.
- `onResume?: (() => void) | undefined`: Shows Resume while paused; called when it's pressed.
- `onStop?: (() => void) | undefined`: Shows Stop until the run has stopped or finished; called when it's pressed.
- `pauseNote?: string | undefined`: What pausing does to the step in flight, in the app's words: "It finishes typing the address, then waits."
- `stopNote?: string | undefined`: What stopping does to the step in flight, in the app's words: "It stops now; the form may be half filled in."
- `recording?: AgentScreenRecording | undefined` (default `"off"`): Recording, as the app reports it: "off" (the default), "starting" or "on".
- `recordingNote?: string | undefined`: What is recorded and who can see it, in the app's words. Shown beside Record before it starts, and while it runs.
- `recordingSince?: Date | string | undefined`: When recording started, from the app.
- `onRecord?: (() => void) | undefined`: Shows Record while nothing is recorded; called when it's pressed. Report the change through `recording`.
- `onStopRecording?: (() => void) | undefined`: Shows Stop recording while recording; called when it's pressed.
- `view?: "collapsed" | "expanded" | undefined`: "collapsed" or "expanded", when the app controls the view.
- `defaultView?: "collapsed" | "expanded" | undefined` (default `"expanded"`): The view to start in when the app doesn't control it. Defaults to "expanded".
- `onViewChange?: ((view: "collapsed" | "expanded") => void) | undefined`: Called with the view the person asked for.
- `aspectRatio?: number | undefined` (default `SCREEN_ASPECT`): The screen's width over its height, so the frame keeps its shape while connecting. Defaults to 16 / 10.
- `className?: string | undefined`

## Strings (`agentScreen`)

The fixed words, English by default. Override any of them under `agentScreen` in `<LocaleProvider strings={{ agentScreen: { … } }}>`; the table is exported as `agentScreenStrings`.

- `label`: {agent}'s screen
- `frameAlt`: {agent}'s screen. Now: {action}
- `frameAltNoAction`: {agent}'s screen
- `now`: Now
- `noAction`: No current action reported.
- `connecting`: Connecting to the screen. Nothing is shown until the first frame arrives.
- `connectingShort`: Connecting
- `noFrame`: No frame has arrived yet.
- `stale`: No new frame since {time}
- `staleUnreadable`: This frame's time can't be read, so it may be out of date.
- `lost`: Connection lost. Last frame {time}
- `lostNoTime`: Connection lost. This is the last frame received.
- `lostNoFrame`: Connection lost before any frame arrived.
- `ended`: The session has ended. Last frame {time}
- `endedNoTime`: The session has ended. This is the last frame received.
- `endedNoFrame`: The session has ended.
- `agentControl`: {agent} has control
- `youControl`: You have control
- `personControl`: {person} has control
- `takeOver`: Take over
- `handBack`: Hand back to {agent}
- `pause`: Pause
- `resume`: Resume
- `stop`: Stop
- `ifPause`: If you pause
- `ifStop`: If you stop
- `inFlightUnknown`: The app hasn't said what happens to the step in flight.
- `pausing`: Pausing
- `paused`: Paused
- `stopping`: Stopping
- `stopped`: Stopped
- `finished`: Finished
- `record`: Record
- `stopRecording`: Stop recording
- `recordNote`: Nothing is recorded until you press Record.
- `recordingStarting`: Recording is about to start
- `recording`: Recording
- `recordingSince`: Recording since {time}
- `recorded`: What's recorded
- `expand`: Expand the screen
- `collapse`: Collapse the screen

## Types

### AgentScreenSource

What the screen is drawn from: an image URL per frame (a screenshot after each action), a video URL the browser can play, or a `MediaStream` (from WebRTC, say).

```ts
| { kind: "image"; src: string }
  | { kind: "video"; src: string }
  | { kind: "stream"; stream: MediaStream }
```

### AgentScreenConnection

The connection to the screen, as the app knows it.

```ts
"connecting" | "live" | "lost" | "ended"
```

### AgentScreenRun

Where the agent's run is, as the app reports it. "pausing" and "stopping" are asked for and not yet done.

```ts
"running" | "pausing" | "paused" | "stopping" | "stopped" | "finished"
```

### AgentScreenRecording

Recording, as the app reports it: "starting" is the moment before the first recorded frame.

```ts
"off" | "starting" | "on"
```

## Types from other items

The props above take these. Each is defined by another item; import it from the path shown.

### AgentIdentity

Import: `import type { AgentIdentity } from "@/lib/chat"` (item `chat`, https://candor.themethodic.com/docs/chat.md)

```ts
{
  id: string;
  /** The displayed name, e.g. "Architect agent". Never derived from model output. */
  name: string;
  role: AgentRole;
  specialty?: string | undefined;
}
```

### AgentRole

Import: `import type { AgentRole } from "@/lib/chat"` (item `chat`, https://candor.themethodic.com/docs/chat.md)

What an agent does: "assistant", "coder", "reviewer"… The app registers each role's glyph and hue (AgentRolesProvider).

```ts
string
```

## Rules for using Candor UI

- Components take plain typed data. Never pass model output as HTML or render it as a link; messages are typed blocks.
- A control that can't be used says why: wrap it in `<Locked reason="…">`, don't just disable it.
- Don't show a value you don't have: leave it out (it renders as —) rather than guess. An estimate is labelled as one.
- Nothing is pre-selected in a decision a person makes, and the agent is named from your app's data, never from model output.
- Your app's words go in `LocaleProvider` (fixed strings) and `TermsProvider` (nouns); entity kinds and agent roles in `EntityKindsProvider` and `AgentRolesProvider`.