# Show a computer-use or browser agent's screen

An agent that uses a computer or a browser should be watchable. AgentScreen shows its screen from what your app passes: an image URL for the latest frame, a video URL, or a MediaStream. Beside the frame it shows what the agent is doing now, as text, and that text is also the frame's alternative for a screen reader.

Nothing is drawn until a frame arrives. While `connection` is "connecting" it says so, so a person never mistakes a placeholder for the screen. Pass `frameAt`, the time the frame was captured, and a frame older than `staleAfter` is stamped "No new frame since 10:42". Without `frameAt` Candor doesn't judge how fresh the picture is. A lost connection or an ended session keeps the last frame and says it is the last one.

The simplest source is a frame per action. With a computer-use tool, your own code carries out each action and takes the screenshot the model asked for; store that image, and send the page its URL, the time it was taken and the action in your own words. Server-sent events are enough for that. For a continuous picture of a remote browser, use WebRTC and pass the MediaStream from the peer connection's `track` event.

Control, Pause, Stop and recording come from your app's state and go back through callbacks. Take over hands the screen to a person and Hand back returns it. Pause and Stop each say what happens to the step the agent is part-way through, in your words through `pauseNote` and `stopNote`, or say that you haven't said. Recording says what is recorded before it starts, says when it is about to start, and stays marked while it runs, collapsed or expanded.

Web page: https://candor.themethodic.com/patterns/agent-screen-stream

## Components

- **Agent screen** (`agent-screen`, Pro; https://candor.themethodic.com/docs/agent-screen.md): The screen, who has control, Pause and Stop with what happens to the step in flight, recording, and a stale-frame stamp (Pro).
- **Run heartbeat** (`run-heartbeat`, Pro; https://candor.themethodic.com/docs/run-heartbeat.md): The run's own state, which says when nothing has been heard from it rather than calling it running (Pro).
- **Stop all** (`stop-all`, Pro; https://candor.themethodic.com/docs/stop-all.md): Pausing every agent in the workspace at once, with what happens to the runs in flight (Pro).
- **Side panel** (`side-panel`, Pro; https://candor.themethodic.com/docs/side-panel.md): Where the collapsed screen can sit beside the thread (Pro).

## Install

```bash
npx candor-ui add agent-screen run-heartbeat
```

## Code

`use-screen-frames.ts`: The latest frame from your server's event stream, with the connection's state.

```tsx
"use client"

import * as React from "react"
import type { AgentScreenConnection, AgentScreenSource } from "@/components/candor/agent-screen"

type Frame = { url: string; at: string; action: string }

/** The latest frame of a run's screen, from your server's event stream. */
export function useScreenFrames(runId: string) {
  const [frame, setFrame] = React.useState<Frame>()
  const [connection, setConnection] = React.useState<AgentScreenConnection>("connecting")

  React.useEffect(() => {
    const events = new EventSource(`/api/runs/${runId}/screen`)
    events.addEventListener("frame", (e) => {
      setFrame(JSON.parse((e as MessageEvent<string>).data))
      setConnection("live")
    })
    events.addEventListener("end", () => {
      setConnection("ended")
      events.close()
    })
    // EventSource tries again by itself. Until a frame arrives, the screen says the connection was lost.
    events.onerror = () => setConnection((c) => (c === "ended" ? c : "lost"))
    return () => events.close()
  }, [runId])

  const source: AgentScreenSource | undefined = frame && { kind: "image", src: frame.url }
  return { source, connection, frameAt: frame?.at, action: frame?.action }
}
```

`app/api/runs/[id]/screen/route.ts`: A route that sends a frame event for each screenshot and an end event when the session closes.

```tsx
import { canWatchRun, onScreenFrame } from "@/lib/runs" // your own

export async function GET(request: Request, { params }: { params: Promise<{ id: string }> }) {
  const { id } = await params
  // Frames show whatever the agent can see: check the viewer as you would for the run itself.
  if (!(await canWatchRun(request, id))) return new Response(null, { status: 404 })

  const encoder = new TextEncoder()
  const body = new ReadableStream({
    start(controller) {
      const send = (event: string, data: unknown) => controller.enqueue(encoder.encode(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`))
      const stop = onScreenFrame(id, {
        // `capturedAt` is when the screenshot was taken on the agent's machine, not when it reached the server.
        frame: (f) => send("frame", { url: `/api/runs/${id}/frames/${f.n}`, at: f.capturedAt, action: f.action }),
        end: () => {
          send("end", {})
          controller.close()
        },
      })
      request.signal.addEventListener("abort", stop)
    },
  })
  return new Response(body, { headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" } })
}
```

`use-screen-stream.ts`: A remote browser over WebRTC: the peer connection's track and its state.

```tsx
"use client"

import * as React from "react"
import type { AgentScreenConnection } from "@/components/candor/agent-screen"

const connectionOf = (state: RTCPeerConnectionState): AgentScreenConnection =>
  state === "connected" ? "live" : state === "closed" ? "ended" : state === "failed" || state === "disconnected" ? "lost" : "connecting"

/** A remote browser's screen over WebRTC: the peer connection is yours, set up with your own signalling. */
export function useScreenStream(pc: RTCPeerConnection) {
  const [stream, setStream] = React.useState<MediaStream>()
  const [connection, setConnection] = React.useState<AgentScreenConnection>(() => connectionOf(pc.connectionState))

  React.useEffect(() => {
    const onTrack = (e: RTCTrackEvent) => setStream(e.streams[0])
    const onState = () => setConnection(connectionOf(pc.connectionState))
    pc.addEventListener("track", onTrack)
    pc.addEventListener("connectionstatechange", onState)
    return () => {
      pc.removeEventListener("track", onTrack)
      pc.removeEventListener("connectionstatechange", onState)
    }
  }, [pc])

  return { source: stream && { kind: "stream" as const, stream }, connection }
}
```

`run-screen.tsx`: AgentScreen wired to the frames and to your run's own state and calls.

```tsx
"use client"

import { AgentScreen } from "@/components/candor/agent-screen"
import { useScreenFrames } from "./use-screen-frames"
import { useRun } from "@/lib/use-run" // your own: the run's state and the calls that change it

export function RunScreen({ runId }: { runId: string }) {
  const screen = useScreenFrames(runId)
  const run = useRun(runId)
  return (
    <AgentScreen
      agent={run.agent}
      {...screen}
      control={run.controlledBy === "agent" ? "agent" : "person"}
      person={run.controlledBy === "viewer" || run.controlledBy === "agent" ? undefined : run.controlledBy}
      onTakeOver={run.takeOver}
      onHandBack={run.controlledBy === "viewer" ? run.handBack : undefined}
      run={run.state}
      onPause={run.pause}
      onResume={run.resume}
      onStop={run.stop}
      pauseNote="It finishes the action it's on, then waits. Nothing else is clicked or typed."
      stopNote="It stops after the action it's on. Pages it opened stay as they are; nothing is undone."
      recording={run.recording}
      recordingSince={run.recordingSince}
      recordingNote="The screen is recorded, with no sound. Recordings are kept for 30 days and your team can watch them."
      onRecord={run.startRecording}
      onStopRecording={run.stopRecording}
    />
  )
}
```

## Notes

- Left to the app: capturing the screen, storing frames, the signalling for WebRTC, who may watch a run, pausing the agent while a person has control, and storing recordings. Candor sends no input to the remote screen. After Take over, the person uses whatever control surface your app provides, such as a remote desktop view.
- Frames show everything the agent can see, which can include personal data. Serve them from your own origin behind the same check as the run, and keep them no longer than you need.
- Say in `recordingNote` what is recorded, how long it is kept and who can watch it. Whether you need the agreement of the people whose data appears on screen is for you to decide.
- Set `frameAt` from the machine that took the screenshot, not from when it reached your server, so the stamp says how old the picture really is.