Skip to main content

Live Mode: Realtime Voice in Chat

Live mode lets you talk to an agent in realtime voice, inside the same chat you were just typing in. It is a thin orchestration layer over two proven pipelines — the standard text agent and the realtime voice service — both reading and writing the same chat history, so context carries across every mode switch.

You type  ──text turn──▶  Agent (streaming reply)
                              ▲
You speak ──WebRTC─────▶  LiveKit room ──▶ same agent, same history

Where to find it​

Live mode is built into the chat screen — there is nothing to configure:

  • Voice agents (mode voice) show a Start Live Mode banner at the top of an empty chat.
  • All other agents show a small live mode icon next to the microphone button in the message composer, provided the agent has audio generation enabled (see below).
Region availability

Live mode is currently enabled in the Australia (AU) region. In EU and US regions the entry points are hidden — text chat still works normally.

Who can use Live mode​

Agent modeLive mode entry pointRequirement
voiceStart Live Mode bannerVoice agent configured
All other modes (operator, assistant, programmer, orchestrator, desktop)Live mode icon in the composerEnable audio generation turned ON for the agent

Live voice is a voice feature: for non-voice agents, the Enable audio generation capability (allowAudiosGeneration) gates the live mode entry point. If the icon is missing on a non-voice agent, enable audio generation in the agent's settings (see Agent capabilities).

Using Live mode​

Starting a voice segment​

  1. Click the live mode icon (or the Start Live Mode banner on voice agents)
  2. Pick your microphone and speaker in the confirmation dialog
  3. Click start — the agent connects and begins listening

The icon becomes a live status indicator showing the agent's current state: starting, listening, thinking or speaking.

Talking with the agent​

  • Interrupt any time — speak over the agent to barge in, just like a phone conversation
  • Every spoken turn is transcribed and persisted to the chat
  • The agent loads the full conversation history when the voice segment starts, so it continues the same context as your earlier text turns

Sending files during a voice segment​

The attachment button keeps working while voice is live. Attachments are context, not a turn — they never interrupt the voice segment:

  • Files are uploaded through the normal upload flow and registered on the chat
  • The agent learns about the attachment in realtime and can read it with its read_attachment tool — simply ask "what do you make of the document I just sent?"
  • Images are analysed visually, audio files are transcribed, videos are analysed, and documents are read as text

Switching back to text​

Click the live status icon to stop the voice segment. This releases the mode lock immediately and the text composer becomes active again — the conversation continues in the same chat, with the full voice transcript visible in order.

One mode at a time

Only one mode (text or voice) is active per chat. While a voice segment is live, sending a text message is blocked in the app (the backend returns 409 SESSION_MODE_LOCKED). Stop the voice segment first.

If the icon won't engage

If another response is still completing, the mode lock may briefly hold — try again in a moment. Abandoned voice locks (for example after a crash) self-heal automatically after 30 minutes.

Switching chats while live​

If you switch to a different chat while a voice segment is active, the voice segment stops automatically — each live session is bound to its chat.

The unified transcript​

Every turn — text, voice (both sides) and attachment markers — is stored in one ordered transcript with per-message agent attribution. The chat view renders this continuously regardless of which mode produced each message, and the transcript remains available after the session ends.

Debugging live sessions​

During a voice segment you can open the Live Mode Monitor from the debug panel. It shows:

  • Connection status (connected/disconnected)
  • Agent state in realtime (listening / thinking / speaking)
  • Live transcripts as they arrive
  • Measured AI response latency (time from end of your speech to the start of the agent's reply)

How it maps to the Live Sessions API​

Everything the app does in Live mode is available over the public Live Sessions API — build the same mixed text-and-voice experience into your own applications:

In the appAPI equivalent
Opening a chat (session adoption)POST /live with chat_id
Sending a messagePOST /live/{id}/text
Starting / stopping a voice segmentPOST /live/{id}/voice/start / voice/stop
Attaching filesPOST /live/{id}/attachments
Reading the chatGET /live/{id}/transcript
Closing the sessionPOST /live/{id}/end

The full integration guide with code samples is at Live Sessions API Integration, and the OpenAPI reference is published on apidocs.toothfairyai.com (the TF Live Sessions spec).