AI
The Gemini Interactions API: A Unified Surface for Models and Agents
Gemini launched the Interactions API in December 2024 as a beta replacement for generateContent. The core change: conversation state lives on the server. Pass a previous_interaction_id and the server appends new turns without a client-side history array. Agents running multi-minute tasks can use asynchronous execution via polling or webhooks, so HTTP connections stay short.
The unified surface covers both models and built-in agents. One call can run Deep Research, then hand the output to Imagen, all in four lines of code. Server-held context also keeps caching intact. Any whitespace edit in a client-managed prompt breaks the cache. Startups using the Interactions API today see
“two to three times better cache rates” — Philipp Schmid
Agent Skills: Giving Coding Agents Up-to-Date API Knowledge
Agent skills are Markdown instruction files installed into an IDE agent’s agents/skills/ folder. Schmid’s rule: a skill belongs there when it covers
“either something the model cannot do” — Philipp Schmid
reliably on its own, or a workflow preference you always enforce. For the Interactions API skill, the team’s fix for a recurring problem was simple: agents kept defaulting to Gemini 1.5 because the model had never seen the newer API in training. The skill names the available models, but rather than embedding the full reference, it links to docs published as Markdown. Coding agents fetch those pages at runtime, so when a new feature ships, the skill stays current without a reinstall.
Building a Coding Agent Live: Tools, the Agentic Loop, and System Prompts
Schmid built a Python coding agent live using only IDE agent prompts. In the coding agent built live with anti-gravity (25:03), the agent read the Interactions API skill, scaffolded the class with genai client and previous_interaction_id, added read/write file tools with JSON schemas, and wired the agentic loop. When the model ignored the write-file tool, the fix was one prompt: add a system instruction identifying it as a coding agent with file-system access.
Gemini 2.5 Flash produced correct Interactions API code despite the model predating the API’s release. The Interactions API launched in December 2024; the model’s training cutoff predates that. The skill, not training data, was the sole knowledge source.
Gemini Live API: Native Audio, the Live Jukebox, and Real-Time Multimodal Input
Gemini 3.1 Flash Live launched two weeks before this workshop. Schaeff ran the Live Jukebox DJ demo (51:04), a conversational voice agent that calls LIA 3 to produce custom 30-second songs, including a German techno-schlager about the UK AI scene. The model is native audio: it goes sound token to sound token without a text step, which is what enables 97-language preview support and natural code-switching between German and English.
Voice personality comes from system instructions alone. Thirty base voices exist; accent and register are shaped through prompting. Thinking levels are also configurable: lower thinking cuts latency, higher thinking improves tool-use accuracy.
Client vs. Server Architecture and Ephemeral Tokens
The Live API has two connection patterns. In the server-to-server setup, the backend proxies audio and video chunks between the browser and Gemini, adding latency. The faster option: generate an ephemeral token on the server and send it to the client, which opens the WebSocket directly. Ephemeral tokens are short-lived by design, so a leak expires quickly.
Without context window compression, audio-only sessions run 15 minutes; audio-video sessions cap at 2 minutes. Compression adds a sliding window that discards earlier context as the conversation grows. One image frame costs roughly 1,200 tokens, so sending frames only when the visual changes keeps context cost down.
Production Readiness: Limitations, Real Deployments, and Evaluation
Shopify Sidekick and Waymo are in production with the Live API. Hey Ado, an Argentine startup, uses it for Spanish-language voice companions for elderly users. Schaeff’s honest take for complex cases: the cascading pipeline (transcribe, LLM, TTS) wins on observability. With native audio you cannot intercept the response before it is spoken, and compliance requirements like SOC 2 may require significant work on top.
Speaker diarization is not available yet. The model cannot be told to ignore specific voices. Session transcripts are not stored by Google; partners like LiveKit and Pipecat offer that. For hallucination control: detailed system instructions, strict tool definitions, and explicit guardrails.
Q&A
Are there real business use cases for the Live API already in production? Shopify Sidekick, Waymo, and Hey Ado are live; for complex enterprise cases with observability requirements, a cascading pipeline is still more practical. ▶ Watch (1:26:57)
Are session transcripts retrievable from the Live API? Google does not store transcripts; you must store them yourself or use integration partners like LiveKit or Pipecat, which offer full audio and transcript storage. ▶ Watch (1:30:25)
How do you evaluate live voice apps for production? Evaluation complexity depends heavily on your compliance requirements (HIPAA, SOC 2) and guardrail needs; partner integrations like LiveKit provide the observability tooling production use cases need. ▶ Watch (1:39:23)
How do clients handle hallucinations in production voice agents? Detailed system instructions, explicit guardrails, and well-defined tool schemas reduce hallucinations significantly; the demos shown lacked these, which explains the failures. ▶ Watch (1:45:16)
Notable Quotes
two to three times better cache rates Philipp Schmid · ▶ Watch (14:03)
either something the model cannot do Philipp Schmid · ▶ Watch (23:21)
going through text. Thor Schaeff · ▶ Watch (57:10)
Key Takeaways
- The Interactions API stores conversation state server-side, cutting cache costs 2-3x compared to client-managed history.
- Agent skills work best when they link to live docs rather than embedding full API reference, keeping knowledge current automatically.
- Gemini 3.1 Flash Live processes audio natively without a text pipeline, enabling 97-language support and lower latency.