Gemini Live API: Real-Time Voice and Vision

▶ Watch (3:50)

Thor Schaeff opened with a live demo: a radio DJ powered by Gemini 3.1 Flash Live that accepted song requests via speech, called the Gemini API via Lyria to generate a 30-second reggaeton clip. The Live API uses a stateful websocket connection. It sends text, audio, and video frames to the model and receives audio replies plus transcripts. The model processes sound-to-sound, supporting 90 languages and interruption. It can call Google Search or custom tools like Lyria without passing through text.

Interactions API: Unified Interface for Models and Agents

▶ Watch (14:27)

Philipp Schmid presented the Interactions API as a single interface for both models and managed agents. The same client.interactions.create call works for a Gemini 3 Pro model or an anti-gravity agent, with identical structure — model ID, input, tools, and response modality. Server-side state management tracks turns via an interaction ID, eliminating client-side history arrays. The API supports text, audio (TTS with Gemini 3.1 Flash), and image generation. Tools can combine Google Search, Google Maps, and custom function schemas, all orchestrated server-side.

Managed Agents with Anti-Gravity

▶ Watch (20:36)

Philipp announced anti-gravity as a managed agent available in AI Studio and via API. Each agent gets a remote sandbox (its own computer) with code execution, file system access, web search, and agent skills. Environments are persistent and shareable between agents. A credential proxy intercepts outgoing API calls to inject keys without exposing them to the agent. Demo: one API call triggered a weather dashboard for London and Ankara. After 3 minutes, the agent returned an interactive HTML file with Celsius/Fahrenheit toggle. A second demo produced a 3JS anti-gravity game.

Creating Custom Agents via CLI and Coding Agent

▶ Watch (30:12)

Philipp showed how a coding agent (anti-gravity) can create a new managed agent. Using the Gemini API CLI skill, the coding agent parsed instructions, wrote an agent.yaml file with system prompt and environment scaffolding, and deployed the agent. The new agent — an SVG emoji generator — was then invoked with a single API call. It generated a rocket SVG, downloaded via the CLI. The same environment could be reused for follow-up turns. The demo also showed iterating on the weather dashboard by adding light mode, with the agent reading the existing files.

Notable Quotes

the model is going sound-to-sound. So, we’re actually running the transcription on both the input and the output audio Thor Schaeff · ▶ Watch (7:30)

we on the server side stack all of the context together Philipp Schmid · ▶ Watch (16:25)

every agent becomes its own small little box where it can does its work and it can come back to you Philipp Schmid · ▶ Watch (21:53)

one API call, 3 minutes of waiting, and we get our result Philipp Schmid · ▶ Watch (28:59)

Key Takeaways

  • Gemini Live API enables real-time speech-to-speech interactions with 90 languages and tool calling.
  • The Interactions API unifies model and agent access with server-side state management and no client history arrays.
  • Managed agents with anti-gravity provide persistent remote sandboxes, credential proxying, and single-API-call app generation.