Building Real-Time Conversational Agents with ElevenLabs

▶ Watch (03:07)

ElevenLabs builds voice models for generating lifelike speech. At GitHub Universe, Thor Schaeff showed how its CLI initializes a conversational agent in seconds: one config file sets the LLM, voice ID, and system prompt. The demo agent answered questions live. When asked for San Francisco’s weather, it failed, because the underlying LLM had no real-time data access. That gap set up the rest of the demo: give the agent tools, and swap the default voice for something better.

Using GitHub Copilot Coding Agent to Iterate on Agent Configuration

▶ Watch (06:50)

Rather than edit the config manually, Schaeff filed two GitHub Issues and assigned them to GitHub Copilot Coding Agent. The voice swap via Copilot Coding Agent (06:56) produced a draft PR within minutes that swapped in his personal voice clone. The real-time weather tool added by Copilot (07:37) went further: Copilot wrote a get-weather tool using latitude and longitude parameters against an open-meteo endpoint, then opened a second PR.

“we want the hard work to be done by Copilot” — Thor Schaeff

Repository Structure: Agent Configs, Copilot Instructions, and GitHub Actions

▶ Watch (11:04)

The repo holds two things Copilot needs to do this work: agent configuration files that define each conversational agent’s models and prompts, and a .github/copilot-instructions file that explains how the ElevenLabs CLI works. GitHub Actions workflows sit alongside them. When Copilot opens a PR and it gets merged, the Action calls the ElevenLabs CLI to push the updated configuration to the live agent in the background.

Merging PRs and Triggering Automated Deployment

▶ Watch (12:38)

Schaeff reviewed both Copilot PRs and merged them live. The voice change was straightforward. The weather tool took more work: Copilot updated the agent’s system prompt and defined a get_weather tool schema with string-type latitude and longitude fields the LLM fills at runtime.

“that was a bit more impressive than the The Voice ID” — Thor Schaeff

After merging, the GitHub Action ran and deployed both updates. The rebooted agent responded in his personal voice clone. The weather tool ran out of time to demonstrate, but the deployment pipeline completed successfully.

Notable Quotes

we want the hard work to be done by Copilot Thor Schaeff · ▶ 7:53

what we can do is we can build conversational agents Thor Schaeff · ▶ 3:10

that was a bit more impressive than the The Voice ID Thor Schaeff · ▶ 13:46

Key Takeaways

  • ElevenLabs CLI initializes voice agents with configurable models, voices, and prompts in one command.
  • GitHub Copilot Coding Agent handles configuration changes as PRs, triggered by plain-English GitHub Issues.
  • GitHub Actions reads merged PRs and calls the ElevenLabs CLI to deploy updated agents automatically.