The King Use Case: Agents

▶ Watch (0:02)

Text-to-speech now powers chat agents. A central LLM handles text-to-text. Speech-to-text feeds the LLM. TTS voices the reply. Latency is key. Speech-to-text runs in real time to detect turn end. TTS must emit first audio packet as soon as the LLM outputs the first token. Perceived latency drops even if full audio computation finishes 60 seconds later. A demo showed Paul, a cloned voice, answering questions about conference sessions. The audio packet arrived before the sentence completed.

Audio Codecs: Cutting 200,000 Bits to 500 Tokens

▶ Watch (0:10)

Raw speech is 200,000 bits per second. A subtitle track carries only 15 bits per second. TTS codecs compress audio to a few thousand bits per second. Mistral splits audio into 80-millisecond frames and encodes each frame into 37 tokens. That yields 500 tokens per second. Training uses reconstruction losses and adversarial losses. Some tokens are forced to carry text information so the model can reconstruct words from them.

The Dominant Pattern: Backbone Per Frame

▶ Watch (0:14)

Most labs use an autoregressive decoder backbone that generates one frame of audio at each step. A smaller model, often a decoder-only transformer, produces all tokens of that frame in one go. Mistral’s model has 4 billion parameters. It deviates from the pattern: the 37 tokens per frame are generated at once using a diffusion model (flow matching). The approach reduces computation while carrying rich acoustic information.

Conditioning: Text and Voice in One Shot

▶ Watch (0:15)

Mistral’s model takes the full text and a few seconds of reference speech (the voice to clone) as context. It then generates the entire audio. That puts it in the “all context first” category. Latency is 17 milliseconds from text input to first playable audio on a single GPU. The next step is real-time text input streaming, where the LLM outputs tokens incrementally and TTS starts voicing them immediately. Several architectures are being explored: interleaved or dual-stream.

Q&A

Did the demo generate text and audio at the same time? No. A small, fast LLM produced the text instantly, then the TTS model generated the audio from that complete text. ▶ Watch (18:10)

Is the voice cloning part open source? The text-to-speech model weights are open, but the encoder that extracts the voice embedding is proprietary. Users can clone voices only through Mistral’s served API. ▶ Watch (18:36)

What is your take on using speech as an interface? Speech as an interface goes far because central LLMs are extremely capable. Streaming text tokens from the LLM into TTS gives a smooth UX, especially for longer outputs where waiting for full text generation would degrade the experience. ▶ Watch (19:45)

Do you know which architecture for real-time streaming is best? Not yet. Options include interleaved, delayed sequence modeling, or other patterns. The choice determines how soon the first audio packet can be emitted after the first LLM token. ▶ Watch (21:00)

Notable Quotes

So this way in my time of delusion and at the peak of my ego I can discuss with myself on on a complicated problem which is nice. Sarah Chieng · ▶ Watch (7:09)

I hope you learned a few things today. Sarah Chieng · ▶ Watch (17:22)

So the question was like hey, on the demo it looks like we are generating the text and the audio at the same time. Sarah Chieng · ▶ Watch (17:48)

Key Takeaways

  • Mistral’s TTS model generates 500 audio tokens per second from 80 ms frames.
  • First audio packet arrives 17 ms after text input on a single GPU.
  • Voice cloning requires only a few seconds of reference speech.
  • The model uses a 4-billion-parameter backbone with a diffusion decoder per frame.
  • Real-time streaming of LLM text into TTS is the next frontier for latency.