AI
Why Build an LLM From Scratch
Angelos Perivolaropoulos leads the speech-to-text team at ElevenLabs, where his team trained Scribe v2, currently the top model on public transcription benchmarks. This workshop builds a GPT-2-style LLM using pure PyTorch:
“no pre-trained weights, no uh nothing that you can just grab online from like a transformers library” — Angelos Perivolaropoulos
Any laptop with 16 GB of RAM qualifies. Google Colab on a free T4 GPU also works. The four components to implement are a tokenizer, a model architecture, a training loop, and an inference function. Everything beyond that, Angelos said, is optimization and scale.
Tokenization: Character-Level vs BPE
The workshop uses character-level tokenization, producing 65 unique tokens from the Shakespeare corpus. That gives 4,225 possible bigrams, all of which a 1M-character dataset covers many times. GPT-2’s 50,000-token vocabulary would require 50,000 squared data points and a 19-million-parameter embedding table, three times the model itself.
“if we did try to train using a full tokenizer, this will never converge” — Angelos Perivolaropoulos
BPE is the right choice for production. It learns common patterns from training data — keywords like for and enumerate in code-heavy corpora become tokens, while rare strings fall back to character level.
Transformer Architecture in Code
The model config uses vocab size 65, block size 256, 6 layers, 6 attention heads, and embedding dimension 384. Those choices produce 1.8M trainable parameters: 25k from token embeddings, 98k from positional embeddings, 1.2M from the transformer blocks. Each block runs its own layer norm, multi-head attention, and MLP, connected with residual additions so activations shift incrementally rather than resetting per layer.
Scaling means raising layers, heads, and block size together. Jumping block size from 256 to 2 million context without redesigning attention causes an immediate out-of-memory failure. Researchers after GPT-3.5 had to modify the architecture specifically to make longer sequences trainable.
The Training Loop
Cross-entropy loss starts at 4.17, the natural log of 65, because an untrained model distributes probability equally over all tokens. By step 800 it generates word fragments; by step 1,000 recognizable words appear. The loss milestone walkthrough (49:54) shows the model producing coherent Shakespeare-style text once loss falls to 1.0-1.2.
“at around two 2,400 steps is where the that was the optimal performance of this model” — Angelos Perivolaropoulos
The training loop uses AdamW with a 100-step warmup, then cosine decay over 5,000 steps. Batch size is 64 sequences of 256 tokens each. Val loss on held-out data is the stop signal: once it rises while train loss keeps falling, the model has overfit.
Inference and Extending Beyond Text
Greedy decoding always picks the highest-probability token. For transcription that is correct. For text generation it produces repetitive output. Temperature at 0.7 is the practical default, occasionally picking a lower-ranked token to keep generation varied. Top-k sampling then masks tail candidates so an unlikely token can’t sneak through on a bad random draw.
Reasoning models share the same base architecture as this one; the difference is post-training on high-quality chain-of-thought data from specialists. Audio models follow the same transformer fundamentals, but tokenization converts raw audio to mel spectrograms first, and the loss function shifts from cross-entropy to L2 on those spectrograms.
Q&A
Are reasoning models fundamentally different from what you showed, or mostly the same architecture with smarter tricks? Same base architecture: reasoning capability comes from post-training on high-quality chain-of-thought data, not from changes to the transformer itself. ▶ 1:03:43
How different is training audio models compared to text at ElevenLabs? Fundamentals are the same, but audio requires converting to mel spectrograms before tokenization and using losses like L2 instead of cross-entropy. ▶ 1:11:32
Notable Quotes
no pre-trained weights, no uh nothing that you can just grab online from like a transformers library Angelos Perivolaropoulos · ▶ 1:23
if we did try to train using a full tokenizer, this will never converge Angelos Perivolaropoulos · ▶ 12:11
at around two 2,400 steps is where the that was the optimal performance of this model Angelos Perivolaropoulos · ▶ 51:38
Key Takeaways
- A 1.8M-parameter GPT-2-style model trains in 15 minutes on a free Google Colab T4 GPU.
- Character-level tokenization with 65 tokens keeps bigram pairs to 4,225 — small enough for a 1M-character dataset to cover fully.
- Val loss rising while train loss falls is the exact signal that your small model has overfit and should stop training.