AI
Data Curation as a Training Investment
Data curation is underrated even today, and Dieleman is direct about why: academic training rewards comparing results on fixed public benchmarks, not looking at your own data. At production scale that incentive inverts. Time spent cleaning and filtering training data is often a better return than tuning the optimizer or adding model capacity. He declined to share specifics, calling it part of the secret sauce, but the point stands independent of any particular recipe.
Latent Representations Instead of Raw Pixels
Feeding raw pixels into a diffusion model worked early on, but 30 seconds of 1080p video at 30 fps occupies several gigabytes per training example. A trained autoencoder compresses the input through a bottleneck: a 256x256 RGB image becomes a 32x32 latent grid, with extra channels to hold fine-grain detail that would vanish at that resolution.
“that’s the key idea. That’s the point.” — Sander Dieleman
Video gains even more. The extra time dimension carries heavy redundancy, so latent compression can shrink tensor size by two orders of magnitude. The difference between fitting data in memory and not fitting at all.
How Diffusion Models Actually Generate Images
Diffusion trains a denoiser: feed in a noisy image, predict the clean original. Because many images could match the same noisy observation, a single-step prediction averages them and comes out blurry. The model takes a small step toward that blurry prediction, adds fresh noise to suppress error accumulation, and repeats until one point remains.
“So what does that mean for diffusion?” — Sander Dieleman
Gaussian noise has a flat frequency spectrum; natural images follow a power law with high frequencies lowest in energy. Adding noise drowns high frequencies first. The denoiser rebuilds structure coarse-to-fine, which Dieleman calls “spectral autoregression.” That ordering lets training weight perceptually important frequencies more heavily.
Classifier-Free Guidance and the Quality-Diversity Trade-off
Run the denoiser twice per step: once with the text prompt, once without. The gap between those two predictions points toward the region of image space the prompt selects. Scale that gap up and step in that direction. Diversity drops, quality rises. At high guidance, images are visibly better but noticeably similar.
A 2021 OpenAI paper (GLIDE) was the last to compare guidance-off versus guidance-on. After that, everyone left guidance on permanently.
“uh guidance is sort of a no-brainer” — Sander Dieleman
Diffusion models run smaller than LLMs partly because this trick lets them punch well above their parameter count.
Distillation and Fine-Grained Control Signals
Sampling traces a nonlinear path through input space. Consistency model distillation replaces tangent-prediction with endpoint-prediction: given your position on that path, predict where it ends. One-step sampling usually degrades quality because the network must compress 50 passes into one. Three-step consistency with interval-based training works better in practice.
Control signals extend beyond text prompts. Reference images let users condition on appearance without describing it. Camera motion and event timing require structured signals text cannot carry, and this data rarely exists at pre-training scale. These are added in post-training, alongside preference tuning via reinforcement learning or direct preference optimization.
Q&A
Does high guidance scale cause the distinctive visual style of models like Midjourney? Partly, but Dieleman said post-training (which selects a narrow slice of the pre-training distribution) plays the larger role; guidance at the wrong noise level does produce over-saturation, and varying the guidance scale across the sampling trajectory helps. ▶ 32:15
Can classifier-free guidance be applied to text diffusion models? Yes in principle, but Dieleman said guidance works best when the conditioning signal has a semantic gap from what is being generated, which is why text-to-image guidance is so effective and why it sees less use in autoregressive language models. ▶ 33:45
Does taking more denoising steps always improve sample quality? Up to a point: more steps reduce approximation error on the nonlinear sampling path, but there is a limit, and rectified flow research focuses on making that path straighter so fewer steps suffice. ▶ 37:29
Notable Quotes
that’s the key idea. That’s the point. Sander Dieleman · ▶ 7:58
So what does that mean for diffusion? Sander Dieleman · ▶ 17:52
uh guidance is sort of a no-brainer Sander Dieleman · ▶ 27:40
Key Takeaways
- Autoencoders compress video tensors by up to 100x, making diffusion training on high-resolution video feasible.
- Diffusion generates images coarse-to-fine because Gaussian noise masks high frequencies first, then low frequencies.
- Classifier-free guidance amplifies conditional vs. unconditional prediction gaps, dramatically improving output quality at the cost of diversity.