AI
From Classic RL to Language Model Training
Language models act as agents. Environments include data, harnesses, and scoring rules. The shift from supervised fine-tuning to reinforcement learning with verifiable rewards means environments become dynamic systems the model interacts with. In Tic-Tac-Toe, the agent generates moves, the environment tracks the board and opponent, and rewards (+1 for win, 0 for loss) guide learning through trial and error without human examples.
Building Environments with Verifiers
Verifiers, an open-source library by Prime Intellect, provides base classes for single-turn, multi-turn, and tool environments. Environments are Python packages. The library abstracts model serving via an OpenAI-compatible API endpoint. It handles parallel trajectories and integrates with training frameworks like Prime RL and Sky RL. The Reverse Text environment demonstrates a single-turn setup with 1,000 text paragraphs, a parser, and a longest common subsequence reward.
Designing a Tic-Tac-Toe Environment
The environment assumes the model plays as X first, outputs a move between 0 and 8 inside move tags, and the opponent plays randomly. Reward functions include winner reward (weight 1) and format reward (weight 0.2). Improvements added variable opponent skill via minimax with configurable random move probability, enforced reasoning with think tags, penalised invalid moves with -0.1, and used stratified sampling to stabilise training.
Training Results and Insights
The Liquid AI model LFM-2 started as a weak player. A supervised fine-tuning warm-up with 200 synthetic winning games from GPT-5 mini taught format and valid moves. Then GRPO-based RL training on a 96 GB GPU raised performance: it dominates random players and draws 85% of games against an optimal opponent. Invalid moves dropped to near zero. A second training run with higher opponent skill and higher temperature produced a perfect Tic-Tac-Toe master that beat GPT-5 mini.
Lessons Learned
Batch size of at least 256 prevents model collapse. Hidden biases in minimax (always picking the first optimal move) caused the model to memorise a single opponent pattern. Starting from a reasoning model requires long context budgets; it is safer to start from an instruct model and transform it. Inspect rollouts and test the model interactively after training. Wait for slow RL runs to mature before tweaking.
Notable Quotes
“The model is no longer limited by the quality of human examples. Through trial and error, it can discover more efficient reasoning strategies.” Samuel Colvin · ▶ Watch (7:47)
“We gave it a space to play and guided it through rewards.” Samuel Colvin · ▶ Watch (39:10)
“If you can define a clear reward signal, you can build an environment and train a small, specialized model to beat a large closed model on a specific task at a fraction of the cost.” Samuel Colvin · ▶ Watch (39:26)
Key Takeaways
- Verifiers provides reusable, modular RL environments for LLMs.
- A small model with SFT warm-up and RL can surpass a large closed model on a narrow task.
- Batch size and opponent skill variability critically affect training stability.