Vision-Language-Action Models Generalize Beyond Training Data

▶ Watch (1:13)

Kanishka Rao’s team added action tokens to a vision-language model. They placed a pile of toys in front of the robot and asked it to pick up the extinct animal. The robot had never seen a dinosaur toy in its training data. It used its understanding of “extinct” from internet text and images to select the dinosaur. That moment proved that digital AI knowledge transfers to physical actions. The same model also understood “pick up the soda” versus “pick up the fridge” in a live demo.

Simulation Handles Locomotion, Teleoperation Tackles Dexterity

▶ Watch (9:33)

Simulation handles the robot body. Boston Dynamics builds accurate simulators for their hardware. Training with reinforcement learning in simulation produces policies for walking, running, and dancing that transfer to real robots. For manipulation, simulation fails. Objects are too varied. Teleoperation collects real-world data. Pilots wear VR headsets to see through the robot’s eyes. That data trains vision-language-action models. The two buckets remain separate for now.

Dexterity Is the Hardest Unsolved Problem

▶ Watch (15:55)

Kanishka Rao stated that robots can code operating systems in 24 hours but cannot scramble eggs. Dexterity is the hardest unsolved problem. All state-of-the-art manipulation models are vision-based. They use wrist cameras to infer forces from pixel changes. The robot folded origami with no touch sensor. But reliable dexterity for tasks like picking keys from a pocket or using a screwdriver is years away. Hardware for tactile sensing is not ready.

Industrial Deployments First, Homes in a Decade

▶ Watch (22:38)

The immediate entry point is manufacturing. Boston Dynamics’ Stretch unloads 50–60 lb boxes from trucks in unregulated temperatures. Spot performs 500 inspection passes daily without losing focus. Atlas is designed for backbreaking physical labor. Safety and cost limit humanoid robots to controlled industrial environments now. Kanishka Rao expects 5–10 years before robots appear in homes. Dexterity and safety puzzles must be solved first.

Notable Quotes

we can, you know, code up operating systems in like whatever 24 hours and we can solve complicated math but we can’t scramble eggs. Kanishka Rao · ▶ Watch (16:05)

balancing is solved, I would say. I don’t know if that’s a hot take anymore, but yeah, balancing is solved. Kanishka Rao · ▶ Watch (24:34)

they’re very good at whole body control. Kanishka Rao · ▶ Watch (24:18)

Key Takeaways

  • Vision-language-action models transfer digital intelligence to physical robots.
  • Simulation trains locomotion; teleoperation collects data for manipulation.
  • Dexterity (fine manipulation) is the unsolved core problem for general-purpose robots.