Perception Beyond Fixed Labels

▶ Watch (3:24)

With Gemini Robotics ER, developers no longer train custom vision models for each environment. The VLM architecture uses semantic grounding to locate objects described in natural language. The model identifies the most-used tool or an overheating component, not just predefined classes. This open vocabulary detection works through the Python SDK. The model also reasons about physics. It understands that a glass bottle is fragile and a heavy dish exceeds the robot’s capacity.

Temporal Reasoning and Code Execution

▶ Watch (6:37)

Built on the Gemini 3 Flash backbone, ER 1.6 processes video and multi-image inputs. A robot can pass 30 seconds of frames and ask if the gripper secured the object. The model analyzes motion between frames, not just the final state. This replaces complex heuristic code for success detection. For difficult images, the code execution tool lets the model generate Python to crop and rotate images. An example reads an upside-down chip ID on a circuit board. The model crops the chip area and rotates the image for readable text.

From High-Level Tasks to Motor Values

▶ Watch (9:19)

The ER model handles task orchestration by breaking “put the blue block in the orange bowl” into function calls: open gripper, move to coordinate, close gripper, move to drop-off zone. It also generates waypoint trajectories around obstacles. For execution, VLA models map camera pixels and natural language directly to motor values. A robot performed a slam dunk with a basketball net despite no specific training. The VLA uses the Gemini backbone for general world understanding. ER and VLA together close the perceive-plan-actuate loop.

Natural Interaction and Safety

▶ Watch (13:39)

The Gemini Live API enables low-latency bidirectional audio and video streaming. A robot can hear “That screw looks a little loose. Can you tighten it?” and trigger a function call to act. Camera frames stream for constant visual context, allowing plan changes at runtime. Safety uses the Swiss cheese model with multiple layers: semantic, physical, operational. The Asimov safety benchmarks are grounded in real injury data from the NEISS database, covering 10,000+ reports. Industrial ISO standards validate the benchmarks.

Notable Quotes

We aren’t just teaching machines to see the world, but we’re enabling them to perceive it with advanced spatial awareness and understanding. Paul Ruiz · ▶ Watch (1:07)

The model uses semantic grounding to locate whatever you describe in natural language. Paul Ruiz · ▶ Watch (3:37)

Gemini Robotics ER has what we call physical common sense. Paul Ruiz · ▶ Watch (4:37)

We like to think of it as the Swiss cheese model of defense. Paul Ruiz · ▶ Watch (16:43)

Key Takeaways

  • Gemini Robotics ER 1.6 provides open vocabulary detection and physical common sense.
  • VLA models map camera pixels directly to motor values for real-time control.
  • Safety benchmarks use real injury data from NEISS and industrial ISO standards.