On-Device Agent Capabilities with Gemma 4

▶ Watch (2:07)

The Gemma 4 edge models (2B and 4B parameters) consume 1 to 2 GB of RAM after quantization. New agent features include built-in function calling for tool use, native structured JSON output baked into the model architecture rather than prompt engineering, and a chain-of-thought thinking mode. The models interact with local APIs while keeping inference on device. Google released them under Apache 2.0 on Hugging Face.

▶ Watch (7:10)

The open-source gallery app showcases on-device agent skills. Demos included a Wikipedia query skill, a mood tracker that analyzes sleep trends over seven days, an image-to-music generator that pairs a photo with a song, and an animal sound creator driven by text prompts. All processing stays on the phone or laptop. Users can write and share their own skills through the app without leaving it.

Lite RT Enables Cross-Platform Deployment

▶ Watch (11:11)

Lite RT is Google’s on-device framework built on TensorFlow Lite. It accepts models from TensorFlow, PyTorch, and JAX after conversion to the TFLite format. The same model file runs on Android, iOS, macOS, Linux, Windows, web, and IoT devices. Lite RT already powers over 100,000 apps with billions of active users. A new CLI tool with Python bindings simplifies deployment.

Hardware Acceleration Delivers 13x Performance Boost

▶ Watch (15:40)

CPU and GPU acceleration is universal. NPU integration with Qualcomm and MediaTek provides 3 to 10x performance and energy improvements. On iOS the Gemma 4B model hits 56 tokens per second. Lite RT shows up to 35x faster inference than Llama on mobile. A live demo on a Raspberry Pi robot ran the model on CPU through Lite RT, responding to speech commands by wiggling its antenna sharpies.

Q&A

Can face unlock run locally on phones? Yes, Google’s face unlock uses the same Lite RT framework on Pixel and other devices already. ▶ Watch (19:53)

Can a home security camera do on-device face recognition? Streaming from a home camera is possible, but authenticity checks may require additional algorithms compared to phone deployment. ▶ Watch (20:20)

Are there examples of distributing workloads between edge and cloud? The speaker mentioned health agent or coaching agent architectures being explored, but did not share specific examples. ▶ Watch (21:53)

Does Lite RT support open-weight models other than Gemma? Yes, the Hugging Face page hosts other open-weight models. Developers can convert any model to TFLite format and run it through Lite RT. ▶ Watch (22:58)

Notable Quotes

we’ve seen up to like 35 times faster performance. Chintan Parikh · ▶ Watch (19:00)

the NPU is going to give you at least like 3 to 10x improvement in performance Chintan Parikh · ▶ Watch (16:07)

like roughly 56 tokens per second Chintan Parikh · ▶ Watch (18:35)

It has uh 100,000 plus apps, billions of active users Chintan Parikh · ▶ Watch (11:55)

Key Takeaways

  • Gemma 4 edge models (2B, 4B) add on-device function calling, chain-of-thought reasoning, and structured JSON output.
  • Lite RT runs models from TensorFlow, PyTorch, and JAX on Android, iOS, Linux, Windows, and IoT devices.
  • NPU acceleration delivers 3–13x speedups; Lite RT is up to 35x faster than Llama on mobile.