Why On-Device AI on Android

▶ Watch (0:49)

On-device AI processes prompts and data directly on the device. Sensitive data like banking or medical records never leaves the phone. No internet required. No per-inference cost. Google supports this with Gemma and Gemini Nano models. Gemma is an open model collection. Gemini Nano is optimized for battery and performance on supported devices. Prompts written for Gemma 4 will work on next-generation Gemini Nano automatically.

ML Kit GenAI: AI Core and Developer Preview

▶ Watch (3:11)

AI Core is an Android system service that manages Gemini Nano on supported devices. It optimizes inference latency and power consumption. Apps do not handle model updates. AI Core isolates each inference request; input data and outputs are never stored. The AI Core Developer Preview gives access to preview models like Gemini Nano 4. The model selection API targets those previews in code. The speech recognition API transcribes from microphone or audio files. On Gemini-powered devices, it uses one small adapter for all languages, replacing 100-200 MB per-language models. Compatible with Android 31+.

Prompt API: Prefix Caching and Structured Output

▶ Watch (8:19)

Prefix caching stores intermediate LLM state for shared prompt prefixes. For a 1,000-token fixed prefix and 100-token dynamic suffix, inferences were 2x faster on most devices, 4x on Pixel 9. Two management options: automatic and manual. Structured output API lets apps define Kotlin data classes as output format. Generate type content returns consistent results instead of error-prone JSON prompts. The automated prompt optimizer (APO) uses server-side models like Gemini Pro to find the optimal prompt given an evaluation set.

LiteRT-LM for Custom Models

▶ Watch (12:34)

For custom solutions, LiteRT-LM manages small models on device. Example: an agritech app translating English to Marathi (Varhadi dialect) offline. Fine-tune Gemma 270M using Hugging Face TRL library and a public English-Marathi dataset in Colab. Convert the model with the LiteRT-LM CLI. Integrate via LiteRT-LM API: initialize engine with hardware backend (CPU, GPU, NPU), load model, instantiate conversation object, use sendMessageAsync for streaming. Google AI Edge portal benchmarks the LiteRT-LM file across 100+ real Android devices in Google’s labs before production.

Notable Quotes

Input data and the resulting outputs are never stored anywhere on the device. Caren Chang · ▶ Watch (3:52)

for most devices, inferences were on average two times faster for prompts with a 1,000 token fixed prefix and a 100 token dynamic suffix with prefix caching enabled. Caren Chang · ▶ Watch (8:46)

We hope you get a head start on refining prompt accuracy and exploring use cases before the models launch on consumer devices. Caren Chang · ▶ Watch (5:02)

Key Takeaways

  • Use ML Kit GenAI APIs for turnkey on-device AI: speech recognition, prefix caching, structured output.
  • Fine-tune Gemma 270M and convert to LiteRT-LM for custom offline models.
  • Benchmark custom models across 100+ Android devices with Google AI Edge portal.