AI
Two Tiers of On-Device AI
Two deployment patterns are dividing the market. The first is system-level GenAI: a 2-5B parameter model baked into the OS and shared across apps. Android calls this AI Core; Apple has Apple Intelligence. Apps call a prompt API instead of bundling their own weights. The second is in-app GenAI, where a custom model ships with the app itself. These models run below 1B parameters and target every device, not just premium flagships. Google calls models below that threshold TLMs (tiny language models). Below 500M parameters, fine-tuning for a specific task is not optional.
Gemma 4 on Device: Thinking, Function Calling, and Real Throughput
Gemma 4 shipped two mobile-first sizes: E2B and E4B. The “E” designates effective RAM footprint: 2 GB and 4 GB respectively. Both are multimodal (audio, image, text) and include built-in thinking plus function calling, the combination that makes agent skills possible on device.
“our ability to now do skills on device.” — Cormac Brick
On a high-end Android phone, the E2B model reaches thousands of tokens per second on GPU. A Raspberry Pi delivers 133 tokens per second on CPU, enough for image analysis. The E2B and E4B are on the Android AI Core roadmap, so a future OS update will make them available system-wide without per-app downloads.
Agent Skills: Lazy Loading and Constrained Decoding
Loading every skill’s full instructions at startup would overwhelm a 2-4B model. Instead, the skill registry shows only a one-line description per skill. The model calls a load_skill tool when it decides a skill is relevant, fetching the full skill.md on demand. Short context, higher reliability. The skills demo: mood tracker, Wikipedia, and mood music (20:41) runs on the 4B model with no custom fine-tuning.
Constrained decoding adds a second reliability layer. When generating a tool call, the runtime constrains output to the specific JSON schema for that tool, not generic JSON. For the 2B model this matters significantly; larger models need it less.
LiteRT-LM: The Deployment Pipeline from Transformers to Device
LiteRT-LM is the open-source runtime powering AI Gallery. It takes a single .llm file (tokenizer and weights bundled) and runs on mobile, desktop, and embedded. The build path: Hugging Face model, through LiteRT-Torch for quantization, then the .llm artifact. CPU and GPU use a JIT workflow, one file for many devices. NPU requires ahead-of-time compilation via a vendor compiler plug-in, producing a chip-specific artifact. The app-level API is the same either way. Third-party models including Qwen 6B and Apple’s FastVLM (500M parameters, vision-language) already run in Gallery through this pipeline.
AI Edge Eloquent: Modular Tiny Models in Production
AI Edge Eloquent (iOS) runs two Gemma 3 270M derivatives in sequence: one for speech recognition, one for text polishing. The polishing model strips filler words, corrects homophones, and applies a user-defined bias list for technical terms. Both were fine-tuned on synthetic data generated by a larger cloud LLM, then quantized. The AI Edge Eloquent live demo (55:44) shows the full pipeline running offline. On the Mobile Actions eval, fine-tuning lifted accuracy from ~40% to 86%, a 20-40 point gain Brick described as typical for sub-500M models.
Q&A
What was the function-calling reliability before fine-tuning, and after? Baseline was around 40%; after fine-tuning the 270M Mobile Actions model on 10 functions it reached 86%, with eight of the ten functions individually scoring above 93%. ▶ 1:04:30
How do you handle safety for edge and tiny models? Brick deferred model-level safety to the Gemma team, noted that OS vendors layer input/output safety checkers on top for system GenAI, and said tiny models’ narrow task scope limits the attack surface by design. ▶ 1:09:51
Notable Quotes
our ability to now do skills on device. Cormac Brick · ▶ 15:09
that were over 90 like 93% type thing so Cormac Brick · ▶ 1:05:14
there’s a really really low barrier to Cormac Brick · ▶ 31:11
Key Takeaways
- Gemma 4 E2B runs on 2 GB RAM and achieves thousands of tokens per second on high-end Android GPU.
- Skill lazy-loading keeps context short enough for 2-4B models to stay reliable on device.
- Fine-tuning adds 20-40 eval points for sub-500M models, making it essential for production.