AI
MLX Brings LLM Inference to Apple Silicon
MLX is Apple’s framework for machine learning on Apple Silicon. It runs on the same unified memory chip used by iPhones, iPads, and Macs, which means the CPU and GPU share memory without copying data between them. Locally AI, Grondin’s app, sits on top of MLX and gives users a native chatbot that runs models entirely on device, with no server call required.
The MLX package ecosystem now covers text, audio, image generation, and video. Omni models and text-to-speech are both supported. The iOS entry point is the MLX Swift LM GitHub repo, and Python developers can use MLX VLM instead.
Sourcing Quantized Models from the MLX Community
Hugging Face’s MLX Community organization is where quantized weights live. Nearly 4,000 to 5,000 models are uploaded there, and when a lab releases a new model, a quantized version typically appears within 30 minutes. Variants cover BF16 through MXFP4, plus 4-bit, 5-bit, 6-bit, and 8-bit options.
To use one, copy the model ID from Hugging Face and pass it to MLX Swift LM. The framework handles the download directly. No manual weight conversion is required.
Choosing the Right Quantization Level
Full-precision weights are too large for iPhone memory, so a quantized version is mandatory. Grondin recommends staying between 4-bit and 8-bit. Below 4-bit, output quality starts to degrade noticeably and the models are generally not useful. At the small end, a 350-parameter liquid model runs fast enough to work inside iOS Shortcuts for text-processing automation. Larger models like Gemma 4 fit best at 4-bit or 8-bit depending on available RAM.
The practical ceiling is 8-bit on small models. On older iPhones, expect around 20 tok/s at 4-bit, still fast enough for most apps.
Performance on Device and the LM Studio Acquisition
Grondin ran the Gemma 4 live demo at 40 tok/s (06:01) on stage, offline, generating a long streamed response in real time. Gemma 4 8-bit quantized to 4-bit hit 40 tokens per second on the latest iPhone.
“40 token per second is more than acceptable for a lot of use cases.” — Adrien Grondin
He also announced that Locally AI had just been acquired by LM Studio, a desktop app that lets users download models from Hugging Face, run them with Llama.cpp or MLX, and expose a local server with OpenAI or Anthropic response formats. The combined toolchain covers both mobile and desktop on-device inference.
Q&A
Does this support tool calling? MLX Swift LM supports tool calling; structured generation is not yet built in but third-party packages are working on it, including one from Hugging Face. ▶ 9:02
Notable Quotes
40 token per second is more than acceptable for a lot of use cases. Adrien Grondin · ▶ 6:15
Locally have been acquired by LM Studio. Adrien Grondin · ▶ 7:51
Key Takeaways
- MLX Swift LM lets any iOS developer embed a quantized LLM in under 10 minutes of setup.
- 4-bit quantization is the practical floor for iPhone; below that, output quality degrades noticeably.
- Gemma 4 at 4-bit quantization hits 40 tok/s on the latest iPhone, running fully offline.