For two years, running LLMs on a Mac felt like a party trick that almost worked. Apple Silicon had the raw compute — the unified memory architecture was clearly built for memory-bound workloads like transformer inference. But the software stack treated every Mac like a generic GPU. It worked. It just never felt like it was running on hardware designed for it.
Ollama 0.19, shipped March 31, changed that. The MLX backend is here, and the numbers aren't incremental — they're the kind that make you re-read the release notes to make sure you didn't misplace a decimal point.
The Numbers
Measured against the previous llama.cpp Metal backend, on the same hardware, running Qwen3.5-35B-A3B with int4 quantization:
- ~1.6x faster prefill (time to first token)
- ~2x faster decode (tokens per second during generation)
On an M5 Max, Ollama is reporting up to 1,851 tokens/s prefill and 134 tokens/s decode. Community benchmarks back this up — one developer measured a 93% improvement in decode speed on their machine, which lines up almost exactly with Ollama's "nearly doubles" claim. When marketing numbers match independent measurements, something real happened.
Why MLX Is Actually Faster
"We switched backends and got 2x" is usually a boring story about compiler flags. This isn't that. MLX is faster because it's the first mainstream inference framework that treats Apple Silicon as what it actually is, instead of what it approximates.
No More Phantom Copies
Every other architecture has separate CPU and GPU memory. When the GPU wants to process a tensor, you copy it from system RAM into VRAM. That's how GPU programming has worked forever. llama.cpp's Metal backend — being portable across wildly different hardware — treats the Mac's GPU the same way. It copies data between address spaces that are, physically, the same pool of RAM.
MLX was built for Apple Silicon from day one. Tensors live in a single address space. The CPU and GPU operate on the same memory without copying. For LLM inference — which is almost entirely memory-bandwidth-bound at low batch sizes — eliminating those phantom copies translates directly into tokens per second. The M-series chips have always had the bandwidth. Now the software actually uses it.
M5 Neural Accelerators
Starting with the M5 chips, Apple added GPU Neural Accelerators — dedicated matrix-multiplication units inside the GPU, exposed through Metal 4's TensorOps. These are Apple's answer to Nvidia's tensor cores, and they matter enormously for LLM prefill, which is essentially one giant matrix multiplication.
Apple's own research paper on M5 and MLX shows up to 4x speedup on time-to-first-token versus an M4 baseline for workloads with large matrix multiplications. Ollama 0.20.4 added specific optimizations for these accelerators, and the improvement is visible in real-world use.
The nuance: older Apple Silicon (M1 through M4) still benefits from the unified memory improvements. You just don't get the Neural Accelerator boost. M4 Max owners are seeing the 1.6-2x gains purely from the memory architecture fix. M5 owners get another multiplier on top.
The Catch (Because There's Always a Catch)
The initial 0.19 release came with real limitations:
- Only Qwen3.5-35B-A3B was accelerated. Other model architectures still fell back to the old
llama.cppMetal backend. - Requires >32GB unified memory. This rules out every base-spec MacBook Air and most base-spec MacBook Pros. You need a Pro or Max configuration.
- Preview quality. Rough edges around caching and long-context sessions.
Versions 0.20 and 0.21 have already broadened support — Gemma 4 is in, mixed-precision quantization landed, and the MLX backend picked up new operator wrappers for Conv2d, masked scaled dot-product attention, and RoPE-with-frequencies. The trajectory is clear even if the current release is still narrow.
What This Means for Local Agent Workflows
Here's why this matters beyond benchmark bragging rights: agent loops are latency-sensitive in ways that simple chat is not.
A single agent task might involve 5-15 model calls — planning, tool selection, execution, verification, reflection. Each call needs to stream back fast enough that the agent doesn't feel sluggish. At old decode speeds, a complex agent task on local hardware took 2-3 minutes. At 2x decode, it takes 60-90 seconds. That's the difference between "I'll wait" and "I'll use something else."
The prefill improvement matters even more for agents. Agents tend to stuff enormous context windows — conversation history, tool outputs, retrieved documents, system prompts. The time to first token is dominated by how fast the model can ingest all that context. A 1.6x improvement on prefill means your agent starts thinking faster, which compounds across every call in the loop.
Combine this with the privacy story — no data leaving your machine, no API costs, no rate limits — and you have a genuine shift. Running a coding agent locally on a MacBook Pro is no longer a curiosity. It's a practical choice for developers who work with proprietary codebases, sensitive data, or just don't want to fund another AI lab's training budget.
Apple Silicon as an Inference Platform
Let's be honest about where things stand. Apple Silicon is not competing with an H100 on raw FLOPS. A single H100 delivers ~2,000 TFLOPS of FP16 compute. An M5 Max delivers roughly 180 TFLOPS of GPU compute with the Neural Accelerators. That's a 10x gap.
But LLM inference at small batch sizes — which is what you're doing on a laptop — is not FLOPS-bound. It's bandwidth-bound. And here Apple Silicon competes. The M5 Max delivers around 546 GB/s of memory bandwidth. An H100's HBM3 delivers 3,350 GB/s. A 6x gap, not 10x, and you're running at batch size 1 while the H100 is designed for batch sizes in the hundreds.
The economics are also tilting. A MacBook Pro with an M5 Pro and 48GB unified memory costs roughly $3,000. You get a machine that runs a 32B parameter model at interactive speeds, runs your IDE, compiles your code, and works on battery. A comparable cloud inference setup — reserving a GPU instance 24/7 — costs $1-3/hour. The MacBook pays for itself in 3-6 months if you're running agents continuously.
What About Foundry Local?
Microsoft shipped Foundry Local around the same time, their answer to on-device inference on Windows machines. It's competent — it uses ONNX Runtime with DirectML acceleration and supports a similar range of quantized models.
But Foundry Local is fighting a hardware problem, not a software one. Most Windows laptops still have discrete Nvidia GPUs with 6-8GB of VRAM, or integrated GPUs sharing system memory through a much less elegant architecture than Apple's unified memory. You can run a 7B model comfortably. A 32B model? Not on most hardware people actually own.
The Mac's advantage isn't the chip — it's the memory architecture. 48GB or 64GB or 128GB of unified memory means you can load model weights that simply don't fit in a typical laptop GPU. The MLX backend finally makes use of that advantage properly.
The Real Takeaway
What's happening here is bigger than Ollama or MLX. The local inference stack on Apple Silicon has gone through three phases:
- The "it works" phase (2023-2024).
llama.cppwith Metal. Slow but functional. Good enough to experiment, not good enough to rely on. - The "it's usable" phase (2025). Better quantization, Ollama's UX, LM Studio. Local inference became a reasonable choice for simple tasks. Still noticeably slower than cloud APIs.
- The "it's competitive" phase (2026). MLX backend, M5 Neural Accelerators, 2x performance jumps. Local inference is now fast enough for agent loops, code completion, and real-time assistance.
Phase 3 changes what you build. When inference is fast enough that agents don't feel sluggish, you stop reaching for cloud APIs as the default. You start designing workflows that assume local inference is available — offline-first, privacy-first, zero-cost-per-query. The architecture of your tools changes when the constraint loosens.
If you're on an M4 Pro or better, update Ollama and try the MLX backend. Run a 14B or 32B model through an agent loop — a coding task, a research task, anything that needs multiple model calls. Feel the difference. Then think about what you'd build if every developer's laptop could run a competent model at interactive speeds, without touching the internet.
That future is already here. Most people just haven't updated yet.