Ollama vs MLX on Apple Silicon: Which Local LLM Stack Hits the Unified Memory Wall First

Unified memory is the reason a Mac is a good local-LLM box and the reason it is a bad one. There is no separate GPU pond to drain. The model, the KV cache, Chrome, Xcode, and the video call all drink from the same lake. When the lake is gone, you do not get a polite CUDA OOM. You get swap, fans, and a UI that feels like molasses poured on a trackpad.

Ollama and MLX both run on that lake. They do not drink it the same way. Ollama is the friendly daemon people install first: a model library, an API, a llama.cpp-shaped engine that also runs on a Linux box. MLX is Apple’s array framework plus a growing model zoo that wants to be native Metal, not a port. I use both. I hit the wall with both. The wall arrived at different times, for different reasons.

This is not a tokens-per-second leaderboard. Those rot in a month. This is about which stack makes the memory wall visible, and which one hides it until your Mac is already drowning.

What you are actually installing

Ollama is an operator product. Pull a model, ollama run, point a client at localhost. The GGUF catalog is the social layer. Everyone’s blog uses the same names. You can move the habit to a PC later. That portability is the feature. The cost is an engine that is excellent and still not “written for this SoC first.”

MLX is a developer product that grew a CLI culture. You pick a converted model, you think about dtype, you sometimes write a few lines. The Hugging Face MLX community packages are good now. The experience is still closer to “I am running a framework” than “I have a pet server.” If you want an OpenAI-shaped API for a dozen tools, Ollama wins the afternoon. If you want the Mac to do array math the way Apple intended, MLX wins the evening you are willing to spend.

I keep Ollama for anything another app must talk to. I keep MLX for experiments and for the models I want to squeeze harder on the same RAM budget.

The wall is RAM plus cache plus the rest of your life

People quote parameter counts like apartment square footage. The number that wrecked my week was context. A 7B model with a polite 4k window is a houseguest. The same weights with a 32k window and a long chat is a roommate who invited friends. KV cache grows with tokens. Tools that “just increase context” are how a comfortable 16 GB machine starts paging.

On a 16 GB M-series Mac, I treat 7B Q4 as the daily driver and 13B-class as a special occasion with other apps closed. On 24 GB I get sloppy. On 32 GB and up, the wall moves but does not vanish, because I also open IDEs. Unified memory is shared with the software that makes the model useful. A headless inference box can spend 90 percent of RAM on weights. A MacBook cannot, unless you lie to yourself.

Ollama makes it easy to lie. The model list is a tap. You pull something larger because the name is famous. The daemon will try. Activity Monitor will narrate the tragedy. MLX makes it easier to see dtypes and shapes if you are looking. It will not stop you from loading a 70B fantasy on a 16 GB Air. No stack is a parent.

Quantization: GGUF culture versus MLX weights

Ollama’s world is GGUF quants. Q4_K_M, Q5, Q8, the folklore of which suffix is “the good one.” The folklore is useful. It is also how people cargo-cult a file that does not fit. I pick Q4 for chat and I do not chase Q8 on a memory-bound machine so I can feel pure. Purity is a desktop with 64 GB.

MLX weights come in their own quantized flavors. The conversions have gotten easier. The footgun is mixing a tutorial from six months ago with a model card from this week. Pin versions. If generation looks drunk, check the tokenizer and the quant before you blame the prompt.

Quality at a given RAM budget is close enough on everyday writing tasks that I would not switch stacks for a blind A/B on email drafts. I would switch if I am already in one ecosystem’s tooling. I would also switch if one path supports the model I actually want this month. Catalog coverage still moves. The memory wall does not care about the catalog. It cares about bytes.

Where Ollama hits the wall first

Ollama hits the wall first when you treat it like an always-on service plus a browser plus Slack. The daemon is convenient. Convenience keeps models resident in your mind even when they are not resident in RAM. You start a second model to compare. Now you have two houseguests. Some versions and settings unload better than others. I still assume I must quit and check memory if the Mac stutters. Assuming unload is how I have waited on a beachball while a 13B and a 7B argued.

The API ecosystem also encourages long-lived processes: Open WebUI, a VS Code extension, a second laptop sending requests. Each client is harmless. Together they keep context warm. Warm context is the silent RAM tax. I now cap context in the client even when the model file could go longer. The model file is not the budget. The session is.

Ollama on Apple Silicon is fine. It is not the enemy. The enemy is the pull-and-forget loop the UX is designed around.

Where MLX hits the wall first

MLX hits the wall first when you stack a notebook, a conversion, and a training or LoRA toy “just to see.” The framework makes it easy to stay in Python and allocate another tensor. Fine-tuning is how people discover that inference RAM and training RAM are different sports. If you came for chat, do not start a LoRA on the same machine you are using to ship a client project. Rent a box or wait until night.

MLX also hits the wall when a model’s converted size was measured without the extras your script imports. A demo script is lean. Your script that also loads a retriever and a reranker is not. I have blamed MLX for a swap storm that was a sentence-transformer I forgot about. Unified memory does not itemize a moral cause. It just fills.

When I stay in one MLX generate loop, close the extras, and pick a quant that fits with 30 percent headroom, MLX has been the smoother drinker on the same Mac. Headroom is the whole trick. If you fill 95 percent, you have already lost. The window manager needs a sip.

Thermals and the “fast in the first minute” lie

Both stacks can look heroic in a 20-token screenshot. A 10-minute coding session is the test. Fans spin. Clocks drop. Tokens per second sag. People then declare the other framework faster because they measured the cold start of a small prompt. I measure after the chassis is warm and the battery is unplugged, then again on battery. Laptop users who only bench on power adapters live in a fiction.

Memory pressure makes thermal worse. Swap is disk heat and latency. If you are paging, you are not in a framework race. You are in a storage race. Buy RAM next time, or shrink the model now. I say that as someone who has done the sad math on a 16 GB machine I otherwise love.

A working setup that does not fight itself

One default chat model. One optional bigger model I load after quitting the browser. Context capped to what I need for the job, not what the card advertises. No second daemon “for fun.” If I need an OpenAI-compatible endpoint, Ollama. If I need to poke weights or run an MLX-only build, MLX, and I stop Ollama first. Two engines awake is how you double the lake traffic.

I keep a note of resident size for each model I actually use, measured with the rest of my normal apps open. The internet’s “runs on 16 GB” means “runs if you are a monk.” I am not a monk. I have mail.

For agents and tool loops, I budget more RAM than a single completion suggests. Hidden traces add tokens. The wall arrives mid-conversation, which feels like the model got stupid. It got paged.

Who should pick which

Pick Ollama if you want a local API for existing tools, if you might run the same habit on Linux, and if you will be strict about one model at a time. Pick it knowing the catalog will tempt you into a size you cannot host beside a workday.

Pick MLX if you are already in Python, if you want the more native path, and if you will keep headroom. Pick it knowing you will do more homework when a model is not a one-line pull.

Pick neither 70B on a 16 GB laptop. That sentence should be a sticker on the lid. Distilled and small models are not a consolation prize. They are how unified memory remains a feature.

The wall I hit first, in practice

On my 24 GB machine, Ollama hits first when I am sloppy with two models and a long chat. MLX hits first when I am sloppy with a notebook and an extra embedding model. The framework is not the moral. The shared lake is the moral. Apple Silicon local LLMs are great until you pretend the memory is a GPU you do not have.

I still recommend Ollama to people who want a server. I still open MLX when I want to spend the same bytes more carefully. I close Slack more often than I switch stacks. That is the unglamorous optimization that actually moved the wall: fewer roommates in the lake, then a model that fits, then a context that fits the job. Everything else is a benchmark you will screenshot once and never live in.