llama.cpp vs Ollama on a Home PC: When the Raw Engine Beats the Friendly Wrapper

Sora Nguyen

Sora Nguyen

November 16, 2026

llama.cpp vs Ollama on a Home PC: When the Raw Engine Beats the Friendly Wrapper

You already have a model running on your home PC. Maybe you installed Ollama, typed one command and had a chatbot answering in a minute. Then you read forum threads where people tune flags, fit more context on the same card or run a model the day it is released, and almost all of them mention llama.cpp. Is the raw engine worth the extra effort, or is Ollama already doing everything you need? It depends on what you want to control, and the differences are concrete.

How Ollama and llama.cpp are related, and where they now diverge

llama.cpp is an open-source inference engine started by Georgi Gerganov and now maintained under the ggml-org organization on GitHub. It is built on GGML, a C tensor library, and it defined the GGUF file format that most local model downloads now use. It ships a set of command-line tools, most importantly llama-cli for interactive use and llama-server for serving a model over HTTP.

Ollama is a Go application that wraps model management, a background service, a REST API and a model registry around an inference core. For most of its history that core was llama.cpp, and the Ollama README still credits the llama.cpp project in its supported backends section. That changed in part in May 2025, when Ollama announced a new engine for multimodal models. According to Ollama’s own blog post, the project had relied on llama.cpp for model support, but it now uses the GGML tensor library directly from Go for newer architectures, so each model’s implementation can be self-contained.

So the two share a foundation (GGML and GGUF) but are no longer simply “the engine” and “the wrapper.” Some Ollama models run on its own engine, and features land in one project before the other. You are comparing two runtimes with different priorities: llama.cpp exposes nearly every knob, while Ollama hides most of them behind defaults.

Getting a model loaded: GGUF files versus the Ollama registry

With Ollama, the default flow is ollama pull or ollama run followed by a model name from Ollama’s library. Models are stored as content-addressed blobs in Ollama’s own directory, which is convenient until that directory grows large on a small system drive. If that has already happened to you, there is a walkthrough on moving Ollama models to another disk without downloading them again.

Ollama can also run GGUF files that are not in its library. Hugging Face documents a direct form:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0

Or you can import a local file with a Modelfile:

FROM /path/to/model.gguf
PARAMETER num_ctx 8192

followed by ollama create my-model. Ollama’s import docs note that it does not quantize GGUF models on import; if you want a different quantization, you produce it first with a tool such as llama.cpp’s llama-quantize.

llama.cpp works with plain GGUF files on disk. You point it at a file with -m, or let it download one from Hugging Face with -hf:

llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080

There is no registry and no blob store. A model is a file you can copy, put on any drive and share with other tools that read GGUF, and you can keep several quantizations of one model side by side.

The knobs llama.cpp gives you that Ollama keeps behind defaults

This is the real dividing line. Ollama exposes a short list of parameters in its Modelfile reference: num_ctx, sampling settings such as temperature, top_k, top_p and min_p, stop sequences, num_predict and a few others. Server-wide behavior is set with environment variables such as OLLAMA_CONTEXT_LENGTH, OLLAMA_KV_CACHE_TYPE, OLLAMA_FLASH_ATTENTION, OLLAMA_NUM_PARALLEL and OLLAMA_KEEP_ALIVE. GPU layer placement is mostly decided for you based on available memory.

llama-server exposes far more, directly on the command line. A few that matter on home hardware:

  • -ngl / --n-gpu-layers sets how many layers go into VRAM. Recent builds default to choosing automatically, but you can override it when you want to leave room for something else on the GPU.
  • -c / --ctx-size sets the context window per server.
  • -ctk and -ctv set the data type of the K and V cache separately, so you can trade a little quality for a much smaller cache.
  • -fa / --flash-attn switches flash attention on, off or to auto.
  • -ot / --override-tensor lets you decide which tensors live on which device, using patterns.
  • --n-cpu-moe keeps the mixture-of-experts weights of the first N layers on the CPU, which is one of the most effective ways to run a large MoE model on a card that cannot hold it.
  • --jinja uses the model’s own Jinja chat template, which matters for tool calling with newer models. It is on by default in current builds.

Computer fan and copper heat pipes inside a PC

The tensor overrides are where the difference shows up most on a mid-range GPU. A MoE model has few active parameters per token but a large total size. With llama.cpp you can keep attention and shared weights on the GPU and push expert weights into system RAM. Ollama makes its own placement decisions, which suit common cases but cannot be steered tensor by tensor. If you are working out how much VRAM you really need for this kind of split, the article on whether a used RTX 3060 12 GB is still the sensible cheap GPU covers the memory side in detail.

Quantization control follows the same pattern. With llama.cpp you choose the exact GGUF file, and you can make your own from a full-precision model with llama-quantize. With Ollama you pick a tag; if the tag you want does not exist in the library, you fall back to a Hugging Face GGUF or an import.

Speculative decoding and other speed tricks

Speculative decoding uses a small, fast draft model to propose several tokens, which the large model then checks in one pass. When the draft guesses well, generation speeds up without changing the output. In llama-server you load a draft model with -md (also spelled --model-draft), and the server docs list further options for draft length and n-gram-based drafting. Older guides mention a --draft flag; the current server README says that flag has been removed, so check llama-server --help on your build rather than copying old commands.

Ollama’s Modelfile reference now includes a draft_num_predict parameter that applies when a draft model is available for a model, so the idea is not foreign to it. The difference is that llama.cpp lets you pair any compatible draft model with any main model yourself, which is what people experimenting with speed usually want.

APIs and front ends: both speak OpenAI, both work with Open WebUI

Both runtimes offer an OpenAI-compatible API, which is what most tools expect. Ollama listens on port 11434 by default and serves both its native /api/ endpoints and an OpenAI-compatible subset under /v1/, including chat completions, completions, embeddings and model listing. llama-server serves OpenAI-compatible chat completion and embedding routes on whatever port you give it, and can expose a reranking endpoint with --rerank.

llama-server also ships a built-in web UI, enabled by default. Open the server’s address in a browser and you get a clean chat interface with no extra container. Ollama has its own desktop app on Windows and macOS, but on a headless Linux box most people add a separate front end.

Open WebUI connects to either one: natively to Ollama, or to llama-server as an OpenAI-compatible connection. If you are choosing a household front end in the first place, the comparison of Open WebUI and LibreChat goes through accounts, permissions and how each handles local and cloud models together.

Model switching is the difference that catches people out. Ollama loads models on demand when a request names them and keeps them in memory for OLLAMA_KEEP_ALIVE (five minutes by default). A plain llama-server process serves the model you started it with; for several models behind one endpoint you run several servers or put a proxy in front. More setup, but nothing gets evicted unexpectedly.

GPU backends: CUDA, ROCm, Vulkan and Metal

llama.cpp has the widest backend list of any local runtime. Its README lists Metal for Apple Silicon, CUDA for NVIDIA, HIP for AMD, Vulkan for most GPUs, SYCL for Intel GPUs, plus a long tail of others. The project publishes Docker images for several of these, including ghcr.io/ggml-org/llama.cpp:server, :server-cuda, :server-rocm, :server-vulkan and :server-intel.

Ollama’s GPU documentation lists NVIDIA CUDA, AMD ROCm, Apple Metal and Vulkan on Windows and Linux, with Vulkan enabled by default when that backend is installed. For many AMD and Intel users that removes the main reason to leave Ollama. Still, with llama.cpp you choose exactly which build you run, and on an AMD card where ROCm is awkward, switching to a Vulkan build of the same tool and comparing is easy. On Apple Silicon both use Metal through GGML.

Several GPUs, CPU offload and many users at once

With two GPUs, llama.cpp lets you choose a split mode with --split-mode and set the proportion each card gets with --tensor-split. Layer splitting is the default and keeps traffic between cards low. There is also an RPC backend for spreading a model across machines, which is more about fitting a model at all than about speed. Ollama will spread a model across multiple GPUs when it does not fit on one, but you do not get per-card ratios.

When a model spills into system RAM, both runtimes keep running, just slower. llama.cpp gives you more say over what spills, through -ngl, -ot and --n-cpu-moe. Ollama makes the choice for you based on free memory at load time.

For concurrency, llama-server uses slots (-np / --parallel) with continuous batching enabled by default, so several users or agents can share one loaded model. Note that the context you set is divided across slots, so plan context size with that in mind. Ollama handles concurrency with OLLAMA_NUM_PARALLEL per model and OLLAMA_MAX_LOADED_MODELS for how many models can sit in memory together. Its FAQ points out that memory use scales with parallel requests multiplied by context length, which is the same trade-off in a different wrapper.

Home office at night with a monitor and desk lamps

New model support and how often things change

llama.cpp moves very fast. New builds are published continuously, and support for new architectures often appears in llama.cpp first, frequently within days of a model release, because model authors and the community contribute it directly. The trade-off is churn: flags get renamed or removed, defaults change and a command from a six-month-old guide may not work today.

Ollama releases less often and packages a narrower set of models. A new model may arrive a little later, or first through its own engine if the team prioritized it. Once in the library, it usually just works with the right template and settings. For GGUFs from Hugging Face that Ollama does not package, compatibility depends on whether its runtime supports that architecture; a GGUF that loads in a current llama.cpp build is not guaranteed to load in Ollama.

Picking a runtime by the job in front of you

Stay with Ollama when:

  • You want a background service that loads whatever model a request asks for and unloads it when idle.
  • Your models come from the Ollama library and you are happy with its quantization choices.
  • Several apps on your network already have an Ollama integration, as many home automation and note tools do.
  • You would rather not track release notes and changing flags.

Move to llama.cpp (usually llama-server) when:

  • You are fitting a model that is slightly too big for your GPU and need tensor-level placement, MoE offload or a quantized KV cache tuned per model.
  • You want a specific GGUF quantization, or to make your own.
  • You want speculative decoding with a draft model of your choice.
  • You need a model the week it is released.
  • Your GPU works better with one particular backend build, such as Vulkan on an AMD or Intel card.
  • You want a single binary with a built-in web UI and an OpenAI-style API, and nothing else running.

Running both is also reasonable if they do not fight over VRAM: Ollama for everyday chat and integrations, and a llama-server on another port for the one large model you tuned by hand. Open WebUI can point at both.

LM Studio sits somewhere in between: it is a desktop app that uses llama.cpp (and MLX on Macs) under the hood with a graphical model browser. If a GUI is what you actually want, the comparison of Ollama and LM Studio covers that path.

A first llama-server setup that mirrors what Ollama gave you

If you decide to try llama.cpp, start with a command that reproduces your Ollama experience, then change one thing at a time:

llama-server 
  -hf ggml-org/gemma-3-4b-it-GGUF 
  -c 8192 
  -ngl 99 
  --host 0.0.0.0 --port 8080

Setting -ngl to a number higher than the model’s layer count means “put everything on the GPU.” Open http://your-server:8080 for the built-in chat, or add it to Open WebUI as an OpenAI-compatible connection with the base URL ending in /v1. Binding to 0.0.0.0 exposes the server to your whole network, so only do that on a trusted LAN or behind a reverse proxy with authentication; the default is to listen on localhost only.

For Docker on an NVIDIA card, use the server-cuda image with --gpus all, a volume for your models and the same flags. Then try a quantized KV cache with -ctk q8_0 -ctv q8_0, compare speed with and without flash attention, and only then move to tensor overrides. Change one flag at a time so you know which one helped.

More articles for you