Local LLM vs ChatGPT: What Do You Actually Lose?

For two weeks this spring I cancelled my ChatGPT habit and did everything on a local model instead. Not as a protest. I wanted to write down, task by task, what I actually lost, because the conversations I kept having about local AI were either “it’s basically as good now” or “it’s a toy,” and neither matched what I saw when I used it for real work.

The setup was not a toy: a desktop with a 24 GB graphics card, running a 32B-class model at Q4 for most things and a 20B-class model when I wanted speed, through Open WebUI in a browser tab that looked a lot like ChatGPT. I kept a text file open and logged every time I reached for the cloud and stopped myself.

By the end the file had four headings. They are the four things you give up: raw answer quality on hard problems, the built-in tools, knowledge of anything recent, and comfortable long context. How much each one matters depends almost entirely on what you ask. This piece is about capability only. Privacy and monthly cost are real questions, but they are separate from whether the answer is any good.

1. Quality: you lose the top end, not the middle

The first surprise was how rarely I noticed the difference. Rewriting a paragraph, turning notes into an email, explaining a stack trace, summarizing a short article, drafting a regular expression, suggesting names for a function: the 32B model handled all of it well enough that I would not have been able to pick it out blind. For the everyday middle of the distribution, a good local model in the 14–32B range is genuinely fine.

The losses showed up at the top end, and they showed up as confident mistakes rather than obvious failures.

The one that made me write the first entry in my file was a SQL question. I had a reporting query with a window function and a join that was double-counting rows in one edge case. I pasted it in and asked why. The local model gave me a fluent, plausible explanation about the window frame, rewrote the query, and the rewrite had the same bug. I asked again with more detail and got a second fluent explanation of a different wrong cause. When I finally broke my rule and pasted the same question into a current reasoning model in ChatGPT, it spent some time thinking, identified the duplicate-producing join, and explained exactly which rows were duplicated.

That pattern repeated. Multi-step reasoning, debugging where the cause is two steps removed from the symptom, math that needs to be carried through several stages, and anything where the model needs to hold several constraints at once: those are where frontier cloud models are clearly ahead. The best of them also “think” for a while before answering, spending extra computation checking their own work. You can run open reasoning models locally, and some are very good, but the ones that fit on consumer hardware are a class below the largest cloud versions, and thinking locally is slow because every one of those hidden reasoning tokens has to be generated on your card.

Writing quality was a subtler loss. Local drafts were competent and a little flat. They leaned on the same transitions and the same tidy three-point structure. The cloud models were better at matching a voice from a few examples and at knowing when to stop.

2. Tools: you lose the features you forgot were features

This was the biggest practical loss, and the one I underestimated most. ChatGPT is not just a model. It is a model wrapped in a set of tools that work without you thinking about them:

• Running code. Upload a CSV, ask for a chart or a pivot, and it writes and executes Python in a sandbox, then shows you the result. I do this more than I realized.
• Reading files. PDFs, spreadsheets, slide decks and images get parsed and handed to the model properly, including tables and scanned pages.
• Web search, triggered automatically when the question needs it, with sources.
• Vision and voice. Photograph a whiteboard or a broken appliance label and ask about it. Talk to it in the car.
• Image generation, memory across conversations, and a document editing mode.

Locally, every one of those is possible and none of them is free. Open WebUI can do web search if you point it at a search engine you run or rent. It can do document retrieval, but its PDF handling depended on which extraction engine I configured, and a scanned invoice came through as nonsense until I added OCR. There are code-execution options, but setting one up safely is an afternoon’s work. Vision models run locally, but the ones that fit alongside a 32B text model on 24 GB are noticeably weaker at reading small print than the cloud ones.

The deeper problem is tool calling itself. For a model to use a tool, it has to decide to call it and format the request correctly. Frontier cloud models are trained heavily on this and rarely fumble it. Local models, especially smaller ones, are much less reliable: they skip the search when they should have used it, invent the result instead of waiting for it, or produce a malformed call that the front end silently drops. In my two weeks, the local setup answered from memory several times when it should have searched, and I only noticed because the answer was out of date.

3. Freshness: the model is frozen on the day it was trained

Every model has a training cutoff, a date after which it knows nothing. Cloud models have one too, but they paper over it with search. A local model without a working search setup is a very well-read person who has been in a cabin since the cutoff and does not know it.

That matters more than it sounds for technical work. Mid-way through the experiment I asked for help with a JavaScript library that had released a new major version a few months earlier. The local model wrote clean, confident code against the old API. Two of the functions it used had been renamed and one had been removed. Nothing in the answer suggested any doubt. ChatGPT, asked the same question, searched, found the migration guide, and used the new names.

The same applies to prices, product specifications, news, laws that changed, software versions, sports results and anything else that moves. For questions about the stable world (how TCP works, what a word means, how to structure an essay) freshness does not matter. For the moving world, a local model without search is a liability precisely because it never says “I don’t know about anything after last year.”

You can fix this partly by feeding the model the current information yourself: paste the changelog, attach the documentation, connect a search engine. That works. It also means you are doing the part the cloud tool did for you.

4. Context: you lose room to paste

Cloud models accept very long inputs: whole reports, long contracts, big chunks of a codebase. More importantly, the best of them stay coherent across those inputs, finding the clause on page 38 that contradicts page 4.

Locally, the context window is limited by memory. Every token of context costs memory on top of the model weights, so on a 24 GB card running a 32B model I could comfortably use around 16,000 tokens before things got tight. That is about 12,000 words, which sounds like a lot until you paste a long contract and a set of meeting notes and ask how they relate. Some models advertise 128K windows, but advertising a window and running it on your hardware are different things.

Even inside the window, smaller models lose track of material in the middle of long inputs more readily than frontier models do. I tested this crudely with a 40-page policy document split into pieces: the local model answered questions about the beginning and end well and was noticeably shakier about sections in the middle. It is also slower: a long paste on a local card can take a while to process before the first word appears.

What you do not lose, and one thing you gain

The list of things that worked just as well locally was longer than the list of losses, and it covered most of what I type on an ordinary day: short rewrites, explanations of concepts that have not changed in years, drafting from notes, classification, extraction from text I paste in, brainstorming, and code questions about stable languages and libraries.

There was also one genuine gain: the model does not change under you. Cloud services update their models regularly, and with each update the tone, the length of answers and sometimes the behaviour on specific prompts shifts. Twice in the last year I have had a carefully tuned prompt stop working after an update. My local model gives the same kind of answer today that it gave two weeks ago, and it will next month too, because the file on my disk has not changed. For anything scripted or repeated, that consistency is worth a lot.

How I decide now

After two weeks I went back to using both, but with a clearer rule than before. I ask four questions about the task, which map to the four losses:

1. Is it hard? Multi-step reasoning, tricky debugging, work where a confident wrong answer is expensive: cloud.
2. Does it need a tool? Running code on data, reading a messy PDF, looking at a photo, searching: cloud, unless I have built and tested that exact tool locally.
3. Does it depend on anything recent? New versions, current prices, news: cloud with search, or local only if I paste the current source in myself.
4. Is the input long? Whole contracts, long codebases, several documents at once: cloud.

If the answer to all four is no, the local model is not a compromise. It is simply the tool on my desk, and it does the job. That turned out to be about two thirds of what I used to send to ChatGPT. The other third is why the subscription came back.