What changes when the model runs on your own machine
I built myself a small tool that searches my own notes and answers questions about them. A few hundred markdown files, an embedding index, a model with 14 billion parameters, all on the laptop. No API key, no network. You call it in the terminal and ask a question.
The question behind it was mundane: how far do you actually get without an API?
The answer I expected was one about quality. The interesting answer was a different one.
The boring finding first
Yes, the local model is worse. It writes more stiffly. On multi-step reasoning across long contexts it loses the thread where a frontier model still holds. Ask it to build an argument out of fifteen sources and you get something that looks like an argument.
But that's the uninteresting part, for a reason you only notice once you use the thing daily: most real work isn't frontier work. It's finding things again, pulling things out, summarising, reformatting, comparing. Where was that. What did I decide back then, and why. Summarise the four notes on this topic. A 14B model does that decently. Not brilliantly – decently. And decently is enough when the alternative is not doing it at all.
The boundary also doesn't run where I assumed, between easy and hard questions. It runs between questions where I can check the answer and questions where I have to believe it.
What actually happens
Two constraints fall away at once, and the second one is the point.
The first: it no longer costs anything per query. That sounds like accounting, but it changes behaviour. I ask differently when asking is free. I experiment. I run the same question three times to see whether the answer holds. I ask the stupid intermediate question I'd have swallowed on a per-token bill. A tool that costs nothing to use is a different tool.
The second: the data doesn't leave the building.
From cost to permission
In my work – regulated healthcare, patient data, Article 9 – the first question about a new tool is never "how good is it". It's: is this even allowed. Where does the data go, who is controller and who is processor, is there a data processing agreement, is a third country involved, how long is it retained, who could look inside if it came to that. With health data that is the precondition, and a tool that fails that chain isn't a worse tool – it isn't a tool.
A model on your own machine doesn't answer that chain better. It makes it moot. There is no transfer, so no recipient, so no contract, so no third country. The question that kills most interesting ideas in regulated environments stops applying.
The difference is permission. And permission decides what can be thought of at all, not just how well it then runs.
I had underestimated that. I'd thought of local models as the cheaper, weaker version of the same thing. They're a different thing.
In practice: ideas that would have died in the conversation are suddenly worth a prototype. Not because they work better, but because you're allowed to build one and show it before anyone has to review a contract. And a thing that runs changes a conversation differently than a concept that looks good.
Where it does stop
So this doesn't read as a recommendation, which it isn't:
It's work. Choosing the model, building the index, tuning the chunking, and none of it is self-explanatory. The machine gets warm and loud. For tasks where reasoning genuinely matters I still reach for the frontier model, and I see no reason to hide that.
The usable rule that came out of it: local for anything that touches my own data and where I can check the result. Remote for anything where I'm buying reasoning and the input is harmless. That's not a position. It's a division of labour.
Why I'm writing this down
Because in almost every discussion about AI in sensitive environments, privacy shows up as the brake – as the thing preventing the interesting part. Since building this tool I think that's a mistake. The constraint was never "no AI". It was "no transfer". Those are two completely different sentences, and the second one has a technical answer.