Run an open-source model locally
Set up an open-weight model on your own machine — and the security trade-offs.
Updated 31 Jul 2026 fast-moving — check a current source
Model names and hardware guidance checked late July 2026 — open-weight releases move faster than anything else covered here.
Running a model locally means the text you send it never leaves your machine. For sensitive work, that is the whole point: no tenant boundary to reason about, because nothing crosses one. The trade-off is that you take on everything the cloud provider used to handle — capability, updates, and security.
What you need
A recent open-weight model and a runner that hides the complexity. Tools like Ollama or LM Studio let you pull a model and chat with it in a few minutes. The practical limit is memory: a small quantised model runs on a modern laptop; larger ones want a capable GPU and plenty of RAM.
The families worth starting with change every few months. As of mid-2026 the compact end — Google’s Gemma, the smaller Qwen builds, and the long-standing Llama and Mistral lines — is where laptop-class options live. Check a current roundup rather than trusting a name you read last year.
Start small. A 7–8B parameter model in a quantised form is enough to judge whether local inference fits your workflow before you invest in hardware.
”Open” no longer means “runnable”
Worth setting expectations before you go shopping. Several 2026 open-weight releases are at frontier scale — hundreds of billions to trillions of parameters. The weights are genuinely open and you may download and modify them freely, but serving them needs a cluster, not a workstation.
So “open-source model” now spans two very different propositions: sovereignty over your infrastructure at datacentre scale, and a decent assistant on the machine in front of you. This page is about the second. Filter any release announcement by parameter count against your actual hardware before it earns your attention.
The security trade-offs, honestly
Local isn’t automatically secure — it’s differently secure.
- You own patching. A model or runner with a known issue stays vulnerable until you update it. Nobody does it for you.
- Supply chain matters. Pull weights and tools from sources you trust. A model file is data, but the surrounding tooling is code that runs on your machine.
- Capability gap. Local models are smaller than frontier hosted ones. For hard reasoning they will underperform — verify output as carefully as ever, per how these tools work.
- Data still needs handling. “On my machine” isn’t “safe” if the machine is unencrypted, shared, or backed up to somewhere uncontrolled.
Local inference removes the cloud boundary problem and hands you the maintenance and hardening that came with it. That’s a fair trade for sensitive work — if you actually do the maintenance.
Where this fits
Local models are one answer to the shadow-AI problem: give people a sanctioned, private option so they stop pasting secrets into random web tools. Pair it with real data-loss-prevention controls rather than treating it as a substitute for them.
Sources
Everything above was checked against these on 31 Jul 2026. Providers change things without notice — if a detail matters to a decision, follow the link.