Quick takeaways
- Ollama is the easiest way to run local LLMs on a laptop or workstation.
- LM Studio offers the best GUI for experimenting with models and chatting locally.
- llama.cpp is the lightest, most portable engine for edge and custom builds.
- vLLM is the best choice for high-throughput local or on-prem APIs.
How to think about AI model choice
Andrej Karpathy explains what large language models are good at and how to pick the right model for a task.
Top picks
Ollama
One-command model downloads and a simple CLI for running Llama, Mistral, Qwen, and more locally.
LM Studio
A polished desktop app for browsing, downloading, and chatting with local models.
vLLM
High-throughput inference with continuous batching for local APIs and small teams.
llama.cpp
Lightweight, CPU-friendly, and highly portable across devices and operating systems.
Tool comparison
| Tool | Best for | Strengths | Considerations |
|---|---|---|---|
| Ollama | Quick local setup | Simple CLI, curated models, easy updates | Less tuning than raw engines |
| LM Studio | GUI experimentation | Discover models, chat UI, local server mode | Desktop only; less suited to production |
| llama.cpp | Edge and custom builds | CPU support, quantization, broad hardware | Lower-level; requires more setup |
| vLLM | Local API serving | High throughput,PagedAttention, OpenAI-compatible | Best with NVIDIA GPUs and Linux |
Popular model choices
| Model family | Best for | Notes |
|---|---|---|
| Llama 4 | General chat, coding, agents | Strong open weights, large ecosystem |
| Qwen 3 | Multilingual and coding | Good performance at smaller sizes |
| Mistral Large 3 | Reasoning and retrieval | Efficient architectures |
| DeepSeek-V3.5 | Reasoning on a budget | Strong distilled versions available |
How to pick
Without AI vs. with AI
| Task | Without AI | With AI |
|---|---|---|
| Data privacy | Teams send sensitive prompts to cloud APIs and negotiate terms. | Local LLMs keep data on owned hardware and avoid third-party exposure. |
| Model choice | Teams are locked into whatever cloud providers offer. | Local deployments let you choose, quantize, and fine-tune open weights. |
| Cost predictability | API bills scale unpredictably with usage. | Local inference trades hardware and electricity for fixed costs. |
| Customization | Vendors limit fine-tuning and prompt engineering options. | Local models can be adapted with domain data and custom inference stacks. |
| Offline access | Users need an internet connection for every AI task. | Local LLMs work offline after the model is downloaded. |
FAQ
What hardware do I need to run a local LLM?
For 7B–14B models, a modern laptop with 16–32 GB RAM or an 8–12 GB GPU works. For 70B+ models, you need multiple high-memory GPUs or a server.
Are local LLMs private?
More private than cloud APIs because data stays on your machine, but check each tool's telemetry and update behavior. Disable cloud sync when available.
Which local model is best for coding?
Llama 4, Qwen 3, and DeepSeek distilled variants are popular. Pair them with a coding-focused prompt or local agent setup.
Can I use local LLMs for production?
Yes, with vLLM or Ollama in server mode and proper monitoring. Expect to tune quantization, batching, and context limits for your workload.
Is Ollama better than LM Studio?
Ollama is simpler for CLI and scripting. LM Studio is better for browsing models and chatting through a GUI. Many people use both.
What is the easiest way to run a local LLM?
Ollama and LM Studio are popular starting points for beginners.
Can I run a local LLM without a GPU?
Yes, smaller quantized models run on CPU, though slower than on GPU.
Are local LLMs as good as cloud models?
Smaller local models can match cloud models on simple tasks. Larger local models need serious hardware.
What is quantization?
It reduces model precision to save memory and speed up inference, with small quality trade-offs.
Can I use a local LLM with my IDE?
Yes, tools like Continue, Cody, and local API servers integrate local models into editors.