How to Run LLMs Locally: Ollama, llama.cpp, and Hardware Requirements

Disclosure: This article may contain affiliate links. We only recommend products we believe in.

Running large language models on your own machine used to require a PhD in CUDA optimization and a server rack in your closet. That changed fast. In 2026, local LLM tooling is mature enough that any developer with a halfway decent machine can get a useful coding assistant running in under ten minutes.

Here’s the practical guide, covering hardware, software, model choices, and the configuration details that actually matter.

Why Run Models Locally?

  • Privacy: Your code never leaves your machine. No terms of service to parse, no data retention policies to worry about.
  • Cost: No API fees after the hardware investment. If you already own a gaming GPU or an M-series Mac, the incremental cost is zero.
  • Speed: No network latency, no rate limits, no waiting in queue behind other users.
  • Offline: Works on a plane, in a coffee shop with bad WiFi, or in an air-gapped environment.
  • Customization: Fine-tune models on your own codebase for suggestions that match your project’s patterns and conventions.

If privacy and cost aren’t concerns, cloud-based tools like Claude or GitHub Copilot are still more capable for most tasks. But the gap is closing every quarter.

Hardware Requirements

Minimum (7B Parameter Models)

  • 16 GB RAM
  • Modern CPU (Apple M1+ or recent AMD/Intel)
  • 10 GB storage per model
  • No GPU required (but slower)

A 7B model on CPU runs at roughly 5-15 tokens per second on a modern processor. That’s usable for chat-style interactions but sluggish for code completion where you want near-instant responses.

  • 32 GB RAM
  • GPU with 8+ GB VRAM (RTX 3070/4070 or better)
  • 50 GB storage
  • SSD (model loading speed matters more than you’d expect)

This is the sweet spot for most developers. A 13B code model on GPU gives you fast completions and solid reasoning without breaking the bank. Apple Silicon users with 32GB unified memory can run these models comfortably too, since the memory is shared between CPU and GPU.

Enthusiast (70B Models)

  • 64 GB RAM
  • GPU with 24+ GB VRAM (RTX 4090, A6000)
  • 100 GB storage

At this tier, you’re approaching the quality of commercial APIs for many tasks. The RTX 4090 remains the best consumer option for running large models, though its $1,600+ price tag makes it a serious investment.

Ollama: The Easiest Path

Ollama is the tool that made local LLMs accessible. It handles model downloads, GPU detection, quantization selection, and serves an API, all with a single command.

Installation

curl -fsSL https://ollama.ai/install.sh | sh

On macOS, you can also install via Homebrew: brew install ollama.

Running Your First Model

ollama run llama3

That’s it. Ollama downloads the model and starts an interactive chat. First run takes a few minutes to download depending on your connection.

Useful Models for Developers

ollama run codellama        # Code generation, multiple sizes
ollama run mistral          # General purpose, fast
ollama run llama3:70b       # Largest, most capable
ollama run phi3             # Small but surprisingly good
ollama run deepseek-coder-v2  # Strong coding model
ollama run qwen2.5-coder    # Excellent for Python/TypeScript

For coding specifically, deepseek-coder-v2 and qwen2.5-coder tend to outperform general-purpose models of the same size. They’re trained on more code data and handle language-specific patterns better. If you need advanced reasoning and step-by-step logic, read our complete guide on how to run DeepSeek-R1 locally using distilled models.

API Mode

Ollama exposes a local API compatible with the OpenAI format, which means most tools that work with OpenAI’s API work with Ollama too:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Explain TCP/IP in simple terms"
}'

You can also use the OpenAI-compatible endpoint at http://localhost:11434/v1/ for tools that expect the standard OpenAI format.

llama.cpp: Maximum Control

For users who want fine-grained control over inference, llama.cpp is the foundation that most local LLM tools (including Ollama) build on. Running it directly gives you access to every parameter:

# Clone and build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make -j

# Run a model
./main -m models/codellama-13b-instruct.Q4_K_M.gguf \
  --n-gpu-layers 35 \
  --ctx-size 4096 \
  --temp 0.2 \
  --repeat-penalty 1.1

Key parameters to tune:

  • n-gpu-layers: How many model layers to offload to GPU. More layers = faster, but limited by VRAM.
  • ctx-size: Context window in tokens. Larger windows use more memory but let the model see more code.
  • temp: Lower values (0.1-0.3) produce more deterministic output, better for code generation. Higher values (0.7-1.0) produce more creative output.

Most developers should just use Ollama. llama.cpp is for situations where you need to squeeze out specific performance characteristics or integrate at a lower level.

Quantization Explained

Models are compressed using quantization to reduce memory requirements. The naming convention (Q4_K_M, Q5_K_S, Q8_0) indicates the bit precision and quantization method:

  • Q4_K_M: 4-bit quantization, medium quality. About 50% size reduction from full precision. Slight quality loss on complex reasoning tasks, but barely noticeable for code completion. Best balance for most users.
  • Q5_K_M: 5-bit. Better quality than Q4 with a modest size increase. Good middle ground if you have the VRAM.
  • Q8_0: 8-bit. Near-original quality. Larger files, needs more memory.
  • FP16: Full precision. Original quality. Requires roughly 2x the RAM of Q4. Only worth it if you have the hardware and need maximum accuracy.

A practical rule: start with Q4_K_M. If the output quality feels lacking for your use case, step up to Q5. Most developers can’t tell the difference between Q5 and FP16 for day-to-day coding tasks.

Connecting Local Models to Your IDE

Running a model in a terminal is useful for testing, but the real productivity gains come from IDE integration.

VS Code with Continue: The Continue extension connects to Ollama and provides Copilot-like tab completion plus a chat sidebar. Configure it in .continue/config.json to point at your local Ollama instance. This is the setup we recommend for most developers exploring self-hosted AI tools.

Tabby: A self-hosted code completion server that indexes your codebase. Point it at your Ollama models and it provides context-aware completions across your entire project.

LM Studio: If you prefer a GUI for managing models, LM Studio provides a desktop app with a built-in chat interface and local API server. Good for developers who don’t want to touch the command line for model management.

Performance Tuning Tips

A few things that make a noticeable difference:

Keep models loaded. Ollama keeps models in memory by default after the first load. Cold-starting a model takes 5-15 seconds; subsequent queries are near-instant. Don’t restart the Ollama service between sessions unless you need the memory back.

Match model size to your task. Use a small, fast model (3B-7B) for code completion where speed matters, and a larger model (13B-34B) for complex questions and debugging. You can run multiple models simultaneously if you have enough memory.

Use the right context window. Larger context windows use more VRAM. If you’re doing simple completions, a 2048-token context is plenty. For analyzing larger files, increase to 4096 or 8192.

Monitor GPU utilization. Use nvidia-smi (NVIDIA) or Activity Monitor (Mac) to check that your model is actually using GPU acceleration. CPU-only inference on large models is painfully slow.

Troubleshooting Common Issues

Model runs slowly: Check that GPU offloading is working. On NVIDIA, make sure CUDA drivers are installed. On Mac, ensure you’re running a build with Metal support.

Out of memory errors: Try a smaller quantization (Q4 instead of Q5/Q8), reduce context window size, or switch to a smaller model. Closing other GPU-heavy applications (games, video editors) frees up VRAM.

Garbled output: Usually caused by a corrupted download. Delete the model (ollama rm modelname) and pull it again.

API connection refused: Make sure the Ollama service is running (ollama serve) and listening on the expected port (default 11434).

What to Expect vs. Cloud Models

Local models are good and getting better, but they’re not at parity with the best cloud options yet. A 70B local model is roughly competitive with GPT-3.5 for code generation. For complex multi-step reasoning, debugging intricate bugs, or working with obscure frameworks, cloud models like Claude and GPT-4 still have a clear edge.

The practical approach: use local models for routine coding tasks, privacy-sensitive work, and offline scenarios. Switch to cloud models when you hit the limits of what local inference can handle. The two approaches complement each other well.