The short answer (July 2026): Ollama (current release v0.32.5, July 27 2026) is a free, open-source tool that downloads and runs open-weight language models on your own hardware with a single command. Install via curl -fsSL https://ollama.com/install.sh | sh on Mac or Linux, or irm https://ollama.com/install.ps1 | iex on Windows, then type ollama run llama3.2 to start chatting in seconds. Models run entirely on your machine and never leave it. The tool itself is free; an optional Pro plan ($20/month) lets you route requests to datacenter hardware for models that exceed your local RAM or VRAM.
Last verified: July 29, 2026, against official docs and pricing pages.
Written by Alexis
What Ollama Actually Is
Ollama wraps open-weight models in a local inference server with a clean CLI and an OpenAI-compatible REST API at http://localhost:11434. You install it once, pull a model, and immediately have a chat interface in your terminal or an API endpoint any library can hit.
The tool handles quantization format selection, GPU layer allocation, and context caching automatically. Model files come from ollama.com/library, stored locally after the first pull. On subsequent runs the model loads from disk into VRAM (or RAM if no GPU is present) in seconds.
Ollama itself is free and open-source. The $20/month Pro tier is strictly for cloud offloading, not a gating mechanism on local features.
Install Ollama
macOS and Linux:
curl -fsSL https://ollama.com/install.sh | sh
On Linux this also installs a systemd service so the Ollama server runs in the background on boot.
Windows (PowerShell):
irm https://ollama.com/install.ps1 | iex
A direct installer download is also available at ollama.com.
Docker:
docker run -d -p 11434:11434 ollama/ollama
Verify the install:
ollama --version
GPU requirements Ollama checks on startup:
- NVIDIA: compute capability 5.0 or higher, driver 550 or newer (driver 570 required for older compute 5.0-6.2 cards)
- AMD: ROCm v7 on Linux; Windows support covers select 7000-series and PRO models
- Apple Silicon: Metal API, detected automatically
- Vulkan: fallback acceleration layer for Intel GPUs and additional AMD hardware on Windows and Linux
If no supported GPU is found, or if the model does not fit in VRAM, Ollama falls back to CPU. Output is correct either way; CPU inference is significantly slower.
Run Your First Model
ollama run llama3.2
This pulls the default 3B tag (2.0 GB) if not already local, then opens an interactive chat. Type /bye to exit.
Run a specific size tag:
ollama run llama3.2:1b # 1.3 GB, fast on limited hardware
ollama run gemma3:4b # 3.3 GB, multimodal
ollama run deepseek-r1:7b # 4.7 GB, reasoning model
Pass a one-shot prompt without entering the interactive session:
ollama run llama3.2 "Explain the CAP theorem in two sentences"
Feed an image to a multimodal model:
ollama run gemma3:4b "Describe this chart. /path/to/chart.png"
Core CLI commands you will use daily:
ollama list # models downloaded locally
ollama ps # models currently loaded in memory
ollama pull qwen3:8b # download without opening a chat session
ollama rm qwen3:8b # delete a model from disk
ollama stop gemma3 # unload a model from memory
ollama show llama3.2 # print parameters, template, and license
Pick the Right Model for Your Hardware
Ollama uses Q4 quantization by default, which cuts file size versus FP16. The file size of the quantized model is a reliable proxy for how much free RAM or VRAM you need.
| Model | Tag | File size | RAM/VRAM needed | Strengths |
|---|---|---|---|---|
| gemma3 | gemma3:270m | 292 MB | 1 GB+ | Edge / Raspberry Pi / constrained devices |
| llama3.2 | llama3.2:1b | 1.3 GB | 4 GB+ | Lightweight agents, fast on-device inference |
| llama3.2 | llama3.2:3b | 2.0 GB | 4 GB+ | General chat on laptops with 8 GB RAM |
| qwen3 | qwen3:4b | 2.5 GB | 6 GB+ | Multilingual text, 256K context window |
| gemma3 | gemma3:4b | 3.3 GB | 6 GB+ | Text plus image input on consumer laptops |
| mistral | mistral:7b | 4.4 GB | 8 GB+ | Instruction-following, function calling |
| deepseek-r1 | deepseek-r1:7b | 4.7 GB | 8 GB+ | Reasoning chains, math, logic tasks |
| llama3.1 | llama3.1:8b | 4.9 GB | 8 GB+ | Tool use, 128K context, strong general capability |
| gemma3 | gemma3:12b | 8.1 GB | 12 GB+ | Multimodal reasoning, single mid-range GPU |
| qwen3 | qwen3:14b | 9.3 GB | 12 GB+ | Coding and reasoning on 16 GB cards |
| llama3.1 | llama3.1:70b | 43 GB | 48 GB+ | Production-grade reasoning, multi-GPU rigs |
When a model fits entirely in VRAM you get full GPU throughput. When it partially fits, Ollama offloads layers to RAM automatically; throughput drops proportionally with the number of CPU-offloaded layers. For models that will not fit locally at any quantization level, Ollama Cloud can route requests to datacenter hardware.
Use the Local REST API
Ollama runs at http://localhost:11434 and accepts the same request shape as the OpenAI API, so any client that supports a custom base URL will work against it without code changes.
Generate a completion:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "What is a transformer model?",
"stream": false
}'
Chat with message history:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"messages": [
{"role": "user", "content": "Explain attention mechanisms"}
],
"stream": false
}'
Generate embeddings (use a dedicated embedding model for best results):
curl http://localhost:11434/api/embed -d '{
"model": "nomic-embed-text",
"input": "The quick brown fox"
}'
Check which models are loaded:
curl http://localhost:11434/api/ps
Set "stream": true (the default) to receive token-by-token output as newline-delimited JSON. Set "stream": false to wait for the full response, which is simpler for scripting.
Customize Models with a Modelfile
A Modelfile bakes a system prompt, context length, and sampling parameters into a named local model. You create it once, then ollama run my-model and the configuration is already there.
Create a file called Modelfile:
FROM llama3.2
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
SYSTEM """
You are a concise technical assistant. Answer in plain text only.
Show your reasoning step by step before giving the final answer.
"""
Build and run it:
ollama create my-assistant -f Modelfile
ollama run my-assistant
Key parameters that change output quality most noticeably:
temperature(default 0.8): lower values give more deterministic output; raise it for creative tasksnum_ctx(default 2048): context window in tokens; increase this when you need to paste long documents, but it raises VRAM usetop_p(default 0.9): controls output diversity alongside temperaturerepeat_penalty(default 1.1): reduces looping in long outputs
You can also build from a local GGUF file: FROM ./my-model.gguf. This is the path to use if you have downloaded a model from Hugging Face directly.
Integrate with Python or JavaScript
Both official libraries wrap the REST API with typed functions. They require Ollama to be running locally (or pointed at a remote host).
Python (requires Python 3.8 or newer):
pip install ollama
from ollama import chat
response = chat(
model='llama3.2',
messages=[{'role': 'user', 'content': 'Summarize the halting problem'}]
)
print(response.message.content)
Streaming:
stream = chat(
model='llama3.2',
messages=[{'role': 'user', 'content': 'Write a regex that matches ISO dates'}],
stream=True,
)
for chunk in stream:
print(chunk['message']['content'], end='', flush=True)
JavaScript / TypeScript:
npm i ollama
import ollama from 'ollama'
const response = await ollama.chat({
model: 'llama3.2',
messages: [{ role: 'user', content: 'What is a closure?' }],
})
console.log(response.message.content)
Point either library at a different host:
from ollama import Client
client = Client(host='http://192.168.1.10:11434')
const ollama = new Ollama({ host: 'http://192.168.1.10:11434' })
Both libraries expose the full API surface: chat, generate, embed, pull, rm, list, ps, create, copy, push.
FAQ
Can Ollama run without an internet connection after setup?
Yes. Once a model is pulled to disk, the inference server runs entirely offline. The ollama serve process does not phone home. Only ollama pull and ollama push require a network connection.
How do I run Ollama as a background service on Linux?
The install script sets up a systemd service automatically. Check status with systemctl status ollama and enable auto-start with sudo systemctl enable --now ollama. If you installed manually, the service unit file is not created and you will need to start ollama serve yourself.
Can I use Ollama with LangChain, LlamaIndex, or other AI frameworks?
Yes. Set the base URL in your framework to http://localhost:11434/v1 and use any model name you have pulled. Most frameworks that support OpenAI-compatible endpoints require no other code changes.
What happens if a model is too large for my GPU?
Ollama splits the model automatically, running layers that fit in VRAM on the GPU and the rest on CPU. Generation throughput drops proportionally with the number of CPU-offloaded layers. The practical fix is to use a smaller size tag or a quantization-aware-trained (QAT) variant of the model, which some models like gemma3 offer.
How do I see what is currently in memory and free it?
ollama ps lists loaded models, their memory footprint, and how long they have been resident. Models unload automatically after 5 minutes of inactivity by default. Force an unload immediately with ollama stop <model>.
If this saved you a few hours of setup, get AI Weekly free, 3 issues a week, read by 40,000+ practitioners.