Ollama is an open-source runtime for executing 📝Large Language Models (LLMs) locally on personal or server hardware, keeping prompts and data on-device.

Ollama bundles model weights, configuration, and runtime into a single Modelfile and exposes a unified CLI and REST API for downloading, managing, and querying models without a cloud connection. It ships for macOS, Linux, and Windows and supports a curated library of popular models including 📝LLaMA, 📝Mistral, Qwen, DeepSeek, and Gemma, alongside any custom fine-tune in GGUF format. Its OpenAI-compatible API served on localhost:11434 simplifies migration from cloud inference and has made Ollama the default runtime for privacy-preserving and offline AI workflows.

Key Facts

Category: Local LLM runtime
Origin: Launched 2023 as an open-source project
License: MIT
Platforms: macOS, Linux, Windows
API: OpenAI-compatible REST on localhost:11434
Model formats: GGUF (quantized), Modelfile (packaged)
Home: ollama.com

How It Works

Modelfile — a declarative bundle of weights, parameters, system prompts, and adapters; analogous to a Dockerfile for language models
CLI — ollama pull, ollama run, and ollama create handle the full lifecycle from download to inference to custom packaging
REST API — an HTTP server on localhost:11434 exposes endpoints compatible with the OpenAI API, so existing SDKs work with a base-URL swap
Quantization — ships quantized GGUF variants (Q4, Q5, Q8) so consumer hardware can run models that would otherwise require datacenter GPUs
Official distributions — first-party ollama/ollama Docker image and an ollama-python library make Ollama a drop-in component in containerized stacks and Python agents
Integrations — native support in 📝Claude Code via 📝Model Context Protocol (MCP), plus LangChain, LlamaIndex, Continue, and most agent frameworks

Why It Matters

Ollama makes local inference practical for non-specialists. Before it, running an LLM on your own hardware required stitching together llama.cpp, GGUF files, and Python dependencies; Ollama reduces that to a single install and a single command. For builders working with sensitive data, sovereignty-oriented architectures, or offline-first systems, it removes the cloud dependency without sacrificing developer ergonomics.

FAQ

What is Ollama used for?

Running large language models locally — on a laptop, workstation, or self-hosted server — without sending prompts or data to a cloud provider. Common use cases include privacy-sensitive chat, offline agent workflows, and local RAG pipelines.

Which models does Ollama support?

The built-in library includes LLaMA, Mistral, Qwen, DeepSeek, Gemma, Phi, and dozens of others. Any model available in GGUF format can be imported via a Modelfile.

Is Ollama free?

Yes. Ollama is open-source under the MIT license. Costs are limited to hardware and electricity — there are no per-token fees.

How does Ollama compare to llama.cpp?

Ollama is built on top of llama.cpp and wraps it with model management, an OpenAI-compatible server, and a simple CLI. llama.cpp remains the lower-level inference engine; Ollama is the developer-friendly surface.

Does Ollama work with the OpenAI SDK?

Yes. Ollama exposes an OpenAI-compatible /v1/chat/completions endpoint, so most OpenAI client libraries work by pointing the base URL at http://localhost:11434/v1.

📝LLaMA — Meta's open-weight model family, Ollama's flagship supported lineage
📝Mistral — high-efficiency European open-weight models available in Ollama
📝OpenClaw — local agent runtime Ollama pairs with for sovereign AI stacks
📝Model Context Protocol (MCP) — standard for connecting Ollama-hosted models to external tools

Ollama runs on the Mac Mini that hosts 📝Brian Bot and its 57+ agent ecosystem on 📝OpenClaw. For anything I don't want leaving the local machine — experimental prompts, draft voice memos, unpublished memos passing through agent pipelines — the stack routes through Ollama instead of the Anthropic or OpenAI APIs. The value isn't cost savings; it's the discipline of a default-local architecture where cloud inference is a deliberate opt-in, not the path of least resistance. Ollama made that default possible without me having to build it myself.