Ollama & Local Models

InitRunner supports running agents against local LLMs served by Ollama or any OpenAI-compatible endpoint (vLLM, LiteLLM, llama.cpp server, etc.). This requires zero additional dependencies — it reuses the openai SDK already bundled with the core install.

Quick Start

1. Install and start Ollama:

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh
ollama serve

2. Pull a model:

ollama pull llama3.2

3. Scaffold a role:

initrunner new --template ollama --model llama3.2

4. Run the agent:

initrunner run agent.yaml -i

How It Works

Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1. When provider: ollama is set (or a base_url is specified), InitRunner constructs a PydanticAI OpenAIProvider with that endpoint instead of calling the real OpenAI API. A dummy API key ("ollama") is set automatically so the SDK doesn't look for OPENAI_API_KEY in the environment.

Configuration

Minimal Ollama Role

name: local-agent
description: Agent using local Ollama model
model:
  provider: ollama
  name: llama3.2          # Run: ollama pull llama3.2
prompt: |
  You are a helpful assistant.

Model Config Reference

model:
  provider: ollama               # required — triggers local model setup
  name: llama3.2                 # required — model name as known to Ollama
  base_url: http://localhost:11434/v1  # default for ollama; override for remote
  temperature: 0.1               # default: 0.1
  max_tokens: 4096               # default: 4096

Custom OpenAI-Compatible Endpoints

The base_url field works with any provider, not just Ollama. Use it to point at vLLM, LiteLLM, llama.cpp, or any other server that exposes an OpenAI-compatible API:

model:
  provider: openai
  name: my-model
  base_url: http://my-server:8000/v1

When base_url is set on a non-ollama provider, InitRunner injects no API key. PydanticAI's OpenAIProvider falls back to OPENAI_API_KEY from the environment, and when that is unset too it substitutes a placeholder key, so keyless local servers (vLLM, llama.cpp) work with no credential at all.

If your endpoint authenticates with a different variable, name it with api_key_env. Keep base_url set: dropping it would send requests to OpenAI instead of your server.

model:
  provider: openai
  name: my-model
  base_url: http://my-server:8000/v1
  api_key_env: MY_ENDPOINT_API_KEY

The key is resolved from the environment first, then the credential vault. If api_key_env names a variable with no value in either place, the run fails at startup with an actionable error rather than sending an unauthenticated request. See Providers for more custom-endpoint examples.

Embeddings

Ollama also serves embeddings. When using ingestion or memory with Ollama, configure the embedding model in the embeddings section:

model:
  provider: ollama
  name: llama3.2
ingest:
  sources:
    - "./docs/**/*.md"
  embeddings:
    provider: ollama
    model: nomic-embed-text        # Run: ollama pull nomic-embed-text
    # base_url: http://localhost:11434/v1  # default

Embedding Config Reference

Default Embedding Models

Example: Local RAG Agent

Full local RAG stack — no external API calls or API keys:

name: local-rag
description: Local RAG agent with Ollama
tags:
  - rag
  - ollama
model:
  provider: ollama
  name: llama3.2
prompt: |
  You are a knowledge assistant. Use search_documents to find relevant
  content before answering. Always cite your sources.
ingest:
  sources:
    - "./docs/**/*.md"
    - "./docs/**/*.txt"
  chunking:
    strategy: fixed
    chunk_size: 512
    chunk_overlap: 50
  embeddings:
    provider: ollama
    model: nomic-embed-text

ollama pull llama3.2
ollama pull nomic-embed-text
initrunner ingest role.yaml
initrunner run role.yaml -i

Example: Memory Agent

Long-term memory works fully offline with Ollama:

name: local-memory
description: Local agent with memory
model:
  provider: ollama
  name: llama3.2
prompt: |
  You are a helpful assistant with long-term memory.
  Use remember() to save important information.
  Use recall() to search your memories.
memory:
  max_sessions: 10
  semantic:
    max_memories: 1000
  embeddings:
    provider: ollama
    model: nomic-embed-text

Docker

When running InitRunner inside Docker, localhost won't reach the host machine. Use host.docker.internal instead:

model:
  provider: ollama
  name: llama3.2
  base_url: http://host.docker.internal:11434/v1

InitRunner automatically detects Docker environments (via /.dockerenv) and logs a warning if base_url contains localhost or 127.0.0.1.

Alternatively, run Ollama in the same Docker network:

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
  agent:
    build: .
    environment:
      - OLLAMA_HOST=http://ollama:11434/v1

model:
  provider: ollama
  name: llama3.2
  base_url: http://ollama:11434/v1

CLI

Scaffold an Ollama Role

initrunner new --template ollama --model mistral

This generates an agent.yaml pre-configured for the ollama provider with the specified model (or llama3.2 by default). Pass --output to write a different filename. The Ollama reachability check (a ping to http://localhost:11434/api/tags, with a warning when Ollama is not running) belongs to the interactive builder, not to the --template path.

Available Templates

Any template works with --provider ollama:

initrunner new --template basic --provider ollama --model codellama
initrunner new --template rag --provider ollama --model llama3.2
initrunner new --template memory --provider ollama
initrunner new --template daemon --provider ollama
initrunner new --template ollama  # dedicated template with Ollama-specific comments

Troubleshooting

"Ollama does not appear to be running"

Start the Ollama server:

ollama serve

On macOS, you can also launch the Ollama desktop app.

Connection refused at runtime

Verify Ollama is running and accessible:

curl http://localhost:11434/api/tags

If using a remote Ollama instance, set base_url explicitly:

model:
  provider: ollama
  name: llama3.2
  base_url: http://remote-host:11434/v1

Model not found

Pull the model before running:

ollama pull llama3.2

List available models:

ollama list

Slow responses

Local models are limited by your hardware. Tips:

• Use smaller models (llama3.2 3B is faster than llama3.1 70B)
• Increase timeout_seconds in guardrails for larger models
• Use GPU acceleration (Ollama auto-detects CUDA/Metal)

EmbeddingModelChangedError on ingestion

You switched embedding models. The CLI will prompt you to confirm wiping the store and re-ingesting. To skip the prompt, use --force:

initrunner ingest role.yaml --force

Popular Ollama Models