Hermes Agent is model-agnostic by design — the docs list local Ollama servers alongside OpenRouter and Nous Portal as first-class inference backends. Pair the two and you get an AI agent where nothing leaves your server: no per-token bills, no third-party seeing your prompts, no provider deciding to change the rules under you.

Before you get excited: this is the honest version of that guide, including the part where we tell you a €5 VPS cannot do it. Local inference is a hardware question first and a software question second.

Why run the model locally at all?

Three reasons hold up in practice:

Privacy that is actually total. With a cloud model, your agent's memory stays on your VPS but every prompt still travels to the provider. Local inference closes that last gap.
Flat costs. An always-on agent that thinks a lot can burn real money in API tokens. A local model costs the same whether it processes ten requests or ten thousand.
No dependency on anyone's terms. The vendor-lock-in lesson from April 2026 applies to inference too: providers change pricing and policies overnight. ollama pull does not.

The trade-off is capability per euro. Small local models are far behind the frontier models your agent could reach through an API — for complex multi-step tasks, the difference is very noticeable.

What hardware does local inference actually need?

The blunt table, for quantized (Q4) models on CPU-only VPS hardware:

Model class	RAM needed	Experience on CPU
3–4B (e.g. small Llama/Qwen variants)	6–8 GB	Usable for chat, noticeably slow on long replies
7–8B	10–12 GB	Workable but patient — fine for async messaging, frustrating for interactive use
13B+	16 GB+	Only worth it with a GPU

Two Hermes-specific caveats that most Ollama guides skip:

Hermes expects a large context window — community guidance puts the practical minimum around 64k tokens for the agent loop to work well. Ollama models default to much smaller contexts, so you must raise it (below), and context memory comes on top of model memory. Budget several extra GB of RAM for a 64k window.
The agent itself also runs on the same box — add the gateway's 400–950 MB on top of whatever the model takes (sizing guide).

Realistic minimum for a pleasant local setup: 8 GB RAM for a small model, 16 GB for 7–8B with the context Hermes wants. If that is not in budget, skip to the hybrid setup at the end — it is what we actually recommend.

Step 1: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Ollama installs as a systemd service listening on 127.0.0.1:11434 — localhost-only by default, which is exactly what we want since Hermes runs on the same machine.

Step 2: Pull a model

Start small and see how your hardware copes before committing to bigger weights:

ollama pull qwen2.5:7b

Test it directly first:

ollama run qwen2.5:7b "Say hello in one sentence."

If the reply takes longer than your patience, drop to a 3B model before blaming Hermes.

Step 3: Raise the context window

Create a Hermes-friendly variant of the model with a larger context. Write a Modelfile:

printf 'FROM qwen2.5:7b\nPARAMETER num_ctx 65536\n' > Modelfile

ollama create qwen2.5-hermes -f Modelfile

Watch your RAM when you first use it — long contexts are where memory quietly explodes. If the OOM killer visits, halve num_ctx or move up a VPS tier.

Step 4: Point Hermes Agent at Ollama

Run the model picker:

hermes model

Choose the custom OpenAI-compatible endpoint option and give it Ollama's local API:

Endpoint: http://127.0.0.1:11434/v1
Model name: qwen2.5-hermes
API key: anything non-empty (Ollama ignores it)

From here, everything in the main Hermes tutorial applies unchanged — gateway, channels, systemd, backups. The agent neither knows nor cares that its brain is local.

Model choice in Hermes is one command, so you do not have to pick a side:

Local model for routine, private, always-on work: message triage, reminders, notes, anything touching data you would rather keep at home.
API model via OpenRouter or Nous Portal for the hard stuff: multi-step reasoning, coding, long documents.

This keeps your token bill small and your private data local, without making a 7B model pretend to be a frontier model. Switch with hermes model whenever the task changes shape.

What does it cost?

For local inference you are buying RAM, not tokens. The 8 GB Power tier at Virtua.Cloud's Hermes Agent VPS handles a small model plus the agent; for 7–8B models with Hermes-sized context, look at our larger Linux VPS plans. Hourly billing means you can benchmark a bigger tier for an afternoon, measure real tokens-per-second on your workload, and only then commit. That beats any table in any guide, including this one.