Hermes llama.cpp support means you can now run AI models fully on your own computer from inside Hermes Agent, with no account, no API key and no internet connection once the model is downloaded. Hermes installs and maintains llama.cpp for you, picks the right engine for your hardware, and rates every model in its catalog against your GPU and RAM so you know what will actually run. In this guide I cover how the one-click setup works, how Hermes picks a model, the settings worth knowing and when local models make sense. Watch the video below, then try it.

Key takeaways

  • Hermes now installs and maintains llama.cpp with the right engine: CPU, CUDA, Metal, Vulkan or ROCm.
  • Start from Settings → Providers → Local Models or hermes model → Local models.
  • Models are colour-rated: green fits your GPU, amber uses system RAM, red is too big.
  • Want local models set up properly? Get your Backlink Gameplan, or try Polar Rank free for 3 days.

What is llama.cpp, and why does it matter for Hermes?

llama.cpp is the open-source engine that runs AI models on ordinary computers. It’s what a lot of local AI tools use under the hood. Until now, using it with Hermes meant installing it yourself, choosing the right build for your graphics card, downloading a model file and pointing Hermes at the server.

According to Hermes’ Local Models docs, Hermes now does all of that for you. It installs and maintains llama.cpp with the right engine for your machine, rates every catalog model against your GPU and memory, and starts and stops the model server along with Hermes.

The headline benefit is privacy and independence: the model runs on your machine, so there’s no account, no API key and no network needed after the download.

How to set up local models in Hermes

There are two ways in, and both work on Windows, macOS and Linux:

From there it’s three steps: Install runtime, Download a model, then Use it. Downloads can pause and resume, so a big model on a slow connection isn’t a problem.

Hermes picks the engine for you. Every backend runs the same llama.cpp build (b11370 at the time of writing). On Linux, NVIDIA cards get CUDA, while AMD and Intel cards get Vulkan. macOS uses Metal, and every platform has a CPU fallback. If you want to force a backend, the config has a local_runtime.backend setting that accepts auto, cuda, metal, vulkan, hip or cpu.

Want private local models running in your business? Inside the AI Profit Boardroom you get the Agent OS — Hermes, OpenClaw, OpenCode, Claude and Codex in one place — plus step-by-step Hermes training and four live coaching calls a week.

Want a plan for your business first? Get your Backlink Gameplan, or try Polar Rank free for 3 days.

Join the AI Profit Boardroom →Get Your Backlink Gameplan →Try Polar Rank Free for 3 Days →

How Hermes picks a model for your machine

This is the part I like most. Instead of guessing which model your computer can handle, Hermes rates every model in its catalog against your hardware and colour-codes it:

Colour What it means
Green Fits fully in your GPU memory — the fastest option
Amber Spills into system RAM — it runs, but slower
Red Too big for this machine

Hermes recommends the highest-quality model that fits fully in GPU memory at a predicted 20 tokens per second or more. It never goes below 4-bit quantisation, and every recommended model gets at least 64K of context, which grows as needed.

The catalog includes models such as Qwen3.8 27B, Qwen3.8 Flash Next, Qwen3.6 35B-A3B and DeepSeek V4 Flash. You’re not limited to the list: “Find more models” searches any GGUF model on Hugging Face, and “Add model file” loads a .gguf you already have on disk.

Want the right local model picked and tuned for your hardware? Get your Backlink Gameplan → or try Polar Rank free for 3 days →

Get Your Backlink Gameplan →Try Polar Rank Free for 3 Days →

Settings worth knowing

If you already run your own llama-server, Hermes detects it too. By default it checks port 8080; to check other ports, add them under local_runtime.detect_ports. A server you started yourself shows up in /model.

local_runtime:
  enabled: false
  backend: auto   # auto|cuda|metal|vulkan|hip|cpu
  detect_ports: [8081]

You can also point Hermes at a llama.cpp server directly with the llamacpp provider (aliases llama.cpp and llama-cpp):

providers:
  llamacpp:
    base_url: http://127.0.0.1:8081/v1
    model: my-model

LM Studio users get their own provider profile as well, and the Ollama model picker no longer flickers.

System requirements

Platform Supported engines
Windows CUDA, Vulkan (x64) or CPU
Linux CUDA (CUDA 13 driver), Vulkan or CPU
macOS Metal or CPU

You don’t need a top-end graphics card to try it. The colour ratings tell you honestly what your machine can run, and a smaller green model will usually feel better than a bigger amber one.

When local models make sense (and when they don’t)

Local models are a great fit when:

They’re a weaker fit for the hardest coding and reasoning work, where the biggest hosted models are still ahead. The smart setup is both: a local model for everyday and private work, and a hosted model for the heavy lifting. Hermes’ fallback option makes that easy.

If you want to compare specific local models I’ve tested, see my guides to the Flint local coding model, Spark X2 6.4B and Nemotron 3.5 Bolt in Hermes.

My verdict on Hermes llama.cpp

Running models locally used to be a weekend project. Hermes turns it into a few clicks: it installs the engine, tells you what fits, downloads the model and manages the server. If you’ve wanted a private, offline agent, Hermes llama.cpp support is the easiest way I’ve seen to get one.

About Julian Goldie

I’m Julian Goldie — founder of Goldie Agency and the AI Profit Boardroom. I test new AI agents almost every day for 432K+ YouTube subscribers and run Hermes, OpenClaw, OpenCode, Claude and Codex side by side in my own Agent OS. Get your Backlink Gameplan, try Polar Rank free for 3 days, or join the AI Profit Boardroom.

Want another angle? Read the AI Profit Boardroom guide to Hermes llama.cpp.

FAQ: hermes llama.cpp

Does Hermes Agent support llama.cpp?

Yes. Hermes installs and maintains llama.cpp for you from Settings, Providers, Local Models or hermes model, and you can also point it at your own llama-server with the llamacpp provider.

Do I need an API key to run local models in Hermes?

No. Local models run fully on your machine with no account, no API key and no network once the model is downloaded.

Which GPUs work with Hermes local models?

NVIDIA cards use CUDA, AMD and Intel cards use Vulkan on Linux, Macs use Metal, and every platform can fall back to CPU.

How does Hermes choose which local model to recommend?

It rates each model against your GPU and RAM and recommends the highest-quality one that fits fully in GPU memory at a predicted 20 tokens per second or more, never below 4-bit.

Can I use my own GGUF model in Hermes?

Yes. Use Find more models to search GGUF models on Hugging Face, or Add model file to load a .gguf file you already have.

Ready to put Hermes to work in your business? Join the AI Profit Boardroom for the Agent OS, setup training and daily updates.

Or get your Backlink Gameplan and we’ll map out your setup, or try Polar Rank free for 3 days.

Starting from zero? Get the free AI course and community.

Join the AI Profit Boardroom →Get Your Backlink Gameplan →Try Polar Rank Free for 3 Days →Get the FREE AI Course + Community →

Related reading

Last updated October 2026. This page is a living guide to hermes llama.cpp — the facts here move fast and it is updated as they do.

Leave a Reply

Your email address will not be published. Required fields are marked *