GGLib 发布：把 llama.cpp 本地模型库变成 OpenAI 兼容 API 提供方

The local model runtime that makes llama.cpp behave like an API provider.

Running a local model with an agentic client means remembering where your GGUFs live, figuring out llama-server flags for each model, and managing sampling parameters. Even then, the model still produces broken tool calls, gets stuck in loops, and ignores your sampling config.

GGLib handles the whole stack. It reads each GGUF's metadata and chat template to figure out the right launch flags, tool-call format, and sampling defaults, with no per-model config. It launches llama-server and runs an intelligent proxy between your client and the model. To your client, your entire local GGUF library looks like an OpenAI-compatible provider. Clients query /v1/models, pick a model, and everything just works.

Get running

Pre-built binary

Download a binary from the Releases page (macOS, Linux, Windows) and run it:

./gglib up

apt (Ubuntu 24.04+, Debian 13+)

curl -fsSL https://mmogr.github.io/gglib/apt/gglib-archive-keyring.gpg \
  | sudo tee /usr/share/keyrings/gglib-archive-keyring.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/gglib-archive-keyring.gpg] https://mmogr.github.io/gglib/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/gglib.list
sudo apt update && sudo apt install gglib   # the desktop app is gg-lib-gui

The repository carries stable releases only. Release candidates are on the Releases page as prereleases.

Build from source

Requires Rust, Node.js 20.19+, and a C++ toolchain with CMake. gglib config check-deps prints your platform's exact install commands.

git clone https://github.com/mmogr/gglib.git && cd gglib
make setup   # checks deps, builds everything, installs the gglib CLI
gglib up

make setup also offers two optional extras, and goes on without either: faster downloads, and the image runtime that draws pictures with an image model.

A build from source keeps its library in the same per-user folder as an installed gglib, so the two share models, settings and llama.cpp; CONTRIBUTING says how to keep a development build apart.

gglib up reads your hardware, installs llama.cpp if missing, recommends a model that fits your VRAM, starts the proxy, sends a test request to prove it works, and prints the settings to paste into your client. You have a working endpoint at http://127.0.0.1:8080/v1.

What it fixes

Everything between the OpenAI request and llama-server is the product:

• Tool-call repair: tool_choice: "auto" can leave llama.cpp unconstrained, producing malformed arguments that crash the client's executor. GGLib validates each tool call in the answer's first choice against the client's schema (no other choice is judged) and silently re-issues failures: with tool_choice: "required", which activates llama.cpp's own grammar, or, on a turn gglib's own grammar constrained, as a second draw under it. The client gets the re-issue's answer when that validates, and the original when it does not. Measured through a real proxy on 2026-09-20: on Llama 3.2 3B over a schema-stress suite, 11 of 11 re-issues produced a conformant call, and the proxy arm as a whole — repair among several things it does — scored higher on 8 of 15 matched pairs and lower on none; on Qwen3.8-27B nothing broke a schema, so repair attempted nothing. Details →, reading →
• Loop defense: agentic clients replay the full conversation each turn, so the proxy scans the incoming history for tool-call batches repeated back to back and answered the same way, observation-tool spam, and repeated response text anywhere in the session. A batch whose answer keeps changing is an agent polling for output, not a loop, and does not trip it. What happens to a stuck session is one setting: by default the request is forwarded with a note telling the model what it has repeated, so a client with no recovery path from a refusal still gets an answer; --loop-guard-mode refuse answers 400 instead, before a model swap or another generation, and off stops the scan. No run has measured how often it fires or what the note changes yet (#1047). Details →
• Sampling authority: a 5-level hierarchy (request → profile → per-model → global → floor) resolves every sampling parameter server-side. Client values are untrusted by default, so no request can silently override your configuration. Details →
• Context defense: tool-heavy sessions balloon past the context window because most clients don't compact for custom endpoints. The proxy truncates oversized tool and assistant messages before forwarding, scaled to the model's actual context size, not a fixed floor. Details →
• Dialect normalization: model-specific markup (Qwen XML tool calls, bare <think> tags) is parsed and re-encoded into strict OpenAI-format events. Details →
• KV cache tiering: three layers, all auto-configured. q8_0 KV quantization halves VRAM usage versus llama-server's default, a host-RAM prompt cache is auto-sized from your system memory on every launch, and opt-in disk slot persistence lets KV state survive model swaps and restarts. Sliding-window and hybrid-attention models are detected and excluded from disk caching automatically. Details →
• Capability detection: at import, GGLib reads GGUF metadata and renders the chat template against probe conversations to derive reasoning support, tool-call dialect, and speculative-decoding config. No per-model setup. Details →
• Admission control: alternating models (chat + embeddings) batch swaps instead of thrashing. Details →
• Images: a screenshot pasted into Copilot, or attached with gglib q --image or in the chat page, goes to a model linked to its projector, which a download from a repository that has one fetches and links. An image is counted against the context by its pixels, not by its base64, and one sent to a model with no projector is refused by name before anything is loaded, where llama-server alone answers HTTP 500 once the model is up. Details →

Client configuration

The endpoint is http://127.0.0.1:8080/v1. No API key is needed on loopback; use any placeholder if your client requires one. Model names come from gglib model list (shown as qwen3.6 below).

Append :coding to a model name (e.g. qwen3.6:coding) to select a sampling profile. See Sampling → Inference profiles.

Day-to-day usage

# Download a model directly from a HuggingFace repo
gglib model download bartowski/Qwen3-4B-GGUF

# Search HuggingFace for models
gglib model search "gemma 3 instruct"

# List your installed models
gglib model list

# Start the proxy (all models available via /v1/models)
gglib proxy

# Ask a one-shot question
gglib q "Explain the builder pattern in Rust"

# Pipe context in
cat error.log | gglib q "What went wrong here?"

# Ask about a screenshot (the model must be linked to a projector)
gglib q --image shot.png "what is the error?"

# Link a model to its projector (an mmproj GGUF), which gives it image input
gglib model update <model> --projector <file>

# Interactive chat
gglib chat

Configure

Optional, after first run:

gglib config settings set --default-context-size 131072
gglib config settings set --proxy-autostart true --close-to-tray true

Benchmark and tune

# Compare the same prompt across multiple models side-by-side
gglib benchmark compare -p "Explain ownership in Rust" -m Qwen3-4B -m Gemma3-4B

# Measure raw throughput (prompt processing + generation) via llama-bench
gglib benchmark perf -m Qwen3-4B

# Sweep sampling parameters to find the best settings for a model,
# scored on tool-call accuracy, loop avoidance, and task completion
gglib benchmark tune -m Qwen3-4B --sweep temperature=0.2,0.5,0.8 --apply

# A/B test: run the agentic suite against raw llama-server vs. through
# GGLib's request pipeline, on the same model and seeds; --proxy adds an
# arm through a real GGLib proxy, tool-call repair on, and a raw baseline
gglib benchmark agentic -m Qwen3-4B --seeds 12345,67890 --proxy

The agentic benchmark includes its own A/A arm and positive control so it can't overclaim. Full methodology: ADR 0004's log.

Interfaces

All interfaces share the same database and model directory.

gglib proxy dashboard streams live proxy state to your terminal:

Active connections (1)
  slot 0  Qwen3.5-4B  task 426  generating  7.36 t/s

Slots (llama.cpp /slots)
  slot 0  [████████████░░░░░░░░]  58%

VRAM residency
  Qwen3.5-4B (Q8_0)  6.4 GiB loaded
  Model swaps        0

Prompt cache
  Reused       2,278 of 8,641 prompt tokens (3 requests)
  Last request 2,278 of 5,979 tokens from cache

Client sampling dropped (trust_client_sampling off)
  temperature: 3 requests

The desktop GUI doubles as a quick way to test models and as an always-on proxy host. --proxy-autostart, --close-to-tray, and --start-at-login keep the endpoint available without a terminal window.

Remote access

Your desktop has the GPU; your laptop has the question. gglib remote puts the desktop's proxy on the laptop's loopback over a direct, end-to-end encrypted connection between the two machines — no VPN, no account, and no relay that can read a request.

# desktop
gglib remote enable --invite             # shows a ticket + six-digit code, once

# laptop, within two minutes
gglib remote join <ticket>-<code>        # binds a local port that is the desktop
gglib q --remote -m <model> "Why is this test flaky?"   # a model the desktop serves

The code is single-use and dies in two minutes; the ticket is fresh every enable; the tunnel enforces the same API key the proxy does, and /mcp is closed over it unless you say otherwise. On the laptop, the chat page's rail switches between its own chats and the desktop's, which it reads and carries on live without copying them. Details →

Security

Everything binds 127.0.0.1 by default. Do not expose the endpoint to the public internet. Optional bearer API key on loopback; auto-minted if you bind externally, and put on the loopback proxy — the OpenAI-compatible one on :8080, not the daemon's management API on :9887 — the moment you enable remote access. The daemon's API asks its own token, minted at every start and readable only by your account; gglib web opens your browser on a link that carries it (--no-open only prints the link). Host-header allowlist and local-only CORS are always on. No multi-tenancy or rate limiting. Details in gglib-proxy; the tunnel's model is in Remote access.

Architecture

An 18-crate Rust workspace (17 under crates/, plus the desktop app in src-tauri) and a React front end. Dependencies point one way: adapters → facades → infrastructure → gglib-agent → core, and scripts/check_boundaries.sh checks part of that in CI. See crates/README.md for the layer diagram and the catalog of crates, each with its own README, CONTRIBUTING.md for conventions, and generated API docs.

Documentation

• CLI command reference
• Client configuration and images
• Sampling resolution
• Tags & capability detection
• KV cache tiering
• Tool-call repair
• Remote access
• Architecture Decision Records
• Full API docs

License

AGPL-3.0-only. Anyone may use, modify and redistribute gglib, commercially or not, under its terms: distributing it, or letting users reach a modified copy over a network, means offering them the corresponding source under AGPL-3.0. If you cannot meet those terms, @mmogr may offer a separate commercial license; ask.