LlamaStash 发布：单二进制本地 LLM 管理器，内置 OpenAI 兼容代理

Zero-overhead, terminal-native local-LLM manager.

A fast TUI and CLI with init wizard for managing local LLMs. One Rust binary that's a TUI, a CLI, a daemon, and an OpenAI-compatible proxy. llama.cpp is the direct, zero-overhead default backend (vs raw llama-server), plus Lemonade for NPU / multi-engine inference, vLLM and SGLang for safetensors, and a generic backend that runs any other OpenAI-compatible server you declare in config, like ds4 for DeepSeek-V4. See benchmarks.

Contents

• Why
• Install
• Quickstart
• Agent Skills
• Features
• Benchmarks
• Screenshots
• Configuration
• CLI exit codes
• Platforms
• Roadmap
• Contributing

Why

Heavy abstractions (Ollama, LM Studio) hide llama.cpp; raw llama-server use is tedious. LlamaStash is a fast, transparent manager that is also a first-class shell-tool surface for agents — one binary, daemon on demand, same primitives in the TUI and the CLI.

AI agents installing this for a user: jump to INSTALL.md § For AI agents. The non-interactive install + verify contract, and exit-code branching live there.

Install

Pick whichever channel you prefer — all install the same binary. Full per-platform notes, troubleshooting, and the agent-friendly non-interactive path live in INSTALL.md.

# macOS + Linux, one-shot
curl -fsSL https://llamastash.dev/install.sh | sh

# Windows 11 (PowerShell, no admin elevation)
irm https://llamastash.dev/install.ps1 | iex

# Homebrew (macOS + Linuxbrew)
brew install llamastash/llamastash/llamastash

# Arch Linux (AUR — pick one)
yay -S llamastash       # source build

# From crates.io (any platform with a Rust toolchain)
cargo install llamastash

# Windows via Scoop bucket
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash

Windows: 64-bit Windows 10 1809+ / Windows 11, PowerShell 5.1+, Windows Terminal recommended for the TUI; the bundled llama-server needs the VC++ 2015–2022 Redistributable (x64). See INSTALL.md.

Then run llamastash init — the interactive wizard installs llama-server for your hardware, downloads a starter GGUF, writes a tuned config, and smoke-launches it.

Quickstart

# Open the TUI. Scans default caches; daemon auto-spawns on demand.
llamastash

# List discovered models. TTY → padded + table; piped or
# `--no-colors` → TSV bytes. `--json` is the agent contract.
llamastash list
llamastash list --json | jq

# Launch a model by name, name substring, path, or canonical id.
llamastash start qwen-coder --ctx 16384 --reasoning on

# `run` is an alias for `start`, and the positional takes a YAML
# launch file: one model, its presets, nothing saved to config.yaml.
llamastash run qwen-coder.yml

# Drive a smoke-test request against the running endpoint.
curl -s http://127.0.0.1:41100/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen-coder", "messages": [{"role": "user", "content": "hi"}]}'

# Stop it.
llamastash stop qwen-coder

Tip — mouse focus. Mouse capture is off by default so the terminal keeps native click-and-drag text selection. To opt in on every TUI run, alias the binary in your shell rc:

# bash / zsh
alias llamastash='llamastash --mouse-focus'

# fish
alias llamastash 'llamastash --mouse-focus'

Or set it permanently in config.yaml:

mouse_focus: true

Either source flips on click-to-focus for the Models list, the right pane, and the tab labels (Settings/Logs/Chat/Embed/Rerank). Most terminals still expose a bypass modifier (Shift on iTerm2 / Alacritty / foot / wezterm, Option on Apple Terminal) so ad-hoc selection stays reachable.

Tip — shell completions. llamastash completions <shell> prints a completion script for bash, zsh, fish, PowerShell or Elvish. Subcommands, flags and fixed value lists complete; model names do not. Install it once:

# bash
mkdir -p ~/.local/share/bash-completion/completions
llamastash completions bash > ~/.local/share/bash-completion/completions/llamastash

# zsh — put `fpath+=(~/.zsh/completions)` above `compinit` in ~/.zshrc
mkdir -p ~/.zsh/completions
llamastash completions zsh > ~/.zsh/completions/_llamastash

# fish
llamastash completions fish > ~/.config/fish/completions/llamastash.fish

Full subcommand reference: docs/usage.md. Proxy client setup (including an OpenCode example): docs/usage.md#opencode-setup. Prefer a Vulkan llama-server build on AMD/NVIDIA hosts: docs/usage.md#preferring-a-vulkan-llama-server-build. Architecture and IPC contract: docs/architecture.md. When things go wrong: docs/troubleshooting.md.

Agent Skills

The CLI ships with an Agent Skills manifest so supported agents can load repo-specific instructions for using llamastash as a local model-management CLI.

• Canonical skill bundle: skills/llamastash/

Any supported agent (recommended): install the bundled skill with the Agent Skills CLI. It auto-detects your harness and writes the bundle to the right path:

npx skills add llamastash/llamastash

Claude Code plugin marketplace: install the repo as a plugin, then install the bundled skill:

/plugin marketplace add llamastash/llamastash
/plugin install llamastash@llamastash
/reload-plugins

Manual install examples:

# OpenClaw
mkdir -p ~/.openclaw/skills && cp -r skills/llamastash ~/.openclaw/skills/

# OpenCode
mkdir -p ~/.config/opencode/skills && cp -r skills/llamastash ~/.config/opencode/skills/

The skill teaches agents to prefer --json, branch on LlamaStash's documented exit codes, reuse exact discovered model names, and read status --json proxy.listen before configuring an OpenAI-compatible client.

Features

Full detail per feature in FEATURES.md — including trade-offs, contracts, and links into docs/usage.md.

Zero-to-chat in one command

• llamastash init — first-run wizard that detects hardware, installs llama-server, picks + downloads a starter GGUF, writes a tuned config, and smoke-launches.
• Hardware-aware model recommender with a VRAM-fit filter + composite ranking over a CI-refreshed benchmark snapshot.
• llamastash doctor — typed, agent-branchable findings; always exits 0.

Discovers what you already have

• Auto-scans HuggingFace, Ollama, and LM Studio caches, plus user paths.
• Rich GGUF intelligence — architecture, params, quant, native context, chat template, KV-cache-aware memory estimates.
• Multimodal projector detection — auto-pairs an mmproj GGUF beside a model, loads it with --mmproj, flags vision (◉) / audio (♪) in the TUI.
• MTP speculative decoding — auto-detects MTP-capable GGUFs (embedded or separate mtp-*.gguf head) and enables --spec-type draft-mtp by default for ~2x decode with no quality loss; --mtp auto\|on\|off, a ↯ TUI badge, live draft-acceptance in status. pull fetches the mmproj / MTP-head companions too.
• Smart deduplication — symlinks collapsed, split GGUFs unified, Ollama blobs named.
• Live filesystem watching — new GGUFs appear without a restart.

Launches anything, supervises everything

• Daemon-on-demand — one binary as TUI + CLI + daemon; running models survive TUI close.
• Multi-model concurrency — per-model port from a configurable range, /health-probed state machine.
• GPU-aware built-in arch defaults — sensible flags per (architecture, gpu_backend) with zero YAML.
• Intelligent context auto-fit — when ctx is unset, llamastash picks the largest context that fits free VRAM (or RAM, CPU-only) from the GGUF attention geometry. Sidesteps llama.cpp --fit's 4096 collapse on Linux 7+ iGPUs (AMD Strix Halo) where unified-memory free space is mis-reported.
• Typed launch-knob editor with (source) chips and a layered preset → last-params → arch-defaults → built-ins resolver.
• Multi-GPU device selection — pin a model to one card (--device) instead of letting llama.cpp split it across every GPU; the picker lists exactly what llama-server --list-devices reports.
• Named presets, favorites, last-params recall — presets live in config.yaml (per-model or per-arch), edited from the CLI, the TUI Ctrl+P save dialog, or by hand; a preset cycle row in the Settings form picks one in a keystroke.

A TUI that doesn't get in your way

• Keyboard-driven everywhere — vim-style hjkl + Ctrl+F/Ctrl+B paging, 0/$ top/bottom, gt/gT tab cycling; / filter, u/c/p yank, ? help.
• Right pane is your smoke test — Logs / Chat / Embed / Rerank over the same OpenAI-compatible endpoints external clients use.
• In-TUI HuggingFace browser — search, sort, paginate, per-file hardware fit, download strip with cancel.
• Theming and rebinding — five themes + custom palette; every action rebindable.
• Accessible by default — dual-encoded status (color + glyph), readable on mono terminals.
• Adaptive layout — works from 60 cells up — below 100 cells the right pane goes drill-in-only; list columns and hint chips drop by priority rank as the pane shrinks so the model name stays readable.

First-class CLI for agents and scripts

• Subcommands cover every TUI capability with --json as the stable agent contract.
• Documented exit codes per failure class — pin numbers, not message text.
• Colored TTY output, byte-stable TSV when piped — existing awk / column pipelines keep working.
• llamastash pull <hf-repo> — same primitive as the wizard, with disk-space prechecks.
• llamastash recommend — the recommender on its own, agent-friendly.
• llamastash integrations — point OpenCode, Aider, Continue.dev, Zed, pi.dev or Codex CLI at the proxy, with your favorites registered and effort pickers wired for reasoning models.
• Reproducible pulls via --revision <SHA>.

Drop-in OpenAI + Ollama proxy

• OpenAI-compatible endpoint at http://127.0.0.1:11435/v1 by default (or the next free port up to 11440) — drives every discovered model through one URL; OpenCode, Pi (pi.dev), Cline, llm-cli, the OpenAI SDKs all work as-is. Auto-starts the requested model; falls back to a Ready peer with audit headers (x-llamastash-served-by, x-llamastash-fallback-reason) when launch fails. The default port is 11435 (one above Ollama's well-known 11434) so a llamastash daemon and an Ollama install can co-exist without a port collision.
• Anthropic Messages API on the same proxy — /v1/messages + /v1/messages/count_tokens forward to llama-server's native endpoints, so Claude Code and the Anthropic SDK attach via ANTHROPIC_BASE_URL (key sent as x-api-key). llamastash init's Claude Code integration drops a sourceable claude-code.sh so you opt in per-shell (source claude-code.sh && claude) without hijacking its global config. Tool calling works out of the box (--jinja on by default).
• Browser web UI at http://127.0.0.1:11435/ui/ — opens the running model's stock llama.cpp web UI through the proxy on one port-stable origin, so you never chase the ephemeral backend port. A chooser (plus /ui/switch) handles several running models; reachable over LAN behind the same key via a browser Basic-auth prompt.
• Ollama discovery surface — GET /api/tags / /api/version / /api/ps, POST /api/show so tools that auto-detect Ollama via OLLAMA_HOST recognise llamastash and fall through to the OpenAI-compat endpoints for inference.
• Ollama drop-in mode is opt-in. Enable with --ollama-compat (or proxy.ollama_compat: true / LLAMASTASH_OLLAMA_COMPAT=1) and the proxy claims port 11434, answers GET / with the byte-exact "Ollama is running" handshake string, and works as a transparent replacement for the official ollama CLI and other Ollama-Go-based clients. Leaving compat off keeps the safe coexistence default (port 11435, "LlamaStash is running" identity).
• Loopback by default, opt-in LAN with auth — the proxy binds 127.0.0.1 and runs keyless for the same-machine threat model. Expose it on the LAN with --proxy-host 0.0.0.0 (or proxy.host) and llamastash auto-generates a bearer key, requires it on every request, and refuses to bind a routable address with no key unless you pass --insecure-no-auth. TLS is on the roadmap; LAN mode is plaintext for now (trusted network or front with a reverse proxy). The control plane and llama-server children always stay loopback.

NPU & multi-engine via Lemonade (experimental)

• ⚠️ Experimental — new and lightly road-tested; behaviour and config may change. llama.cpp stays the stable default.
• A pluggable backend seam. llama.cpp is the direct, zero-overhead default; Lemonade (lemond) plugs in as a second backend for engines llama.cpp can't reach — NPU inference on AMD Ryzen AI / XDNA, plus ROCm / ONNX / others. Default-on when the lemond binary resolves (like vLLM); force via --lemonade / LLAMASTASH_LEMONADE=1, or set backend.lemonade.enabled: false to opt out. Zero footprint when the binary is absent.
• You install Lemonade; LlamaStash drives it. No auto-install — LlamaStash finds lemond (PATH or backend.lemonade.servers), supervises the shared umbrella, discovers its models, routes inference through the proxy, and evicts idle models by API unload. See Lemonade setup.

vLLM — safetensors HuggingFace repos (experimental)

• ⚠️ Experimental — validated against vLLM 0.27.1 on a single Strix Halo / ROCm box; behaviour and config may change.
• The non-GGUF half of your cache. Safetensors repos sitting in ~/.cache/huggingface were invisible to LlamaStash before; now they appear in the catalog and launch through vllm serve. No competition with llama.cpp — a GGUF still binds llama.cpp, and vLLM claims safetensors repos only.
• You install vLLM; LlamaStash drives it. Default-on when a vllm launcher resolves (PATH or backend.vllm.servers); force with --vllm / LLAMASTASH_VLLM=1, opt out with backend.vllm.enabled: false. Zero footprint when absent. On ROCm, where vLLM ships only as a container, point the config at a small wrapper script — the recipe is in vLLM setup.
• Nine native knobs (--kv-cache-memory-bytes, --gpu-memory-utilization, --tensor-parallel-size, --dtype, --kv-cache-dtype, --quantization, --max-num-seqs, --enforce-eager, --trust-remote-code) in the launch picker and presets; --ctx maps to --max-model-len.

SGLang — safetensors HuggingFace repos (experimental)

• ⚠️ Experimental — validated against SGLang 0.5.18 on a single DGX Spark (GB10, unified memory); behaviour and config may change.
• The same rows vLLM serves. Safetensors repos launch through sglang serve; a GGUF still binds llama.cpp. With both engines installed a repo lists both and auto picks vLLM — --backend sglang selects SGLang.
• You install SGLang; LlamaStash drives it. Default-on when a sglang launcher resolves (PATH or backend.sglang.servers); force with --sglang / LLAMASTASH_SGLANG=1, opt out with backend.sglang.enabled: false. Zero footprint when absent.
• A token cap, not a byte cap, on unified-memory hosts. SGLang's only deterministic bound on its KV pool is --max-total-tokens, so the launcher divides the shared byte budget by the model's KV bytes per token, read from config.json. Eight native knobs in the launch picker and presets; --ctx maps to --context-length.

Generic — any OpenAI-compatible server

• Engines LlamaStash has no backend for. Declare a binary (or a wrapper script around docker run), its args with {port} / {model} placeholders, its readiness path and its own knobs in config.yaml. LlamaStash reserves the port, waits for ready, routes the proxy and stops it with a per-entry grace floor. Validated with gufo, Halogen (Docker) and CIRU on a Strix Halo box.
• Run a GGUF through another engine. An entry with model: "<glob>" becomes a server option on the matching catalog rows, next to llama.cpp, in the TUI Server row, --server, and presets. Its knobs show up in the launch editor, presets and last-used like any other.
• ds4 for DeepSeek-V4. antirez's ds4 (ds4-server) runs as a generic entry with its MTP, DSpark, SSD-streaming and disk KV cache flags as knobs. Same argv and speed as the dedicated backend it replaces. See Running ds4 as a generic server.

Built to be safe to run

• Bearer-token loopback control plane (runtime.json 0600) — the per-daemon token + URL live in $XDG_STATE_HOME/llamastash/runtime.json; same-UID trust, no network exposure.
• Hardened fetch substrate — HTTPS-only, host allowlist, redirect/body-size caps, IP-literal refusal.
• Archive-bomb defenses on installers — entry/size/ratio caps; SHA-256 verified before extract.
• Atomic, mode-checked config + state writes — 0600 final mode; corrupt state quarantined, not fatal.
• Side-by-side daemons — isolated instances via LLAMASTASH_*_DIR (state / config / cache); each daemon publishes its own runtime.json.

Note: This is beta software. Rough edges are to be expected. Windows and macOS support is not as well-tested as Linux; Same goes for non-AMD GPUs. Please report issues if you hit them. The llama-server builds are unmodified upstream binaries; any bugs in them are out of scope for LlamaStash.

Benchmarks

LlamaStash spawns the unmodified upstream llama-server. Three suites track what that means in practice — Suite A asserts the wrapper adds no measurable overhead vs raw llama-server, Suite B compares LlamaStash-as-shipped against Ollama + LM Studio on the same hardware through their OpenAI-compatible endpoints, Suite C measures the proxy hop vs hitting llama-server directly (TTFT p50 +0.45 ms, decode unchanged). Full write-up + per-workload tables: docs/benchmarks.md.

Each cell below is decode tok/s / TTFT ms on the chat_turn workload (50-token prompt → 64 tokens decode). LlamaStash matches raw llama-server within ≤1% in normalized mode on every platform. Re-run on your hardware: make bench-end-to-end (Suite B) or make bench-overhead (Suite A).

AMD APU - Linux (Ryzen AI Max+ 395 / Radeon 8060S 128GB unified, system ROCm 7.2.3, llama.cpp build b9440 (e6123e208))

chat_turn normalized mode, decode tok/s / TTFT ms. One bench JSON per row (no averaging).

¹ LM Studio's bundled ROCm vendor libraries (v6.4) abort in ggml_cuda_error during backend init on gfx1151 (Strix Halo) across all LMS-shipped runtimes. System ROCm 7.2.3 loads the same models cleanly via raw llama-server, so this is an LMS vendor-bundle limitation. LMS Vulkan numbers are in the benchmark blog and in the final report.

AMD APU — Vulkan addendum (LlamaStash vs LM Studio, 2026-06-01)

Same backend (Vulkan b9440 / vulkan-avx2@2.18.0), same GGUF bytes. raw llama-server and Ollama omitted: the wrapper-overhead claim already covered by the HIP table; Ollama mainline has no Vulkan support.

NVIDIA - Linux (RTX 3050 Ti, 4 GiB VRAM, llama.cpp build b9360)

✦ LlamaStash leads raw llama-server in defaults mode (12–16% decode, 33–49% TTFT) via the hardware-aware defaults overlay; normalized mode: within ≤0.5 tok/s. Vulkan decode ≥ CUDA on this hardware in 26 of 28 cells (median +5%). Curated report with six findings: linux-nvidia-final.md.

Apple M1 - macOS (16 GB unified memory, Metal, llama.cpp build 9330 (328874d05))

✦ LlamaStash leads raw llama-server on M1 in defaults mode (99.0 vs 92.3 tok/s, 15 vs 19 ms TTFT) because its Metal defaults overlay injects hardware-optimal knobs at startup. Normalized mode: 92.2 vs 91.5 — within 1%. Curated report: macos-m1-final-report.md.

Screenshots

Configuration

LlamaStash reads $XDG_CONFIG_HOME/llamastash/config.yaml on Linux (fallback ~/.config/llamastash/config.yaml), ~/Library/Application Support/llamastash/config.yaml on macOS, and %APPDATA%\llamastash\config\config.yaml on Windows. A fully-annotated sample lives at config.example.yaml — copy it to the path above and edit. Run llamastash config to open the active file in $EDITOR, or llamastash config bindings to print every effective keybinding as YAML. The full schema reference is in docs/usage.md.

Quick tour of the top-level keys:

Environment variables:

Default scan paths

When model_paths and --model-path are empty, LlamaStash walks these caches automatically. Each bucket is independently toggleable via disable_default_cache_paths.<bucket>: true in config.yaml, or globally via --no-scan / LLAMASTASH_NO_SCAN=1.

Files anywhere under these roots that end in .gguf (and aren't .gguf.part) get parsed and added to the catalog.

CLI exit codes

Every non-interactive subcommand returns a documented exit code so agent scripts can branch on failure class. Pin against numbers, not message text — they are the public contract.

Note on sysexits.h: the numbers above are deliberately reused from <sysexits.h> for familiarity, but LlamaStash's meanings diverge from the standard ones. Scripts that import EX_NOHOST (68) expecting "host unreachable" will get our "stop failed"; EX_DATAERR (65) is reused for "daemon unreachable", not "data error". Branch on LlamaStash's table above, not the libc constants.

Platforms

Linux (x86_64, aarch64), macOS (Apple Silicon, Intel), and Windows 11 (x86_64). One binary, one TUI, one CLI — the daemon's control plane is bearer-token-authed HTTP loopback on every platform, and the supervisor uses the OS's native process-group semantics (POSIX setsid + signals, Windows Job Objects + CTRL+BREAK). Windows AMD GPU detection and aarch64-pc-windows-msvc are on the roadmap.

Supported llama-server version

LlamaStash hands GPU/CPU placement and context sizing to llama.cpp's --fit (on by default), so it needs a llama-server that has it.

• Minimum: build b7410 (2025-12-15), the first release carrying --fit / --fit-ctx (llama.cpp PR #16653). Older builds abort on the unknown argument the moment a model launches.
• Recommended: a recent build (b8500+, 2026). The May 2026 AMD GPU-stack updates (kernel, amdgpu firmware, ROCm) materially improved --fit on unified memory; verified on b9245.

llama.cpp has no semantic versioning, no stable branch, and no stability policy (discussion #16111) — it tags ~10-14 rolling builds a day. So the build number is a floor for flag existence, not a behaviour guarantee; LlamaStash's own pre-spawn admission control is what actually prevents out-of-memory launches regardless of build. llamastash init installs a known-good build for your hardware.

Roadmap

Tracked in detail in TODO.md. The headline items on deck:

• MLX backend — if the surface area lands cheaply alongside llama.cpp.
• llama.cpp version pinning — prevent silent drift / incompatibility on brew upgrade.
• MCP and LAN-exposed HTTP surfaces — Model Context Protocol, plus auth + TLS + LAN binding for the proxy. The loopback OpenAI-compatible proxy ships today (see Drop-in OpenAI + Ollama proxy); the rest of the v1 R34 deferral (Anthropic compat, MCP, network exposure) stays on the roadmap.
• Per-PID VRAM attribution via NVML's nvmlDeviceGetComputeRunningProcesses. Today the right pane shows per-model RAM + CPU%; VRAM is reported only at the host level.
• Docker-ready packaging — official images plus a documented docker run path.
• Windows AMD GPU detection — pick a probe path (DXGI / WMI / ADLX). 0.0.2 shows "GPU detection unavailable" on Windows AMD hosts.
• aarch64-pc-windows-msvc — Snapdragon X / Surface Pro coverage. Deferred from 0.0.2.

Contributing

Bug reports, design discussion, and PRs welcome. Start with CONTRIBUTING.md.

AI Usage

Multiple AI Coding Harnesses and LLMs were heavily used to create this project.

License

MIT © Deepu K Sasidharan

Terms of use

• The Software shall be used for Good, not Evil.
• This software shall not be used for any military purposes including intelligence agencies.

Related projects

• kdash — Kubernetes dashboard TUI by the same author. LlamaStash borrows its engineering and release scaffolding from kdash: the org layout (llamastash/llamastash, llamastash/homebrew-llamastash, llamastash/llamastash.github.io), the brew-tap structure, the cli.rs subdomain setup, and the release-on-tag workflow shape. The product itself is independent.
• jwt-ui — JWT decoder / encoder TUI by the same author.

Star History

If LlamaStash is useful to you, a star helps other people find it.