Framework 16 本地 LLM 推理方案：用 AMD RX 7700S 在 Fedora 上跑 ROCm

Fast, well-tuned local LLM inference on a Framework 16 laptop with the AMD Radeon RX 7700S (8 GB) discrete GPU — on Fedora, with no cloud and no NVIDIA.

If your machine matches the spec below, ./setup.sh gets you a running, GPU-accelerated, OpenAI-compatible endpoint at http://127.0.0.1:8080 in one command.

Why this exists

"Can a laptop with no real GPU run local AI?" turned out to be the wrong question for this hardware — the Framework 16 dGPU module is a real 8 GB RDNA3 card (gfx1102), in the same tier as a desktop RX 7600. The hard part isn't capability, it's getting ROCm to build and pin correctly on Fedora. This repo encodes the solution, measured on the actual hardware.

Target hardware

Component	Spec
Laptop	Framework 16
CPU	AMD Ryzen 9 7940HS (8C/16T, AVX-512)
dGPU	Radeon RX 7700S — Navi 33, gfx1102, 8 GB VRAM
iGPU	Radeon 780M (gfx1103) — present, not used for inference here
RAM	32–64 GB recommended
OS	Fedora 44 (ROCm 7.1, Mesa 26.2 RADV) — also built on Fedora 43 / ROCm 6.4

The XDNA 1 NPU in the 7040-series has no Linux LLM runtime — inference is GPU + CPU only.

Quickstart

git clone <this-repo> framework16-local-llm && cd framework16-local-llm
./setup.sh            # deps + build (ROCm & Vulkan) + download Coder-7B + install service
# → http://127.0.0.1:8080  (web UI in a browser, OpenAI API at /v1)

Or step by step: ./setup.sh deps, ./setup.sh build, ./setup.sh model coder-7b, ./setup.sh service. Run ./setup.sh doctor anytime to verify the stack, and ./setup.sh update to pull the latest llama.cpp and rebuild cleanly.

Two backends — pick by workload

Measured on the RX 7700S, Qwen2.5-7B Q4_K_M, full offload (Fedora 44, llama.cpp 2026-10):

Backend	Prompt processing (pp512)	Token generation (tg128)	Best for
ROCm	1302 t/s	52 t/s	Long context / RAG / agentic coding (prefill-bound) — the all-rounder
Vulkan	1000 t/s	54 t/s	Short-prompt chat, zero-maintenance fallback

ROCm wins prefill by ~30%; decode is now near-parity (Vulkan +4%, down from +20% in June). The always-on service and llama-go both default to ROCm; Vulkan still decodes faster on small models (3B: +21%), so use -b vulkan for snappy 3B chat. History and the per-upgrade breakdown are in docs/BENCHMARKS.md.

Model capacity (8 GB VRAM)

Model	Fits VRAM?	pp512	tg128	Use
3B Q4	✅ full	2581 t/s	87 t/s	Agentic, autocomplete, tool-calling
7B Q4	✅ full	1302 t/s	52 t/s	Default daily driver
14B Q4	⚠️ partial (44/48 layers)	584 t/s	19 t/s	Max quality, slower

7B is the sweet spot. 14B (8.37 GiB) overflows 8 GB and runs partially on CPU — usable for quality-first single queries, not rapid chat. See docs/BENCHMARKS.md.

Usage

bin/llama-go                      # ROCm server, 32k context (default)
bin/llama-go -b vulkan -m 3b      # Vulkan, fastest decode on the 3B
bin/llama-go --chat               # interactive terminal chat
bin/llama-go -m 3b --prompt "hi"  # one-shot with the 3B model
bin/llama-bench-shootout 3b 7b 14b  # re-run the size sweep

bin/llama-vulkan-mode             # temp-swap to Vulkan :8081, restore ROCm service on exit
systemctl --user {status,restart,stop} llama   # manage the always-on service

Use from opencode (or any OpenAI-compatible client)

The server speaks the OpenAI API at /v1, so any compatible client works (opencode, Open WebUI, the openai SDKs, Continue, …). Register both backends as providers, each pointed at its port. For opencode, add to ~/.config/opencode/opencode.json:

llama-rocm (:8080) is the always-on systemd service — available immediately.
llama-vulkan (:8081) is on-demand: run bin/llama-vulkan-mode first. It frees the GPU from ROCm (8 GB fits only one 7B at a time) and restores the service when you exit.
The model id is just a label — llama-server serves whatever GGUF it has loaded; apiKey is a required-but-ignored placeholder.
limit.context is essential. Agentic clients inject huge system prompts + tool schemas; without a declared context limit, opencode assumes a tiny window and compaction-loops even on a one-line question. Match it to the server's -c (the service runs 32768). Restart opencode and start a fresh session after editing the config.

Layout

setup.sh                 one-shot installer (deps|build|update|model|service|doctor)
bin/llama-go             daily-driver launcher (backend + model + pinning)
bin/llama-vulkan-mode    temp-swap to Vulkan on :8081, auto-restore ROCm service
bin/llama-bench-shootout ROCm-vs-Vulkan benchmark
bin/_common.sh           shared config, paths, GPU pinning, model catalog
config/models.conf       model catalog with measured ngl recommendations
systemd/llama.service.in service template
docs/                    BENCHMARKS.md, HARDWARE.md (incl. Fedora-upgrade troubleshooting)
tests/                   test_binary_broken.sh (stale-binary detection)

Paths are overridable via ~/.config/fw16-llm/config (FW16_HOME, MODELS_DIR, GFX). License: MIT.