Quackster/tessera：C++20 多后端 LLM 推理引擎

Status: Heavy development. The API changes without notice. Greedy text generation is stable on the 27B targets. DFlash2 and MTP speculation run end to end. The code is usable for development and for backend tests. It is not ready for production use.

Tessera is a C++20 LLM inference engine. It runs open weight models on GPU. It has two compute backends. Vulkan supports cross vendor GPUs. ROCm supports AMD GPUs. One binary uses one backend. The backend is set at configure time.

First class targets are Qwen 3.8 27B in GGUF and Qwen 3.8 in MXFP4, with and without DFlash2 speculative decoding.

Why AMD (ROCm) and Vulkan only? The project author tests with a 7900 XTX and two R9700 cards. There is no recent NVIDIA GPU in the lab. The last NVIDIA GPU used here was a 1080 Ti. Backend support follows hardware available for testing.

Features

See PROGRESS.md for full status.

Chat UI

Lightweight, dark-mode AI chat UI with a simple sidebar for API keys and conversation management. It supports streamed model responses, collapsible thinking output, retry controls, token limits, and rendering rich content like inline SVGs directly in the chat.

Architecture

Core code is backend agnostic. All device work goes through the Backend interface. Only src/backends/vulkan/ includes Vulkan headers. Only src/backends/rocm/ includes HIP and ROCm headers. No vendor type crosses the backend boundary.

Models are data plus one module per architecture. Each architecture owns its config parse, weight map and layer assembly behind the Architecture interface (today src/models/qwen3_5/). It drives generic kernels for GEMM, attention, MLP, MoE, norms, RoPE, activations, and embeddings. Generic code has no per model branches.

Speculative decoding plugs in behind the strategy interface. It is off by default. DFlash2 is the current strategy. The non speculative path is the reference baseline. It stays stable within per backend tolerance.

Layout:

• include/tessera/ holds the public API.
• src/core/ holds the engine, the model, and the weight loaders.
• src/models/ holds one architecture module per model family.
• src/backends/vulkan/ holds the Vulkan backend.
• src/backends/rocm/ holds the ROCm backend.
• src/spec/ holds speculative decoding strategies.
• src/serve/ holds the HTTP layer.
• tools/cli/ holds the thin CLI.
• tests/ holds the single GoogleTest binary.
• docs/PROGRESS.md tracks progress.

See AGENTS.md for architecture rules and for hard rules.

Model support

Model weights live outside the repo. Each variant uses one flat directory. Model paths are runtime configuration. Code and tests never hard code model paths.

CLI examples:

# GGUF target, greedy decode.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf

# GGUF target with MTP speculation. The MTP head drafts from the target, so it needs no draft path.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf --speculate

# MXFP4 target, greedy decode.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8

# MXFP4 target with MTP speculation.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 --speculate

# MXFP4 target with the DFlash2 draft checkpoint.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8

# MXFP4 target and DFlash2 draft over the HTTP server.
./cmake-build-vulkan/tessera-cli serve --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 --port 8080

The CLI loads the model and uploads weights. It attaches the draft strategy when --draft or --speculate is set. It prints tensor count, total elements and device bytes. With --max-completion-tokens N it runs N greedy decode steps from --prompt-text and streams the decoded text to stdout as each token is generated. --speculate drafts with the MTP head on both run and serve. --draft drafts with the DFlash2 checkpoint. The DFlash2 draft is tested against the MXFP4 target, not the GGUF. --mmproj plus --image prepend image tokens. The image token id is detected from <|image_pad|>; no flag sets it.

Calibration

Calibration measures the engine settings that depend on the machine. It keeps the fastest value. It changes only speed. It never changes the produced tokens. Run it once on each machine and model, before the first run or serve.

The calibrate subcommand loads the model one time, warms it, and sweeps the candidate values for each setting. The sweep has no time cap; it runs to completion, so the prompt set sizes the run. It writes a JSON file and prints a report with the identity, the sweep points, the chosen settings, and an example run command.

# Sweep on ROCm with the DFlash2 draft and write the result.
./cmake-build-rocm/tessera-cli calibrate \
  --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 \
  --calibration ~/.config/tessera/calibration.json

The calibrate flags:

• --model <path> model to measure. Required.
• --draft <dir> DFlash2 draft checkpoint. It enables the draft block and draft context sweeps.
• --calibration <file> file to read and write. The environment variable TESSERA_CALIBRATION supplies it when the flag is absent. One of the two is required.
• --context <n> context length in the hardware key. Default 4096.
• --gpu <n> device index. Default 0, the first device.
• --kv-f16, --kv-q8, --kv-q4, --kv-fp8 KV cache type. Default int8 (kv8).
• --prompts <file> fixed prompts, one per non-empty line. Default three fixed prompts.

The settings it sweeps are the same settings in the tables below:

• mxfp4_split_target and mxfp4_split_cap on ROCm only. On Vulkan the fp8 kernel does not exist, so the command reports both as not applicable.
• prefill_chunk_tokens on both backends.
• attention_split on both backends, when the prompts reach the split threshold (1024 keys).
• draft_tokens and draft_context only when a draft is attached.

Apply a saved entry to run or serve with --calibration <file>, or set TESSERA_CALIBRATION. The engine uses the saved value for the current machine and model unless an explicit flag overrides it. An explicit flag always wins.

# Serve with the calibrated values for this machine and model.
./cmake-build-rocm/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 \
  --calibration ~/.config/tessera/calibration.json \
  --port 8080

The file is JSON. It holds one entry per hardware key. The key joins the backend, the device name, the model name, the context, the KV type, and the strategy. Each entry stores the chosen settings, the measured decode and prefill rates, the memory-bound context for the KV type, and the date. A machine, model, context, KV type, or strategy that differs from the key does not match the entry. The path is runtime configuration and is never hard-coded.

Usage

Simple text prompt with the CLI:

./cmake-build-vulkan/tessera-cli run \
  --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --prompt-text "Explain gravity in one sentence." \
  --max-completion-tokens 64

--max-completion-tokens is optional. Without it the run fills the remaining context. Output tokens stream to stdout during the run, so the first token shows at once instead of after the full run.

The CLI applies the model chat template by default. It uses the raw text with --no-chat. String prompts need a model with a tokenizer. Both formats provide one: GGUF carries it in metadata, and the MXFP4 loader parses the HuggingFace tokenizer.json (and the chat template from tokenizer_config.json or chat_template.jinja). The CLI and the HTTP server accept string prompts for either format. String prompts still fail on a model without a tokenizer, such as a bare DFlash2 draft directory.

C++ API with a text prompt:

#include "tessera/engine.hpp"

auto engine = tessera::Engine::Create({});
if (!engine) return 1;
auto model = (*engine)->LoadModel(
    tessera::ModelOptions{"~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf"});
if (!model) return 1;

const tessera::Tokenizer* tokenizer = (*model)->GetTokenizer();
std::string text = "Explain gravity in one sentence.";
if (auto rendered = (*model)->ChatPrompt(text)) text = *rendered;
auto ids = tokenizer->Encode(text);

tessera::GenerateOptions options;
options.max_completion_tokens = 64;  // Optional. Zero fills the remaining context.
options.prompt_tokens = *ids;
auto generated = (*engine)->Generate(**model, options);
if (!generated) return 1;
auto text = tokenizer->Decode(*generated);

HTTP API with the server:

# GGUF target.
./cmake-build-vulkan/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --port 8080

# MXFP4 target with the DFlash2 draft. MTP speculation uses --speculate instead.
./cmake-build-vulkan/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 \
  --port 8080

curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain gravity in one sentence.", "max_completion_tokens": 64}'
curl http://127.0.0.1:8080/health

max_completion_tokens is optional. Without it the server fills the remaining context. Speculation is set when the server starts and applies to every served turn.

With --api-key set, requests need Authorization: Bearer <key>. GET /health and GET /metrics stay public.

Command-line arguments

The first argument selects the mode: run decodes a prompt, serve starts the HTTP server, and --list-gpus lists the devices and exits. The table gives every flag, its scope, and its default.

The KV cache is fp32 when no --kv-* flag is set. The sampling defaults match the Qwen 3.8 27B defaults.

Environment variables:

• TESSERA_API_KEY supplies --api-key when the flag is absent.
• TESSERA_MXFP4_W4A8=1 enables the served MXFP4 target's W4A8 linear activation.
• TESSERA_CALIBRATION supplies --calibration when the flag is absent.
• TESSERA_MXFP4_SPLIT and TESSERA_MXFP4_SPLITCAP override the split-K target and cap as a diagnostic (they win over the request value).
• TESSERA_TILED_MIN_ROWS, TESSERA_TILED_MIN_COLS and TESSERA_PREFILL_ATTN_PAIRS override the tiled-GEMM dispatch thresholds and the prefill attention work budget for measurement.

Build and test

Requirements:

• CMake 3.30 or later.
• A C++ compiler with C++23 support. The code stays C++20 style.
• For Vulkan: Vulkan SDK and glslangValidator.
• For ROCm: ROCm toolchain, rocminfo, and a supported AMD GPU.

Configure and build:

cmake -B cmake-build-vulkan -DTESSERA_BACKEND=vulkan
cmake --build cmake-build-vulkan -j

cmake -B cmake-build-rocm -DTESSERA_BACKEND=rocm
cmake --build cmake-build-rocm -j

CMake defaults to a Release build. Measure speed only on Release. Build serially. The gcc-15 toolchain crashes under parallel builds.

Run tests:

ctest --test-dir cmake-build-vulkan --output-on-failure
ctest --test-dir cmake-build-rocm --output-on-failure

All runtime coverage lives in one test binary. Add test functions to tests/*.cpp. Do not add new CMake test executables. Device tests skip cleanly when no device is present. Numerical checks use per backend tolerance.

Verified devices:

State the backend used when reporting results.

Current progress

See PROGRESS.md.

License

Tessera uses the Apache License 2.0. See LICENSE.