SLOM：在 0.5–3B 小模型上运行工具调用智能体的持久化编排框架

SLOM runs tool-using agents on 0.5–3B models as durable workflows: every model call and tool call is journaled to SQLite, so a crashed worker's runs resume elsewhere without paying for finished model calls again. There is no orchestration server to run: it's a Python package plus a SQLite file.

SLOM is the orchestration layer above SLIM, a small inference server for local models. SLIM decides whether a request runs now (admission control, deadlines, 503 + Retry-After) and explicitly leaves retry policy to the caller. SLOM is that caller.

Why this exists

A 0.5B model on a CPU takes seconds per call and is unreliable in ways that compound over a multi-step task:

• It's wrong in the format, not just the content. It emits JSON wrapped in prose, Python dicts instead of JSON, or repeats the same tool call forever.
• Its server pushes back. A busy inference server answers 503; a model reload takes it away for a minute.
• Agent runs are long and expensive to restart. A crash at step 5 of 6 shouldn't cost five model calls again.

The usual answer to the last point is a workflow engine like Temporal, which is a cluster to run before your first agent works. SLOM implements the part an agent loop needs (journaled steps, leases, fenced recovery, classified retries) in about 800 lines, and backs each guarantee with a test that breaks it on purpose and a measurement.

Key features

• Durable execution. await ctx.step(name, fn, arg) journals each result. After a crash, finished steps return their recorded result (~0.03 ms) instead of running again. Leases with fencing tokens let a crashed or stalled worker's runs move to another worker, safely. Tested by killing a real worker process mid-call.
• Failure-aware LLM client for slim-serve and other OpenAI-compatible /v1/completions servers. It maps 503/429/504/5xx/4xx, in-band stream errors and truncated streams to retry decisions, with jittered backoff that never undercuts Retry-After. A client-side concurrency cap matches server capacity. It always streams, so cancelling a run stops generation on the server.
• An agent loop built for small models. Bounded steps and repairs; a lenient parser that never raises on model output; a repeat-call guard; a forced answer when the model loops; two prompt protocols (generic JSON with prefill, and Qwen2.5's native tool-call format).
• Typed tools. A function plus a pydantic argument model. Bad arguments and tool errors go back to the model; unexpected exceptions are retried. Comes with a safe AST calculator and idempotency_key() for tools with side effects.
• HTTP API and CLI:
  • submit (idempotent run_id, optional ?wait=true), inspect with live step progress, cancel;
  • slom serve runs the API and a worker in one process, slom worker scales out on one host;
  • slom run "question" runs a single task in the current process.
• Observability. Prometheus metrics that separate live work from replayed steps, structured logs, and X-Request-ID = run.step.attempt propagated into slim-serve's logs.
• Evidence.
  • 124 tests: unit, engine semantics, API, contract tests against the real slim-serve v0.1.0, and a process-kill recovery test. They were checked by mutation (9 deliberately broken guarantees, all caught).
  • Benchmarks with committed raw results.

Architecture

flowchart LR
    client([Client]) -->|"POST /v1/runs · GET /v1/runs/id"| api

    subgraph host["one host"]
        api["API<br/>validate · submit · read"]
        db[("SQLite journal (WAL)<br/>runs · steps · leases")]
        subgraph w["worker process × N"]
            loop["claim loop<br/>lease · heartbeat · cancel"]
            wf["agent workflow<br/>render → model → parse → tool"]
            llmc["LlmClient<br/>stream · retry class · cap"]
            tools["tools"]
        end
        api --> db
        loop -->|"claim / heartbeat / finish<br/>(fenced by lease token)"| db
        wf -->|"ctx.step → journal"| db
        loop --> wf --> llmc
        wf --> tools
    end

    llmc -->|"SSE /v1/completions<br/>X-Request-ID"| slm["slim-serve<br/>llama.cpp · MLX"]
    slm -.->|"503 Retry-After · 504 · 400"| llmc
    prom([Prometheus]) -.->|"/metrics"| api
    prom -.->|":9464/metrics"| loop

Loading

• Request path. The API validates a run and inserts it as pending; it never executes anything. A worker claims the oldest claimable run inside a BEGIN IMMEDIATE transaction (so two workers can't both claim it), sets a lease and bumps the lease token. It then runs the workflow. Each ctx.step either returns a journaled result or executes, retrying as its policy says, and writes the outcome, fenced by the token, before returning.
• Interfaces.
  • A workflow is async def wf(ctx, input) -> output. A step function is async def f(arg) -> T with dataclass or plain types, since results are journaled as JSON.
  • A tool is @tool def f(args: SomeModel) -> str.
  • The LLM boundary is an OpenAI-style /v1/completions stream.
• Failure boundaries.
  • Bad model output, tool failure: the error goes back to the model.
  • Overload or LLM outage: the step retries, then the run fails.
  • Worker crash: other workers take over that worker's runs.
  • A run that keeps killing workers: the run fails after N claims.
  • Changed workflow code: the run fails loudly.
  • Host loss takes everything down: SLOM is single-host by design.
• State. Two tables, runs and steps. Nothing else is persisted.
• External dependencies. An LLM server. That's all.

The sequence diagrams (a run, crash recovery), the thread model and the full failure table are in docs/architecture.md.

Technical design

Each is argued with alternatives and limits in docs/design-decisions.md.

Performance / Evaluation

Everything below was measured with the scripts in benchmarks/, at the commit recorded in each summary.json. Host: a Windows 11 laptop (Intel Core Ultra 7 165U, 14 logical CPUs, 32 GB, CPU only) running normal background software. Methodology and caveats are in docs/benchmarking.md.

1. The orchestration makes a 0.5B model succeed more often

Qwen2.5-0.5B-Instruct (Q4_K_M) on slim-serve, greedy decoding, 42 arithmetic word problems (6 templates × 7, exact integer answers). The question is whether SLOM gets a tiny model to use a tool and finish. It's a narrow test by design, not a general agent benchmark.

¹ Three problems were inspected while designing the repeat guard and forced answer.

• Significant: agent-qwen vs the CoT baseline. On the same problems it fixed 15 and broke 3; exact McNemar test p = 0.008. agent-qwen vs agent-json: 8 fixed, 0 broken, p = 0.008.
• Not significant: agent-json vs CoT (8 fixed vs 4 broken, p = 0.39). I'm not claiming the JSON protocol beats the baseline.
• What the orchestration fixed: all 84 agent runs ended with an answer. None crashed, failed to parse or ran out of steps. Before the forced answer was added, the JSON protocol ended all 3 development problems with no answer at all, re-issuing the same tool call until its repair budget ran out.
• What it can't fix: every one of the 36 wrong agent answers comes from a wrong formula. No tool call in those runs computed the right value. Both agents scored 0/7 on two templates: they subtract the delivery fee in "price × qty + fee", and drop the multiplication in "boxes × size − sold". In 23 of the 36 the answer is exactly what the model's own expression computes; in the other 13 the model also mis-reported the tool's result (e.g. dropped a minus sign).
• Protocol matters: the format the model was trained on beat a generic one by 8 problems, in fewer calls. That's measured on one model; it's the default worth testing, not a law.

Raw rows, transcripts and the analysis are in benchmarks/results/eval-agent/ (analyze_eval.py reproduces the tests from rows.csv).

2. Crash recovery: no lost steps, recovery time set by the lease

A worker process is hard-killed at a random moment during a 10-step run, and a replacement starts immediately. 10 trials per lease length:

The single duplicate is at-least-once semantics, visible: that kill landed between a step's side effect and its journal commit. This is the window idempotency_key() exists for. Finished steps were never re-executed in any trial (also asserted by tests/test_crash.py).

3. Backpressure: match the client to the server, and jitter the retries

slim-serve with the fake engine (2 replicas, queue of 2, 0.2 s per call), 40 runs × 3 calls submitted at once; ideal makespan 12.0 s. Two interleaved repetitions:

• With the cap at replicas + queue, the batch finished about 8% over ideal.
• Oversubscribing is absorbed (every 503 retried, every run done) but costs 2.2–2.3× the makespan.
• The first version of the retry policy had no jitter. Clients rejected together retried together, and the batch ran 4–6× slower still. That's why RetryPolicy jitters and treats Retry-After as a floor. The pre-fix result is kept in results/overload-nojitter/.

4. What durability costs

No model; no-op steps, so this is engine plus journal only. Ranges are over three separate invocations:

FULL is fsync-bound: a few hundred step commits per second per journal, flat with concurrency because SQLite has a single writer. A model call on this CPU takes 0.5–10 s, so the journal adds well under 1% to a model-bound step.

Failure handling

Observability

Metrics (Prometheus; worker on :9464, API on /metrics):

Replayed steps never touch the latency, token or error metrics, so a recovery doesn't look like a burst of very fast model calls.

Logs. Structured, in the same format as slim-serve (SOF_LOG_JSON=true for JSON lines). Run-scoped lines carry run_id.

Correlation. Every model call carries X-Request-ID: <run_id>.<step>.<attempt>. It's stored in the run's result (steps[].llm_request_id) and logged by slim-serve on its access line, so grep <run_id> works across both services. The test suite asserts that the two lists match.

Live progress. GET /v1/runs/{id} returns the steps journaled so far, with attempts and durations, while the run is still going.

Quick start

Requires Python 3.11+ and uv, plus a running slim-serve.

# 1. The model server (separate repo)
git clone https://github.com/ragnarlothbrok53/slm-inference-engine.git
cd slm-inference-engine && uv sync --extra llama
mkdir -p models && curl -L -o models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q4_k_m.gguf
uv run slim-serve serve                     # :8000

# 2. SLOM
git clone https://github.com/ragnarlothbrok53/slom.git && cd slom
uv sync
uv run slom run "A shop sells pens at 17 rupees each. Priya buys 23 pens and pays a 4 rupee delivery fee. How much does she pay?" --protocol qwen

Real output of that command (Qwen2.5-0.5B on a laptop CPU):

model> <tool_call>
{{"name": "calculator", "arguments": {"expression": "23 * 17 - 4"}}
</tool_call>
tool> 387
model> Priya pays 387 rupees for the pens.

answered: Priya pays 387 rupees for the pens.
[2 LLM calls, 1 tool calls, 706+40 tokens, 1.8s in the model]

The orchestration did its job: one well-formed tool call, the result, a clean answer, in 2 calls. The answer is wrong: the model subtracted the delivery fee. In the evaluation it wrote price * qty - fee on all 7 problems of this template, with both protocols. No amount of orchestration fixes a wrong formula that is then computed faithfully. See Performance.

Docker (the whole stack; slim-serve is built from its GitHub tag):

mkdir -p models   # put the GGUF here, as above
docker compose up --build                  # API on :8080, model server on :8000
SLM_ENGINE=fake docker compose up --build  # no model: checks the wiring only
docker compose up --scale worker=3         # more workers on the same journal

Usage

uv run slom serve        # API + worker, http://127.0.0.1:8080

# Submit and wait for the result
curl -s 'localhost:8080/v1/runs?wait=true' -H 'content-type: application/json' \
  -d '{"question": "A crate holds 48 bottles. How many bottles are in 26 crates?", "protocol": "qwen"}'

# Submit without waiting (idempotent on run_id), then poll and cancel
curl -s localhost:8080/v1/runs -H 'content-type: application/json' \
  -d '{"question": "...", "run_id": "order-42"}'
curl -s localhost:8080/v1/runs/order-42          # status, answer, steps so far
curl -s -X POST localhost:8080/v1/runs/order-42/cancel

POST /v1/runs fields: question; tools (a subset of the worker's tools, null = all); max_steps (1–20, default 6); max_repairs (0–5, default 2); max_tokens (per call); temperature; protocol (json | qwen); run_id (an idempotency key).

Adding tools. Build your own worker with them:

import asyncio
from pydantic import BaseModel, Field
from slom import BUILTIN_TOOLS, ToolError, idempotency_key, tool
from slom.config import Settings
from slom.worker import serve_worker

class RefundArgs(BaseModel):
    order_id: str = Field(description="e.g. A-1001")
    amount: float

@tool(description="Refund an order. Returns the refund ID.")
async def refund(args: RefundArgs) -> str:
    if args.amount <= 0:
        raise ToolError("amount must be positive")       # the model sees this and can retry
    # `payments` is your client; the key is stable across retries and crash recovery.
    return await payments.refund(args.order_id, args.amount, idempotency_key=idempotency_key())

asyncio.run(serve_worker(Settings(), tools=[*BUILTIN_TOOLS, refund]))

Writing other workflows. The engine isn't agent-specific:

from slom import RunContext, Store, Worker, Workflow

async def pipeline(ctx: RunContext, doc_id: str) -> str:
    text = await ctx.step("fetch", fetch_document, doc_id)
    summary = await ctx.step("summarize", summarize, text)   # not repaid after a crash
    return await ctx.step("store", save_summary, summary)

worker = Worker(Store("slom.db"), [Workflow.define("pipeline", pipeline)])

Configuration

Environment variables (prefix SOF_), a .env file (see .env.example), or CLI flags (--db, --llm-url, --host, --port), which take precedence. Full definitions, types and bounds are in config.py.

Testing

uv sync                      # dev deps include slim-serve v0.1.0 for the contract tests
uv run pytest                # 124 tests, ~30 s
uv run ruff check . && uv run ruff format --check . && uv run mypy

What the tests pin down (tests/):

• Engine (test_engine.py, test_store.py):
  • finished steps aren't re-executed on resume;
  • four separate connections racing for 20 runs claim each exactly once;
  • expired leases are reclaimed, and the old owner is fenced;
  • a zombie worker stops itself;
  • retries are bounded and honour Retry-After; permanent errors fail fast;
  • failures replay identically; changed workflow code is detected;
  • poison runs are abandoned; cancellation stops the in-flight step;
  • graceful shutdown hands runs back; the concurrency limit holds;
  • the documented retry budget (45–90 s) holds.
• Agent (test_agent.py):
  • tool then answer; repair after invalid output; bounded repairs;
  • the repeat guard and the forced answer; the step budget;
  • tool errors are shown to the model; LLM failures (transient retried, permanent fatal);
  • both protocols;
  • a resumed run makes exactly one more model call, with a byte-identical prompt.
• LLM client (test_llm.py): SSE parsing, the status-code → retry mapping, in-band errors, truncated streams, total deadline, concurrency cap, readiness.
• Contract with slim-serve (test_integration.py, the real v0.1.0 server over a socket):
  • an end-to-end agent run, with request IDs matching one-to-one between the journal and slim-serve's logs;
  • a server with no queue, oversubscribed 4×: every 503 is retried, every run completes, and no model call runs twice;
  • cancelling a run stops generation on the server.
• Crash (test_crash.py): a worker process is killed during a model call, and a new process finishes the run. The finished model call ran once; the interrupted one ran twice, with an identical prompt.

Mutation check. Each of these deliberate bugs makes at least one test fail:

• disabling journal replay;
• removing write fencing;
• a deferred (not IMMEDIATE) claim transaction;
• not retrying 503;
• ignoring the cancel flag;
• ignoring Retry-After;
• not checking the lease on heartbeat;
• removing the repeat guard;
• a non-streaming client.

CI (.github/workflows/ci.yml) runs lint, format, strict mypy and all tests, and brings up the compose stack (slim-serve with the fake engine) to push a run through API → journal → worker → slim-serve and back.

Benchmarking

uv run python benchmarks/bench_engine.py      # journal overhead, FULL vs NORMAL  (~1 min, no model)
uv run python benchmarks/bench_recovery.py    # kill workers, measure recovery   (~3 min, no model)
uv run python benchmarks/bench_overload.py    # vs a real slim-serve, fake engine (~4 min)
uv run python benchmarks/eval_agent.py --llm-url http://127.0.0.1:8000   # real model (~40 min on CPU)

Each writes benchmarks/results/<name>/summary.json (environment, commit, config, summary) and rows.csv (raw rows). Details and caveats are in docs/benchmarking.md.

Design tradeoffs

Why not Temporal? The previous version of this repo used it. Temporal is the right tool for multi-host workers, long timers, signals, versioned workflows and an ops UI. SLOM needs none of these for its target use case, a few workers next to a local model, and each of them would add a cluster to run first. If you need them, the concepts map directly: steps are activities, the journal is the event history, the lease is the heartbeat timeout.

Why SQLite? Several processes making small transactional writes on one host is exactly what SQLite in WAL mode does well, and it ships with Python. DuckDB was considered and rejected: one writing process per file. Postgres is the scale-out path (see below).

Where it breaks down:

• One host. SQLite's locking doesn't hold on network filesystems, so all workers must share a disk (or a VM under Docker Desktop).
• One writer. With FULL, a journal sustains a few hundred step commits per second. That's irrelevant at CPU model speeds (one call per second or more), but a ceiling for fast GPU backends with many workers.
• Recovery waits for the lease. That's 15 s by default: a shorter lease recovers faster and costs more heartbeat writes.
• No workflow versioning. Changing the agent's step sequence fails in-flight runs (loudly). Drain workers before deploying such changes.
• No retention. The journal grows; there's no pruning yet.
• The model is the weak link. The orchestration can make a 0.5B model use a tool and stop looping. It can't make it write the right expression: the remaining errors in the eval are mostly wrong formulas, faithfully computed.

Current limitations:

• ChatML models only, via raw completions (the chat template is applied client-side).
• No constrained decoding (slim-serve has no grammar support).
• No auth or multi-tenancy on the API.
• ?wait=true and idle workers poll the journal.
• The MLX backend path is untested here, as in slim-serve.

At 10x scale I would change:

1. Storage. Move store.py to Postgres: FOR UPDATE SKIP LOCKED claims, the same fencing token, and LISTEN/NOTIFY instead of polling. That allows workers on several hosts. The engine, agent and API don't change.
2. Backpressure. Use a global LLM concurrency budget (a token bucket in the database, or a gateway) instead of a per-worker cap, since workers × cap must stay within the servers' replicas + queue. Route model calls by queue depth (slm_queue_depth) across several slim-serve instances.
3. Output constraints. Use constrained decoding (a GBNF grammar from the tool schemas) in slim-serve instead of prefill-and-repair. It removes invalid output by construction.
4. Deploys. Add workflow versioning (a version on each run, old code kept until the run drains) and retention for finished runs.
5. Tracing. Use OpenTelemetry once tools call other services; the step request ID is where the trace context would go.

Roadmap

• grammar / response_format support in slim-serve, and a constrained-decoding protocol here
• Postgres store behind the same interface
• slom prune --older-than (journal retention)
• A version field on runs, for safe deploys of changed workflows
• Evaluation beyond arithmetic word problems (multi-tool tasks), and on 1.5B / 3B models

Project structure

src/slom/      engine.py · store.py · agent.py · prompts.py · parsing.py · llm.py · tools.py
              api.py · worker.py · cli.py · config.py · contracts.py · metrics.py · logs.py
tests/        engine, store, agent, API, LLM client, slim-serve contract, process-kill recovery
benchmarks/   bench_engine · bench_recovery · bench_overload · eval_agent · data/ · results/
docs/         architecture.md · design-decisions.md · benchmarking.md

License

MIT, see LICENSE.