PagedMoE 发布：在 Apple Silicon 上从 SSD 分页 MoE 专家权重

PagedMoE

No longer updated. This repo is an early version of the streaming engine. It is slower than Global Agent, less safe, and not current with the latest implementations.

Updated benchmarks, and the faster, safer, smoother runtime, are at globalagent.dev.

Run large Mixture-of-Experts (MoE) LLMs locally on Apple Silicon even when the model is larger than your Mac's unified memory.

Page experts from the SSD so models larger than your unified memory actually run. Works on any Apple Silicon Mac; on a 48 GB system it drove a full agentic loop against a ~570 GB Kimi checkpoint. Route prediction, a lean miss-read path, and an optional governed sidecar keep decode moving. Same bit-identical output when pruning is off.

MLX / Apple Silicon. Offloads MoE expert weights to the internal SSD and pages them into Metal on demand - no CUDA, no NVIDIA, no separate server or quantization pass required.

Jump: Quick start | Your Mac | Results by Mac | Requirements | Why it fits | Compatible models | Extras | License | Commercial | Citation | Contact

Baseline below is an M5 Pro, 48 GB - the machine the recipes were tuned on. Smaller and larger Macs scale from it; see Your Mac. Without PagedMoE, none of these load. With it, they run full agent loops - tools, multi-turn, long prompts - at usable speed; experts stay on disk until the router asks.

_{PagedMoE auto-detects your system specs to apply the correct configuration settings and ensure support.}

_{Looking for your Mac?}

Prefill is a cold ~2k-token prompt; decode is wall-clock after warmup on the same KV. Every row is a measured ladder run on the baseline Mac under the prune / wait / sidecar recipe in Tuning. Kimi is the top of the ladder: a 578 GB MoE that will not load resident, streamed under the same recipe as DeepSeek and GLM.

• 235B - shipped recipe (prune 0.7 + wait 0.2 + sidecar) measured ~7 tok/s decode; prune 0.5 ladder run hit 6.67 tok/s (steady, 128 tok warmup)
• GLM-4.7 - shipped recipe measured 6.4 tok/s vs 1.8 tok/s full-mixture on the same A/B (96% token agree)
• 480B - shipped recipe (prune 0.7) measured ~5 tok/s decode at ~270 GB checkpoint size
• Kimi-K2 - steady ladder at prune 0.8 measured 8.75 tok/s decode (peak 31.59 GB; prefill ~27 tok/s). Largest SwitchGLU in the chart.

Defaults leave prune off and are slower on the disk-bound models. Every knob is in expert_stream/config.py.

The RAM gap looks wrong until you remember MoEs are sparse: most experts are idle on any given token, so you do not need the whole library in RAM.

PagedMoE keeps the always-on backbone in memory and pages experts when the router picks them. LRU cache + route prediction (and an optional governed sidecar) keep agent turns warm.

Status: frozen at v0.2.7. This repo is no longer updated. Installs from a clone still work. New work lives at globalagent.dev.

Naming: product / repo / PyPI / CLI = PagedMoE (paged-moe). Python import stays expert_stream; env knobs stay EXPERT_STREAM_*.

What your Mac can run

Pick your Mac. Each link jumps to a measured chart in the same format as the baseline table above - models that stock MLX will not load, with peak RAM, headroom, prefill, and decode.

Macs: 16 GB Air / M2 Pro | 18 GB M3 Pro | 24 GB M4 Pro | 32 GB | 48 GB | 64 GB | 128 GB

paged-moe machine      # show what this Mac resolves to
paged-moe ladder       # full fit table for this Mac

Quick start

Three steps. After that, use mlx / mlx-lm / mlx-openai-server the way you already do - no new load API required.

1. Install PagedMoE into your mlx venv (hooks mlx_lm.load)
2. Edit ~/paged-moe-config.yaml and list the MoE checkpoints to stream
3. Use mlx as normal - loads of those paths go through PagedMoE automatically

1. Install

git clone https://github.com/noahclark556/paged-moe.git
cd paged-moe
chmod +x install_mlx.sh
./install_mlx.sh
# or: ./install_mlx.sh --python /path/to/venv/bin/python
# or: ./install_mlx.sh --models-dir ~/mlx-models

That installs the package, enables the auto-hook, and writes ~/paged-moe-config.yaml (asks which directory holds your MLX checkpoints).

Manual equivalent:

source ~/.mlx-env/bin/activate
pip install -e .
paged-moe install --models-dir ~/mlx-models

2. Edit ~/paged-moe-config.yaml

This file is the control plane. Put real paths to the MoEs you want to stream. Install seeds the tested checkpoint names under the models dir you chose (default ~/mlx-models/) - fix the paths if yours differ.

enable_all: false
debug: false
machine:
  auto: true
models:
  - path: /Users/you/mlx-models/qwen3-next
    env:
      EXPERT_STREAM_SIDECAR: "1"
  - path: /Users/you/mlx-models/qwen3-235-4bit
    env:
      EXPERT_STREAM_SIDECAR: "1"

Only paths listed under models: stream (unless you set enable_all: true). Everything else stays stock mlx-lm. Edit this file anytime; no reinstall.

Optional instead of yaml: drop a .paged_moe marker in the checkpoint folder, or export PAGED_MOE=1 for every load. Kill switch: PAGED_MOE_HOOK=0.

If you use mlx-openai-server, the path here should match model_path in ~/mlx-config.yaml.

3. Use mlx as normal

from mlx_lm import load, stream_generate

# Same call as always. If the path is in ~/paged-moe-config.yaml, it streams.
model, tokenizer = load("~/mlx-models/qwen3-235-4bit")
for chunk in stream_generate(model, tokenizer, prompt, max_tokens=200):
 print(chunk.text, end="", flush=True)

# Same idea for a local OpenAI-compatible server:
python -m expert_stream.server --model ~/mlx-models/qwen3-235-4bit --port 8080
# Point your agent at http://127.0.0.1:8080/v1

Check a path:

paged-moe status
paged-moe which /path/to/model
PAGED_MOE_DEBUG=1 # stderr: [paged-moe] stream via ... / passthrough ...

Who this is for

• Qwen / GLM / DeepSeek-class MoEs on Apple Silicon without a 128-512 GB box (tunable outside that range too)
• Agent loops (tools, multi-turn, long prompts), not just one-shot chat
• Plain mlx-community (or converted) checkpoints, not a special "streaming edition" of two curated models

Who this is not for

• Absolute minimum RAM footprint (~3 GB active). Other stacks optimize for tiny resident sets; this one spends unified memory on a large expert cache so agent turns stay warm
• NVIDIA / CUDA servers. This is an MLX / Mac tool
• Models that already fit in RAM. Stock mlx_lm.load is enough

What you get

1. Drop-in for mlx_lm.load on SwitchGLU MoEs - checkpoint can be 2x or 5x RAM
2. An OpenAI-compatible server your agent already knows how to talk to
3. Bit-identical output to a fully resident run when expert pruning is off
4. Agent-session hardening: prompt-prefix reuse (one prefill, then fork snapshots on trimmable KV), memory that yields to long prefills instead of OOMing Metal

Under the hood, mlx-lm routes MoE expert compute through SwitchGLU. This replaces that with a disk-backed version and keeps attention, KV cache, templates, and sampling. LRU + route prediction + optional slabs hide as much disk latency as the SSD allows.

Built for coding agents on Apple Silicon - Air through Ultra - against models the platform normally will not start.

Requirements

Any OpenAI-compatible client (including mlx-openai-server) can talk to python -m expert_stream.server. That stack is optional; it is not a hard install dependency.

Pins live in pyproject.toml (source of truth). ./install_mlx.sh regenerates requirements.txt from those deps before pip install.

Why it fits

On a normal load these MoEs want roughly their full checkpoint in RAM. PagedMoE keeps a working set resident and frees the rest as headroom - often +50 to +180 GB versus a full resident load, depending on the Mac and the model. Numbers below are the 48 GB baseline; your Mac's peaks and headroom are under Results by Mac.

Only the always-on backbone stays in RAM. Routed experts live on disk until needed. MoE size is mostly experts that almost never fire on a given token.

Experts are fetched from the SSD on demand into a working set that can be dropped later. The model never has to be fully materialized.

A large in-memory expert cache keeps recently used experts warm under your RAM budget.

Expert reads bypass the OS page cache so you are not double-spending memory on an invisible kernel copy of the same bytes.

Route prediction asks upcoming layers what they want and starts those reads before the GPU stalls. That is the big disk-bound decode win. An optional sidecar can learn online prefetch hints on top; wrap stays off unless you turn it on so the default path stays lean and predictable.

Miss reads prefer a small native C pool when it builds; same bytes as the Python path, with an automatic fallback.

Optional decode pruning skips weak experts so fewer SSD reads happen per token (you trade a bit of fidelity for speed).

Memory is reserved for context (KV) so a long conversation does not steal the expert cache mid-session.

Long prefills can temporarily shrink the expert footprint so Metal does not OOM, then warm back up for decode.

Prefill is priced in expert bytes, not FLOPs. A chunk that routes widely streams essentially the whole expert mass once. Layer-fused prefill (on by default) sub-chunks attention inside each decoder layer and runs the MoE once per prompt chunk, so a long agent prompt can be one pass over the experts instead of one pass per attention-sized step. Attention sub-chunk size is EXPERT_STREAM_ATTN_SUB_CHUNK (default 2048); the Metal score-matrix bound still caps it. Watch stderr for [prefill] ... passes over the mass ... blocked on disk ... compute.

Agent turns reuse the prompt prefix instead of redoing the whole history each tool round. On trimmable KV caches (DeepSeek / Qwen3-MoE / GLM and friends) the cold path prefills once, then forks prefix snapshots with deepcopy+trim. Hybrid / non-trimmable stacks still stop mid-prefill at snapshot boundaries.

If the model already fits, it can run fully resident. Streaming is the escape hatch for oversized MoEs. Same API either way.

Compatible models

Anything mlx-lm loads whose MoE layers use SwitchGLU:

No custom checkpoint format. Plain MLX safetensors.

Extras

Things beyond install -> yaml -> chat. Optional; the Quick start path works without them.

Sidecar offline pretrain

Online learning already runs during normal decode when EXPERT_STREAM_SIDECAR=1. For a faster warm-start from a prompt corpus (before or instead of a long agent session):

# Fast warm-start (short decode, few prompts, many fit epochs)
paged-moe-pretrain --quick \
 --model ~/mlx-models/qwen3-235-4bit \
 --corpus /path/to/prompts.jsonl

# Same module form:
python -m expert_stream.pretrain_sidecar --quick \
 --model ~/mlx-models/qwen3-235-4bit \
 --corpus /path/to/prompts.jsonl

Corpus can be a directory of .txt / .md / .jsonl, or one .jsonl with prompt / text / chat messages fields. Weights land under ~/.paged_moe/sidecar/ (override with EXPERT_STREAM_SIDECAR_DIR).

Useful variants:

# Dense-fit only from a wrap bank already filled by live use or a prior run
paged-moe-pretrain --fit-only --epochs 12 --model ~/mlx-models/qwen3-235-4bit

# Clear lock bits so online training resumes (no MoE load)
paged-moe-pretrain --unlock-all

# Ctrl+C saves progress; re-run the same command to continue.
# --fresh / --force wipes slot weights + progress and starts over.

Shipped builds may also seed a small pretrained bundle under expert_stream/pretrained_sidecar/ on install when that allowlist is filled for a release.

OpenAI-compatible server

paged-moe-server --model ~/mlx-models/qwen3-235-4bit --port 8080
# or: python -m expert_stream.server --model <path> --port 8080

Same streaming load path as mlx_lm.load. Point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

Tuning via yaml env

Per-model knobs go under each entry in ~/paged-moe-config.yaml (or the process environment). Common ones:

Full catalog lives in comments at the top of expert_stream/config.py. Wrap / prefill / residency / prune heads stay off unless you turn them on. Native read is on in the sample recipes (falls back to Python if the extension cannot build).

Chart recipes (optional - not the package defaults). Drop under a model's env: in ~/paged-moe-config.yaml:

# DeepSeek-V3.2 / GLM-4.7 class (MoEGate) - matches the ~4 tok/s DeepSeek row
env:
  EXPERT_STREAM_PRUNE: "0.8"
  EXPERT_STREAM_WAIT_ABOVE: "0.2"
  EXPERT_STREAM_SIDECAR: "1"
  EXPERT_STREAM_NATIVE_READ: "1"
  EXPERT_STREAM_ADAPTIVE_PREFILL: "1"

# Qwen3-235B / Qwen3-Coder-480B - same idea, slightly softer prune
env:
  EXPERT_STREAM_PRUNE: "0.7"
  EXPERT_STREAM_WAIT_ABOVE: "0.2"
  EXPERT_STREAM_SIDECAR: "1"
  EXPERT_STREAM_NATIVE_READ: "1"
  EXPERT_STREAM_ADAPTIVE_PREFILL: "1"

PRUNE / WAIT_ABOVE trade a bit of fidelity for disk reads skipped. Leave them unset for bit-identical decode (slower on the big MoEs).

CLI cheat sheet

Results by Mac

Bold is the daily-driver pick. Only models that stream (will not load resident) are listed. Back to Mac links.

Each chart is either sandbox-tested or run on a real machine. Sandbox matches that Mac's RAM, cache budget, and SSD delay on the baseline host, so tok/s can be slightly off a native chip of the same RAM class.

Mac: 16 GB

M2 / M3 / M4 Air, and M2 Pro (16 GB). 235B and GLM will not load.

• Sandbox tested
• On-device

Mac: 18 GB

M3 Pro (18 GB). GLM will not load.

• Sandbox tested
• On-device

Mac: 24 GB

M4 Pro (24 GB). GLM will not load. 235B becomes the daily pick.

• Sandbox tested
• On-device

Mac: 32 GB

M2 Pro / M3 Max class (32 GB). GLM joins.

• Sandbox tested
• On-device

Mac: 48 GB

M4 Pro / M4 Max / M5 Pro (48 GB). 480B, DeepSeek, and Kimi are in the baseline table at the top.

• Sandbox tested
• On-device

Mac: 64 GB

M4 Max / M5 Pro (64 GB). KAT is cache-resident at this size. Numbers are IO-scaled from the 48 GB on-device row (not loaded here).

• Sandbox tested
• On-device

Mac: 128 GB

M5 Max (128 GB). Coder-Next and KAT are resident; 235B and GLM still stream. Numbers are IO-scaled from the 48 GB on-device row (not loaded here).

• Sandbox tested
• On-device

License (dual)

Free under AGPL-3.0 for people and open projects. Companies shipping closed products, hosted services, or appliances without AGPL source disclosure should email for a commercial license:

Noah Clark - noah@qwertycode.org

In-repo summaries: LICENSING.md, COMMERCIAL_LICENSE.md.

Copyright (c) 2026 Noah Clark.

Citation

If you use this in research or write about it, please cite the repository (see CITATION.cff):

@software{clark_paged_moe,
 author = {Clark, Noah},
 title = {PagedMoE: Page MoE experts from disk on Apple Silicon},
 year = {2026},
 url = {https://github.com/noahclark556/paged-moe}
}

Contact

• Commercial license / dual-license: noah@qwertycode.org
• Bugs / features: GitHub issues