OrchestrANT：面向 Python AI 原型开发的编排与 LLM 基准测试工具

👨🏻‍💻 Fast python AI prototyping 🐍

Docs can be found here.

OrchestrANT

Table of Contents

• About The Project
  • Key Features
  • LLM benchmark lab
  • Useful Tools
• Overview
• Getting Started
  • Setup
  • Pre commit hook
  • Dependency updates
  • Installation
  • Deployment Recommendations (Hardware/Software)
• Tests
• Demos
• Contributing
• License
• Contact and Maintainers
• References

About The Project

Four things live in this repository, and that is deliberate (owner decision, 2026-09-15 — no split, no code moves):

• The package. AI workload orchestration: camera pipelines, YOLO monitoring, streaming, and system/GPU metrics.
• The family's LLM benchmark lab. benchmarks/ plus the orchestrant.benchmark runner behind orchestrant-bench — the suite the rest of the family points at when it asks which model to run.
• The template. What the repository description advertises — "Lets bootstrap your Python AI project": the uv/ruff/ty setup, the Cython wheel build, the Sphinx docs and the ANTfrastructure CI wiring are meant to be copied into a new project.
• The Reflex viewer. frontend/, which reads the lab's result manifest directly.

The distribution is OrchestrANT; the importable package is orchestrant:

Console scripts (pyproject.toml [project.scripts]): yolo-monitor, orchestrant-smoke, orchestrant-bench.

yolo-monitor needs no working directory of its own:

• Model. Without --model, it takes the first of these that exists:
  1. $ORCHESTRANT_MODEL
  2. the install's share/orchestrant/models/ ($ORCHESTRANT_DATA_DIR, or beside the interpreter of a packaged install)
  3. resources/models/ in the working directory
  4. resources/models/ in the checkout of an editable install
• Logs. A packaged install writes them to %LOCALAPPDATA%\OrchestrANT\logs or $XDG_STATE_HOME/orchestrant/logs; $ORCHESTRANT_LOG_DIR overrides. Anything else still writes to logs/.
• Execution provider. CUDA, then DirectML, then CPU, whichever the installed ONNX Runtime offers.
• yolo-monitor --self-test loads the model, runs one inference on a blank frame and prints a JSON report. It needs no camera or display, and it is the install check for packaged builds.

The heavy dependencies are extras, chosen at install time:

• app — exactly what yolo-monitor imports beyond the core (numpy, OpenCV, ONNX Runtime); the packaged app's lean set, so it excludes every ml-ai* extra, which brings its own OpenCV and ONNX Runtime

• pytorch-cpu / pytorch-cu130 / pytorch-rocm10 / pytorch-custom — one torch backend, mutually exclusive (pytorch-custom takes your own wheelhouse via --find-links; pytorch-rocm10 was pytorch-rocm71 up to v0.0.28)

• ml-ai / ml-ai-webgpu / ml-ai-nvidia / ml-ai-rocm — ONNX Runtime for that backend plus OpenCV, scikit-learn, mlflow, optuna, IREE and LiteRT

• gpu / gpu-nvidia / gpu-directml / gpu-rocm — the GPU execution provider and the vendor's monitoring library

• frontend — wxPython for the YOLO viewer and Reflex for the benchmark viewer

• test, docs, packaging

Key Features

• GPU monitoring for two vendors through one GPUProbe: NVML for NVIDIA, ADL (Windows) or amdgpu sysfs (Linux) for AMD, no pyadl needed.
• orchestrant-bench: throughput, time-to-first-token, decode rate and a verifiable-answer correctness probe against any OpenAI-compatible endpoint, with the host GPU recorded in every result.
• A Reflex viewer for the benchmark results in frontend/.
• orchestrant-smoke: a shipped wheel smoke test that exercises torch, torchvision, ONNX Runtime, OpenCV, IREE and LiteRT.
• Static analysis is a gate on both CI lanes: ruff check --no-fix, ruff format --check, ty, bandit, vulture and codespell.

See CHANGELOG.md for the history behind each of these.

LLM benchmark lab

The measurement suite — the sweep, the coding / tool-calling / agent-loop benchmarks, the prompts and the tracked results — lives in benchmarks/; the Reflex viewer lives in frontend/. The serving stack it measures is owned by ANTfrastructure's linux/llm-stack/.

Useful Tools

Overview

The versioning of the package can be viewed in CHANGELOG.md.

Getting Started

Setup

Search the tree with ripgrep (rg): winget install --id BurntSushi.ripgrep.MSVC -e --scope user on Windows, apt install ripgrep on Linux.

Feel free to adjust for your own environment. F.e. create a virtual venv with a specific python version.

uv venv
python3.11 -m venv .venv

Pre commit hook

uv venv
source .venv/bin/activate # .venv/Scripts/activate on pwsh
uv pip install pre-commit
pre-commit install
# run on all files once (optional)
pre-commit run --all-files

Dependency updates

bash scripts/linux/renovate-local.sh                   # what is behind (report)
bash scripts/linux/renovate-local.sh --managers pep621 # the pyproject.toml pins
bash scripts/linux/renovate-local.sh --apply --dry-run # the gitlink plan

--apply moves gitlinks only for submodules that declare a branch = in .gitmodules — here only third_party/ANTfrastructure (branch = develop) does, so nothing else can come back REFUSED. Rationale and what --apply moves: dependency-updates.md.

Installation

Ready-made packages: no Python needed. Every build of develop and main produces them as CI artifacts, and a v* tag attaches them to its GitHub release. Each one carries its own CPython, the locked dependencies, ONNX Runtime and the YOLO model, and CI starts each one once (yolo-monitor --self-test) before it ships:

The Linux packages need glibc 2.43 (Ubuntu 26.04 or newer), which the bundled ONNX Runtime asks for. On Windows, OpenCV is the build without Media Foundation, so it also runs on Windows N and Server Core.

The MSIX is signed with a test certificate, not a trusted publisher, so Windows installs it only after you trust that certificate. From an elevated PowerShell:

Import-Certificate -FilePath .\orchestrant-<version>-windows-x64-test-signing.cer -CertStoreLocation Cert:\LocalMachine\TrustedPeople
Add-AppxPackage .\orchestrant-<version>-windows-x64.msix

The MSI needs no such step.

To use OrchestrANT as a Python library instead, there are three ways to install it:

1. Install directly via pip:

   pip install OrchestrANT@git+https://github.com/Kataglyphis/OrchestrANT

   or install a specific tagged version:

   pip install OrchestrANT@git+https://github.com/Kataglyphis/OrchestrANT@v0.0.1

2. Install after cloning the repo:

   git clone --recurse-submodules https://github.com/Kataglyphis/OrchestrANT
   pip install .

   or

   pip install -e .

   (an editable install: changes in the repo will be reflected in your environment)

3. Add as a submodule to your repository:

   git submodule add https://github.com/Kataglyphis/OrchestrANT

   Make sure that all dependencies are installed during your repo’s installation.
   (Not generally recommended, as it can be more complicated.)

Picamera web browser live stream

You need to shared system packages for the picamera2 should be installed via apt (source)

sudo apt install python3-picamera2
uv venv --system-site-packages

Deployment Recommendations (Hardware/Software)

Python package deployment in pure C

For insights into deploying Python packages into production as “binary only” wheels have a look into the corresponding workflows (background: Protect source code).

After creating the wheel you can check content with the command:

# Unzip a .whl
python -m zipfile --extract <ZIP_DATEI> <ZIEL_ORDNER>

This will print all available compatible tags for deployment

pip debug --verbose

./scripts/linux/ci_static_analysis.sh > "ci_analysis_$(date +%Y%m%d_%H%M%S).log" 2>&1

This is the same gate CI runs, and it is a gate: it exits 1 if codespell, bandit, vulture, ruff check --no-fix, ruff format --check or ty check reports anything. Redirecting both streams into a log as above hides that from the terminal — check $?, or drop the redirect.

NOTE: If you want to install your package editable and you previously deployed
it you will need to delete all Cython generated files first. You can use the following
command for it:

find . -type f \( -name '\*.c' -o -name '\*.cpp' -o -name '\*.so' -o -name '\*.pyd' -o -name '\*.html' \) -delete  

Or on windows ... do this

Get-ChildItem -Path . -Recurse -File | Where-Object { $\_.Extension -in '.c', '.cpp', '.so', '.pyd', '.html' } | Remove-Item  

Tests

uv sync --extra pytorch-cpu --extra test
uv run pytest tests/unit
bash scripts/linux/ci_tests.sh   # what the Linux lane runs: pytest + coverage

The LLM lab's own suite, benchmarks/tests, is in testpaths beside tests, so every lane runs it. Its ten harness modules skip on Windows: the candidate sandbox needs POSIX rlimits, process groups and shell fixtures.

Demos

uv run python -m orchestrant.yolo.monitor
uv run python examples/monitoring.py

Contributing

Contributions make open source software better! To contribute:

1. Fork the project
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

MIT — see LICENSE.

Contact and Maintainers

• Primary contact: @Cataglyphis_
• Repository: GitHub

Maintainers:
Jonas Heinle

References

• yolov12