NIMStats：每小时自动跑 22 个 NVIDIA NIM 模型的开源基准测试面板

✨ What is NIMStats?

NIMStats automatically benchmarks 22 NVIDIA NIM models every hour using GitHub Actions and publishes the results to a beautiful, interactive dashboard. No servers, no subscriptions — just fork, add your API key, and go.

⚡ Quick Start

Get your own benchmarking dashboard running in under 5 minutes.

1. Fork & Clone

git clone https://github.com/MauroDruwel/NIMStats.git
cd NIMStats

2. Get a Free API Key

Visit build.nvidia.com → Create a free account → Copy your API key.

3. Add the Secret

In your forked repo: Settings → Secrets and variables → Actions → New repository secret

4. Deploy the Dashboard

5. Run Your First Benchmark

Actions → Benchmark NVIDIA NIM Models → Run workflow

That's it — your dashboard auto-refreshes every hour. ✨

📊 Dashboard Features

🔌 Developer APIs

NIMStats exposes lightweight, static API endpoints for querying the #1 model in different performance categories. Every time the hourly benchmark completes, these endpoints are updated.

Available Endpoints

JSON Response Schema

{
  "best_model": "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning",
  "provider": "nvidia",
  "score": 71,
  "intelligence": 14.9,
  "uptime": 90.6,
  "avg_response_time_ms": 4736.7,
  "best_response_time_ms": 432.0,
  "avg_time_to_first_token_ms": 312.5,
  "avg_throughput_tps": 163.3,
  "total_runs": 720,
  "success_count": 652,
  "wins": 364,
  "last_seen": "2026-07-07T10:00:08Z",
  "generated_at": "2026-07-07T10:08:49Z"
}

🤖 Benchmarked Models

22 models across 11 providers — click to expand

🏗️ How It Works

┌──────────────────── GitHub Actions (every hour) ──────────────────────┐
│                                                                       │
│   ┌─────────────────────┐        ┌─────────────────────┐              │
│   │  Job 1 — Group A    │        │  Job 2 — Group B    │ (parallel)   │
│   │  N/2 NIM models     │        │  N/2 NIM models     │              │
│   └──────────┬──────────┘        └──────────┬──────────┘              │
│              └──────────────┬───────────────┘                         │
│                    ┌────────▼────────┐                                │
│                    │  Merge + commit │ → history.db committed to repo │
│                    └─────────────────┘                                │
└───────────────────────────────────────────────────────────────────────┘
                              │
                   ┌──────────▼───────────┐
                   │  Cloudflare Pages    │ → auto-deploys on push
                   │  (static dashboard)  │   index.html + history.db
                   └──────────────────────┘

Parallel jobs = ~50% faster benchmarks ⚡

🛠️ Customization

Change the benchmark prompt

Edit PROMPT in scripts/test_models.py:

PROMPT = "Your custom prompt here"

Add or remove models

Use the model management script:

# List models in DB vs test_models.py
python scripts/manage_models.py list

# Add a new model to ALL_MODELS
python scripts/manage_models.py add your/custom-model

# Remove a model from ALL_MODELS and purge its data from history.db
python scripts/manage_models.py remove your/custom-model

# Purge all DB models not in ALL_MODELS
python scripts/manage_models.py purge

Or manually edit ALL_MODELS in scripts/test_models.py:

ALL_MODELS = [
    "your/custom-model",
    # ...
]

Change the schedule

Edit .github/workflows/benchmark.yml:

- cron: '0 */6 * * *'  # Every 6 hours instead of every hour

Run locally

# Serve the dashboard
python3 -m http.server 8000
# Open http://localhost:8000

# Run benchmarks manually (requires NIM_API_KEY env var)
export NIM_API_KEY=your_key_here
python3 scripts/test_models.py

📦 Data Storage

history.db is a SQLite database persisted in the repo — the single source of truth. The browser loads it via sql.js (WebAssembly) and queries it entirely client-side. scripts/results.json is a temporary per-job artifact that is never committed.

Schema Architecture:

CREATE TABLE prompts (
  id INTEGER PRIMARY KEY,
  text TEXT UNIQUE
);

CREATE TABLE models (
  id INTEGER PRIMARY KEY,
  name TEXT UNIQUE,
  intelligence_score REAL DEFAULT NULL
);

CREATE TABLE errors (
  id INTEGER PRIMARY KEY,
  text TEXT UNIQUE
);

CREATE TABLE runs (
  id INTEGER PRIMARY KEY,
  timestamp TEXT NOT NULL,
  prompt_id INTEGER REFERENCES prompts(id),
  fastest_model_id INTEGER REFERENCES models(id),
  fastest_time INTEGER
);

CREATE TABLE model_results (
  run_id INTEGER REFERENCES runs(id),
  model_id INTEGER REFERENCES models(id),
  success INTEGER NOT NULL,
  error_id INTEGER REFERENCES errors(id),
  response_time INTEGER,
  tokens_generated INTEGER,
  total_tokens INTEGER,
  time_to_first_token INTEGER,
  PRIMARY KEY (run_id, model_id)
);

Benchmark parameters (default): temperature: 0.7 · top_p: 0.9 · max_tokens: 500 · OpenAI-compatible API. GLM-5.3 variants use reasoning_effort: low and max_tokens: 2048 because thinking is always enabled.

🤝 Contributing

Contributions are what make the open-source community amazing. Any contribution you make is greatly appreciated!

1. Fork the repository
2. Create your feature branch: git checkout -b feat/amazing-feature
3. Commit your changes: git commit -m 'feat: add amazing feature'
4. Push to the branch: git push origin feat/amazing-feature
5. Open a Pull Request

Ideas for contributions:

• 🆕 Add new NIM models to the benchmark list
• 📊 New chart types or dashboard widgets
• 🌐 Internationalization / translations
• 🐛 Bug fixes and performance improvements
• 📖 Improve documentation

Please read through open Issues before starting — someone might already be working on it!

🔗 Resources

• NVIDIA NIM API Documentation
• NVIDIA Model Catalog
• GitHub Actions Docs
• sql.js — SQLite in the browser

📄 License

Distributed under the MIT License. See LICENSE for details.