跳到正文
GitHub·· 6 天前AI 评分38

LLMstats：每 3 小时自动评测 OpenRouter、Groq、Mistral 共 47 个免费模型

Saif658/LLMstats

AI 导读

LLMstats 每约 3 小时自动评测 47 个免费层模型，覆盖 OpenRouter 的 23 个、Groq 的 10 个和 Mistral Experiment 层的 14 个，结果发布到公开的交互式仪表盘。

正文

Real-time benchmarking of free-tier models from OpenRouter, Groq, and Mistral — automated roughly every 3 hours, zero infrastructure.

Live Dashboard Benchmark

🌐 saif658.github.io/LLMstats — live, public, refreshes roughly every 3 hours

Forked from the architecture of NIMStats, rebuilt to compare three OpenAI-compatible providers side-by-side.

✨ What is LLMstats?

LLMstats automatically benchmarks 47 free-tier models roughly every 3 hours — 23 served by OpenRouter (the unified router, 11 upstream providers), 10 by Groq (the LPU cloud), and 14 by Mistral (Experiment tier) — and publishes the results to a live, interactive dashboard. No servers, no subscriptions — just fork the repo, drop in three API keys, and go.

🏎️ Every ~3h 📊 Interactive Charts 🔁 Zero Infra 🔀 Multi-Provider
GitHub Actions cron Speed, throughput, reliability Static site + free CI/CD OpenRouter ↔ Groq ↔ Mistral comparisons

⚡ Quick Start

A working dashboard in under 5 minutes.

1. Fork & clone

git clone https://github.com/Saif658/LLMstats.git
cd LLMstats

2. Get free API keys

Provider Where to get it
OpenRouter openrouter.ai/keys — sign up, generate a key
Groq console.groq.com/keys — sign up, generate a key
Mistral console.mistral.ai/api-keys — sign up, generate a key (La Plateforme / Experiment tier)

3. Add secrets

In your forked repo: Settings → Secrets and variables → Actions → New repository secret

Name Value
OPENROUTER_API_KEY Your OpenRouter API key
GROQ_API_KEY Your Groq API key
MISTRAL_API_KEY Your Mistral API key

4. Enable GitHub Pages

Settings → Pages → Build from "GitHub Actions".

5. Run your first benchmark

Actions → Benchmark OpenRouter + Groq + Mistral Models → Run workflow.

The dashboard auto-refreshes roughly every 3 hours after that. ✨

📊 Dashboard

Tab What you get
📊 Overview 5 KPI cards · success-rate trend · top-10 speed & throughput bars · per-model reliability pills
🏆 Leaderboard Composite score · sortable columns · trend indicators · provider chips (OpenRouter vs Groq vs Mistral)
🔬 Explorer Per-model deep dive · response time history · error breakdown · availability heatmap
⏱ Timeline Filterable run history · expandable run cards with full per-model detail
⚔️ Compare Head-to-head overlay · win-rate stats · side-by-side metric comparison

🤖 Benchmarked models (free tier)

47 models across 3 gateways, 14+ upstream providers

🟣 OpenRouter · 23 models · 11 upstream providers

Upstream Model Note
OpenRouter openrouter/free OR's own free flagship
OpenRouter openrouter/owl-alpha OR preview alpha
OpenAI openai/gpt-oss-120b:free Open-source 120B
OpenAI openai/gpt-oss-20b:free Open-source 20B
Meta meta-llama/llama-3.3-70b-instruct:free Large general-purpose
Meta meta-llama/llama-3.2-3b-instruct:free Lightweight Llama
NVIDIA nvidia/nemotron-3-ultra-550b-a55b:free 550B MoE flagship
NVIDIA nvidia/nemotron-3-super-120b-a12b:free 120B MoE super
NVIDIA nvidia/nemotron-3-nano-30b-a3b:free Compact MoE
NVIDIA nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free Compact reasoning
NVIDIA nvidia/nemotron-nano-12b-v2-vl:free Vision-language
NVIDIA nvidia/nemotron-nano-9b-v2:free Compact 9B
NVIDIA nvidia/nemotron-3.5-content-safety:free Content safety guard
Qwen qwen/qwen3-coder:free Qwen 3 coding
Qwen qwen/qwen3-next-80b-a3b-instruct:free Qwen 3 80B MoE
Google google/gemma-4-26b-a4b-it:free Gemma 4 mid-size
Nous nousresearch/hermes-3-llama-3.1-405b:free Hermes 3 405B
Nex AGI nex-agi/nex-n2-pro:free Nex N2 Pro
Poolside poolside/laguna-m.1:free Laguna m.1
Poolside poolside/laguna-xs.2:free Laguna xs.2
Cohere cohere/north-mini-code:free Cohere North mini-code
Liquid liquid/lfm-2.5-1.2b-instruct:free Liquid LFM 2.5 1.2B
Liquid liquid/lfm-2.5-1.2b-thinking:free Liquid LFM 2.5 1.2B thinking
Cognitive Computations cognitivecomputations/dolphin-mistral-24b-venice-edition:free Dolphin Mistral 24B Venice

🟠 Groq · 10 chat models

Model Note
llama-3.1-8b-instant Ultra-low latency
meta-llama/llama-4-scout-17b-16e-instruct Llama 4 Scout MoE
qwen/qwen3-32b Qwen 3 flagship
qwen/qwen3.6-27b Qwen 3.6 mid-size
openai/gpt-oss-120b Open-source 120B
openai/gpt-oss-20b Open-source 20B
openai/gpt-oss-safeguard-20b Safety-guard 20B
groq/compound Groq compound agentic
groq/compound-mini Groq compound-mini agentic
allam-2-7b Allam (small MoE)

🔵 Mistral · 14 chat models (Experiment tier)

Model Note
mistral-large-latest Mistral Large flagship
mistral-medium-latest Mistral Medium
mistral-small-latest Mistral Small
mistral-tiny-latest Mistral Tiny
ministral-8b-latest Ministral 8B
ministral-3b-latest Ministral 3B
magistral-medium-latest Magistral medium
magistral-small-latest Magistral small
codestral-latest Code-specialized
devstral-latest Developer agent
devstral-small-latest Developer agent small
open-mistral-7b Open 7B
open-mixtral-8x7b Open MoE 8×7B
open-mixtral-8x22b Open MoE 8×22B

🏗️ How it works

┌─── GitHub Actions (every ~3h) ────────────────────────────────────────────┐
│                                                                          │
│   ┌─────────────────────┐         ┌─────────────────────┐                │
│   │  Job 1 — Group A    │         │  Job 2 — Group B    │  in parallel    │
│   │  25 models (mixed)  │         │  22 models (mixed)  │                │
│   └──────────┬──────────┘         └──────────┬──────────┘                │
│              └──────────────┬───────────────┘                           │
│                     ┌────────▼────────┐                                 │
│                     │  Merge + commit │  → benchmark/history.db + docs/history.db │
│                     └─────────────────┘                                 │
└─────────────────────────────────────────────────────────────────────────┘
                                      │
                          ┌───────────▼──────────┐
                          │  Static Dashboard    │  GitHub Pages
                          │  (sql.js client)     │
                          └──────────────────────┘

Parallel matrix ≈ 50% faster ⚡

🛠️ Customization

Change the prompt

Edit PROMPT in benchmark/test_models.py:

PROMPT = "Your custom prompt here"
 Add or remove models

Edit OPENROUTER_FREE_MODELS, GROQ_MODELS, or MISTRAL_MODELS in benchmark/test_models.py. The lists get split roughly in half between group1 and group2 jobs.

 Change the schedule

Edit .github/workflows/benchmark.yml:

- cron: '0 */3 * * *'   # every 3 hours (default)
- cron: '0 */1 * * *'   # every hour
- cron: '0 0 */6 * *'   # every 6 hours
 Run locally 
# Serve the dashboard from the repo root (serves docs/index.html + docs/history.db)
python3 -m http.server 8000
# Open http://localhost:8000/docs/

# Run benchmarks manually (requires API keys as env vars)
export OPENROUTER_API_KEY=...
export GROQ_API_KEY=...
export MISTRAL_API_KEY=...
python3 benchmark/test_models.py            # all models
MODEL_GROUP=group1 python3 benchmark/test_models.py   # one half

📦 Data storage

benchmark/history.db is a SQLite database persisted in the repo. The browser loads it via sql.js (WebAssembly) and queries it entirely client-side. benchmark/results*.json are temporary per-job artifacts that never get committed (.gitignored). On every merge the workflow mirrors the latest benchmark/history.db to docs/history.db next to the dashboard.

Schema:

runs          (id, timestamp, prompt, success_count, total_models, fastest_model, fastest_time)
model_results (run_id, model, provider, success, error, response_time, tokens_generated, total_tokens, response)

Results are pruned to the most recent 720 runs to keep history.db from growing unbounded.

Benchmark parameters: temperature: 0.7 · top_p: 0.9 · max_tokens: 500 · OpenAI-compatible chat completions API.

🤝 Contributing

PRs and ideas welcome:

  • 🆕 Add new free-tier models from each provider
  • 📊 New chart types or dashboard widgets
  • 🐛 Bug fixes and perf improvements
  • 📖 Docs / translations

Inspired by NIMStats · ⚡ Powered by GitHub Actions

来源：GitHub · github.com

OpenRouter开源生态部署工程评测基准#开源/仓库#评测/基准#部署/工程