AI Frontier Database：每小时更新的 AI 模型对比仪表盘

An interactive dashboard mapping the entire AI model landscape — comparing every hosted language model Artificial Analysis benchmarks — 155 of them across 31 providers today — on cost, speed, and intelligence in one place, updated hourly.

🔗 Live dashboard → https://avo-sarkissian.github.io/AI-Frontier-Database/

Hosted free on GitHub Pages. The full interactive dashboard runs entirely in your browser — the same Python chart code is executed client-side via Pyodide (WebAssembly), so there is no server. Charts paint instantly from a pre-rendered snapshot; the live filters/calculators activate after a one-time background load (a few seconds, cached thereafter).

Overview

Every week, new AI models ship from OpenAI, Google, Anthropic, Meta, and dozens of others. Each makes different tradeoffs — some excel at reasoning but cost more, some are fast but less capable, and some punch well above their price point.

AI Frontier makes those tradeoffs visible. A GitHub Actions bot scrapes live benchmark data from Artificial Analysis every hour and surfaces the full competitive landscape through 10 interactive tabs — no sign-in, no API key required.

Dashboard Tabs

Screenshots

Captured from the built site by scripts/capture_screenshots.py, with the header stat bar hidden — those counters change hourly, and an image containing them is wrong within the hour. Regenerate after a UI change with: .venv/bin/python build_static.py && .venv/bin/python scripts/capture_screenshots.py

Data Sources

The LLM dataset currently covers 155 models across 31 providers. Counts move every hour as Artificial Analysis adds and delists models — the header stats on the live site are always authoritative, and this number was last written on a day the catalogue held 155. It has been as high as 329 (before AA pruned ~181 legacy models on 2026-07-24) and as low as 148. Four separate scrapers (data/scraper.py, data/image_scraper.py, data/local_scraper.py, data/video_scraper.py) each pull their own Artificial Analysis source on the same hourly cadence. All four read a rendered page rather than an API, so data/rsc.py holds the Next.js payload parsing they share. Artificial Analysis has retired or gated every JSON endpoint this project used: the image arena went key-gated, video never had one, the models API omitted unhosted open-weight models, and on 2026-08-20 /api/data/website/host-models/performance — the last one in use — began returning 404.

Live Data Refresh

An hourly GitHub Actions workflow (.github/workflows/refresh.yml) is the sole source of truth for the deployed site — browsers can't call the Artificial Analysis API directly (CORS), so a scheduled job does it server-side (four cron entries an hour — :08, :23, :38, :53 — plus manual workflow_dispatch):

1. Runs all four scrapers, falling back to the last-known-good cache per source if a fetch fails.
2. Sanity-checks row counts, per-column health, and per-column medians before proceeding — a rename that zeroes a column, or a units change that leaves the row count intact, both fail the build.
3. Skips the rebuild entirely if nothing changed (git-diff change guard).
4. Runs python build_static.py --data-only to swap the new CSVs into the static bundle without re-vendoring Plotly.
5. Commits and pushes the refreshed docs/ folder as github-actions[bot], which GitHub Pages auto-deploys.

The live site auto-loads the latest published snapshot on open.

GitHub does not honour the schedule. Since 2026-08-26 GitHub's schedule service has been dropping cron runs outright — no run is created, nothing fails — and this repo fell from 21–23 scheduled runs a day to 2–8, with gaps of up to 13 hours (community discussion #206019, #207346; unanswered by GitHub). Four cron entries did not help: on 2026-09-22 GitHub skipped seven consecutive slots. When a model has just been released and the site has not caught up, gh workflow run refresh.yml publishes it in about two minutes.

The reliable trigger is an external scheduler calling workflow_dispatch — the only workaround other affected users report working. Setup, once:

1. Create a fine-grained personal access token at https://github.com/settings/personal-access-tokens/new: repository access only this repository, repository permission Actions: Read and write (Metadata: read is added automatically), nothing else. Note the expiry — the site stops refreshing the day it lapses.
2. Smoke-test it from a terminal. HTTP 204 means a run was created; it shows up in the Actions tab as a workflow_dispatch run within seconds.

   GITHUB_TOKEN=github_pat_... scripts/dispatch_refresh.sh

3. Add a job at a free cron service such as cron-job.org: every hour (or every 30 minutes — the change guard makes a run with nothing new a 35-second no-op), method POST, URL https://api.github.com/repos/Avo-Sarkissian/AI-Frontier-Database/actions/workflows/refresh.yml/dispatches, headers Authorization: Bearer <token>, Accept: application/vnd.github+json and X-GitHub-Api-Version: 2022-11-28, body {"ref":"main"}. Run it once from the service and confirm a workflow_dispatch run appears.

The GitHub cron entries stay as a backstop, and the workflow's concurrency group serialises overlapping triggers.

The freshness badge reports data age, not build age. Each scraper records whether it actually fetched, when, and how many rows (data/raw/scrape_status.json), and the badge shows the oldest successful fetch across all four datasets — because a dashboard is only as fresh as its stalest panel. If any dataset is failing or older than twelve hours the badge turns amber and adds a ⚠; hover it for a per-dataset breakdown. Staleness is recomputed in the browser, so a page left open ages honestly even if the hourly job stops.

That badge used to read the build timestamp. On one real CI run two of three scrapers failed, only the image arena refreshed, and a five-hour-stale catalogue published under "Updated just now" — with the failure step running after the push.

Running Locally

Requires Python 3.11+.

# Clone the repo
git clone https://github.com/Avo-Sarkissian/AI-Frontier-Database.git
cd AI-Frontier-Database

# Create and activate a virtual environment.
# Use .venv, not a name containing spaces: this repo lives under a path with a
# space in it, and a venv whose activate script embeds that path silently
# prepends a directory that does not exist. The script still exits 0 and still
# sets the (name) prompt, so you get the SYSTEM python believing you are in a
# venv — which is how a build once ran under plotly 6.0 and tripled the bundle.
python -m venv .venv
source .venv/bin/activate         # Windows: .venv\Scripts\activate
python -c "import sys; print(sys.prefix)"   # must print .../AI Frontier/.venv

# Install dependencies. requirements.txt is the human-edited file;
# requirements.lock is what CI installs, with a sha256 for every artifact.
pip install -r requirements.txt

# Launch the dashboard
python app.py

Open http://localhost:8050. The scrapers fetch fresh data on startup and run in the background every hour — no manual refresh needed. (They are skipped under pytest, so the suite never touches the network or the committed cache.)

Tests

.venv/bin/python -m pytest -q        # 383 tests, ~30s

Run them from .venv, not the system Python: four full-build tests silently skip when the ambient plotly ships the pre-6.1 validator tree, so a green run outside the venv is not a green run.

The suite is mostly regression tests for defects an audit found, and each one names the failure it prevents rather than the function it calls — the docstrings are the useful documentation of why the code is shaped the way it is. Broadly:

Static site (GitHub Pages)

The public site at https://avo-sarkissian.github.io/AI-Frontier-Database/ is a static, server-free build of the same dashboard, served from the docs/ folder on main.

# Regenerate the static site into docs/ (pre-rendered figures + Pyodide bundle)
python build_static.py

# Or, after a data-only refresh (see Live Data Refresh above):
python build_static.py --data-only

git add docs && git commit -m "rebuild static site" && git push   # Pages auto-deploys (~1 min)

How it works: build_static.py pre-renders every chart's default state to JSON and zips the project's Python (chart builders, data loaders, static_api.py) plus plotly into docs/pycode.zip, and the data CSVs into docs/pydata.zip. The page loads the pre-rendered figures instantly, then boots Pyodide in the background and calls the same Python chart code in the browser for live filtering — so the static site is faithful to the Dash app with zero hosting cost. In normal operation the hourly GitHub Actions bot handles the refresh + rebuild + deploy automatically; the manual commands above are only needed for local iteration or a full rebuild.

Architecture

app.py                     # Dash app entry point, tab routing, background scrapers
components/
  charts/                  # One module per visualization; constants.py holds the
                           #   shared palette, escaping, and fixed-reference helpers
  stack_recommender.py     # Agent Stack tab recommendation logic
data/
  scraper.py               # Live LLM data — Artificial Analysis
  image_scraper.py         # Live image-gen data — AA Image Arena
  local_scraper.py         # Live open-weight model data — AA model leaderboard
  video_scraper.py         # Live video-gen data — AA Video Arena (both generation arenas)
  rsc.py                   # Next.js RSC payload parsing shared by the image + video scrapers
  ingest.py                # Parses scraper output, manages CSV cache and daily history
  scrape_status.py         # Per-dataset ok/fetched_at/rows — drives the freshness badge
  image_models.py, local_models.py, video_models.py   # Per-domain data access helpers
  raw/                     # Live cache, coverage, scrape status + daily snapshots
assets/style.css           # Global dark theme, typography (copied into docs/ by the build)
captions.py                # Every chart caption, read by app.py AND shipped in the manifest
static_helpers.py          # Pure dash-free helpers shared by app.py + the static build
static_api.py              # Browser bridge — Dash callbacks as JSON-returning functions (Pyodide)
build_static.py            # Pre-renders figures + bundles Python into docs/ for GitHub Pages
data_guard.py              # Row-loss, cumulative-drain and per-column median checks
docs/                      # The static GitHub Pages site (index.html, app.js, figures/, pycode.zip, pydata.zip)
scripts/
  capture_screenshots.py   # Regenerates the README screenshots from the built site
.github/workflows/refresh.yml   # Hourly bot: scrape → guard → rebuild → commit + push
tests/                     # 383 tests: data semantics, encoding calibration, control
                           #   behaviour, pipeline integrity, injection surfaces

Two renderings, one source

The Dash app and the static site run the same Python. Everything a chart needs — palettes, captions, filter vocabularies, numeric defaults, provider aliases — lives in one module and is either imported by both or shipped to the browser in docs/figures/manifest.json. That is not tidiness: the recurring defect in this codebase has been two copies of one fact drifting apart, so the rule is derive, never synchronise, and tests assert the two sides agree.

The deployed site executes the Python inside docs/pycode.zip, not the repo's .py files. The hourly bot only rebuilds docs/pydata.zip (the CSVs), so a code change needs a full build_static.py to reach visitors — tests/test_pybundle_freshness.py fails if the two ever diverge. The code zip is split from the data so the browser can keep it: the worker caches it in CacheStorage under manifest.code_version (a hash of its contents), and a returning visitor downloads its ~4 MB again only when the code changes, not on every hourly refresh.

Built With