OpenRouter 如何对 LLM 做基准测试：用代码判分与成本核算选模型

Choose models using evidence from your application’s inputs and output formats. Run identical tasks through OpenRouter and compare graded accuracy, billed cost, and request time. On our 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats under the tested contracts and settings.

TL;DR

• Code checks measure correctness and output compliance.

• Sol passed 180/180 trials on our synthetic set; DeepSeek had the lowest response cost, with 157/180 passes.

• Agreement with the public ranking was weak, with a Spearman correlation of 0.109.

• Before you trust a score, check how many times each task ran, how many answers came back, and whether each bill was verified.

Define the behavior first, then its checks.

Step 1. Write tasks with answers a script can check

A task needs an input, expected answer, and output contract. For invoices, specify included rows, rounding, and acceptable currency suffixes.

We wrote all 40 baseline tasks and 20 hard tasks before the final run.

Compute references independently. Separate prompt-tuning cases from final evaluation. Refund cases test written decisions, without executing transactions.

Step 2. Grade with code, and check the grader

Use exact matching for APPROVE, and schema/value checks for JSON. Normalize only permitted differences. Accepting prose around a number hides a failure when your consumer requires only a number.

Test known-correct, wrong, and malformed responses. For a refund threshold, include exactly $200 and 30 days, then values just above each boundary.

Our development checker rejected valid Python because it omitted ord and type. We corrected it before the final study, specifying CPython 3.12.10/WASI with resource limits. Otherwise, checker failures would have counted against models.

The full-study runner isolates generated code and stops if sandbox setup is missing. The teaching example executes no generated code.

Step 3. Run the three-task Python example

Download the companion ZIP. Before extracting it, compare its SHA-256 checksum with:

a38ee17a2eb7090f9f4849a605cc8214691c889ca0ae40c121259603592178e8

PowerShell:

Get-FileHash .\benchmark-reader-companion.zip -Algorithm SHA256

Linux or macOS:

# Linux
sha256sum benchmark-reader-companion.zip
# macOS
shasum -a 256 benchmark-reader-companion.zip

If the values differ, stop and check the download. After they match, extract the ZIP. You need Python 3.12 or later. The standard-library teaching runner, JSON example, and comparisons passed offline checks on Windows x64 and Linux. macOS remains untested. The bundled full-study sandbox and native Ori setup support Windows x64 only.

From the extracted folder:

python verify_package.py
python three_tasks.py check

The check prints OFFLINE reference and grader checks passed; zero inference calls. The complete script follows in file order. Its companion module handles requests, persistence, and reporting.

"""Three exact contracts; run from the extracted companion (standard library)."""
import argparse
from datetime import datetime, timezone
from decimal import Decimal
import json
from pathlib import Path
import re

from reader_core import execute, reconcile, report, read, load_config, finish_grade

TASKS = [
    {"id": "invoice", "prompt": "Two unpaid lines cost 3 x 12.50 USD and 2 x 4.25 USD. Return only their total with exactly two decimal places.", "expected": "46.00"},
    {"id": "utc", "prompt": "Convert 2026-10-18T10:30:00+02:00 to UTC. Return only YYYY-MM-DDTHH:MM:SSZ.", "expected": "2026-10-18T08:30:00Z"},
    {"id": "refund", "prompt": "Policy: approve only if amount <= 200 USD and age <= 30 days. Amount=200 USD; age=30 days. Return only APPROVE or DENY.", "expected": "APPROVE"},
]

def refund_policy(amount, days):
    return "APPROVE" if Decimal(str(amount)) <= 200 and days <= 30 else "DENY"

def grade(task, text, finish):
    terminal = finish_grade(finish)
    if terminal:
        return terminal
    if not isinstance(text, str) or not text.strip():
        return {"passed": False, "classification": "empty"}
    text = text.strip()
    patterns = {"invoice": r"\d+\.\d{2}",
                "utc": r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z",
                "refund": r"APPROVE|DENY"}
    valid = bool(re.fullmatch(patterns.get(task["id"], r"\d+\.\d{2}"), text))
    passed = valid and text == task["expected"] and finish == "stop"
    return {"passed": passed, "classification": "pass" if passed else
            ("incorrect" if valid else "malformed")}

Before using a score, check its reference and rejection rules. These offline fixtures cover boundary decisions, invalid formats, and truncated answers without calling a model.

def offline():
    money = Decimal(3) * Decimal("12.50") + Decimal(2) * Decimal("4.25")
    assert format(money, ".2f") == TASKS[0]["expected"]
    utc = datetime.fromisoformat("2026-10-18T10:30:00+02:00")
    assert utc.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") == TASKS[1]["expected"]
    assert refund_policy(200, 30) == TASKS[2]["expected"]
    for amount, days, expected in [(200.01, 30, "DENY"), (200, 31, "DENY"),
                                   (199.99, 29, "APPROVE"), (0, 0, "APPROVE")]:
        assert refund_policy(amount, days) == expected
    for task in TASKS:
        assert grade(task, task["expected"], "stop")["passed"]
        for bad in ("wrong", None, "", '{"answer": 46}', "APPROVE please"):
            assert not grade(task, bad, "stop")["passed"]
        assert not grade(task, task["expected"], "length")["passed"]
    assert not grade(TASKS[0], "47.00", "stop")["passed"]
    assert not grade(TASKS[2], "DENY", "stop")["passed"]
    print("OFFLINE reference and grader checks passed; zero inference calls.")

Keep checking separate from spending. The command interface defaults to offline validation and requires an explicit live option and allowance before inference.

def main(tasks=None, grader=grade, default_config="demo-config.json", offline_check=None):
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("operation", nargs="?", default="check",
                        choices=["check", "plan", "run", "report", "reconcile"])
    parser.add_argument("--config", default=default_config)
    parser.add_argument("--run-dir", type=Path, default=Path("runs/demo"))
    parser.add_argument("--live", action="store_true")
    parser.add_argument("--max-usd", default="0")
    parser.add_argument("--resume", action="store_true")
    parser.add_argument("--tasks", help="Optional exact-contract custom task JSON")
    args = parser.parse_args()
    if args.operation == "check":
        if args.tasks:
            parser.error("Use an example entry point with its own offline_check; check does not accept --tasks overrides")
        if offline_check is not None:
            offline_check()
        elif tasks is not None or grader is not grade or args.tasks:
            parser.error("Custom tasks/graders require offline_check=... for their own reference and fixture checks")
        else:
            offline()
        return
    selected = read(args.tasks) if args.tasks else (tasks if tasks is not None else TASKS)
    config = load_config(args.config)
    if args.operation == "report":
        print(json.dumps(report(args.run_dir), indent=2))
    elif args.operation == "reconcile":
        reconcile(args.run_dir, live=args.live)
        print(json.dumps(report(args.run_dir), indent=2))
    else:
        execute(args.run_dir, selected, config, grader, args.max_usd,
                live=args.live, resume=args.resume, plan=args.operation == "plan")

if __name__ == "__main__":
    main()

Configure and inspect the plan

Use the supplied Luna configuration to set the model, repeat count, token allowance and provider constraints:

{
  "models": [
    "openai/gpt-6-luna"
  ],
  "repeats": 1,
  "max_tokens": 1024,
  "settings": {
    "openai/gpt-6-luna": {
      "provider": {
        "require_parameters": true,
        "allow_fallbacks": true,
        "max_price": {
          "prompt": 0.09999999999999999,
          "completion": 0.5,
          "request": 0
        }
      },
      "reasoning": {
        "effort": "low"
      }
    }
  }
}

Sol’s configuration preserves the repeat/token settings but uses its own price ceilings:

{
  "models": [
    "openai/gpt-6.1-sol"
  ],
  "repeats": 1,
  "max_tokens": 1024,
  "settings": {
    "openai/gpt-6.1-sol": {
      "provider": {
        "require_parameters": true,
        "allow_fallbacks": true,
        "max_price": {
          "prompt": 2.0,
          "completion": 10.0,
          "request": 0
        }
      },
      "reasoning": {
        "effort": "low"
      }
    }
  }
}

models selects candidates, repeats specifies requests per task, and max_tokens sets the completion allowance. Reasoning can consume that allowance before visible output. Offline checks make no API calls. Paid runs require the explicit —live flag. Length-truncated responses fail without retry.

Price ceilings use USD per million tokens; Luna’s prompt ceiling is approximately $0.10. If a provider no longer supports the saved settings, the runner must stop rather than silently change them. Provider fallback doesn’t swap models or resend inference.

python three_tasks.py plan --config luna-config.json

Actual offline output:

{"planned_calls": 3, "max_tokens": 1024, "models": ["openai/gpt-6-luna"], "repeats": 1, "estimated_reservations_usd": "0.0023543749999999999565625"}

A reservation is a local cost estimate, not an account hold or guaranteed cap. Plan before choosing an allowance.

Changing tasks, repeats, or tokens changes the estimate. If spending goes over the estimate, the script stops sending new request.

Make a small paid run

Create a key in API-key settings, add credit, and configure a suitable key limit. Set the variable locally in your terminal.

PowerShell:

$env:OPENROUTER_API_KEY = "YOUR_OPENROUTER_API_KEY"

Bash or Zsh:

export OPENROUTER_API_KEY="YOUR_OPENROUTER_API_KEY"

Reuse an existing variable if configured. The package doesn’t load .env. For the shipped three-task Luna configuration:

python three_tasks.py run --config luna-config.json --run-dir runs/my-luna --live --max-usd 0.01
python three_tasks.py reconcile --run-dir runs/my-luna --live
python three_tasks.py report --run-dir runs/my-luna

Reconciliation performs read-only generation metadata lookups. Reports show response coverage, meaning completed responses divided by planned trials, plus passes/graded, failures, verified cost, and median request time. Until a bill is verified, the report keeps its cost unresolved. Request time excludes grading and metadata; it’s not time to first token.

Live results vary. Synthetic examples below illustrate reporting, not model measurements.

Step 4. Add your own task and grader

The supplied custom-tasks.json shows the task format:

[
  {
    "id": "custom-total",
    "prompt": "Four items cost 12.50 USD each. Return only the total with exactly two decimals.",
    "expected": "50.00"
  }
]

python custom_task.py check independently computes its reference with Decimal and tests correct, wrong, and malformed answers. Custom money IDs use the two-decimal grader. Another contract needs its own grader and checks.

The complete json_extraction.py follows in three parts, in file order. It permits JSON whitespace and key-order differences, but rejects missing, extra, or duplicate keys and wrong types. A schema-valid object with a wrong value is incorrect. Filtered or abnormal finishes have separate classifications.

"""A complete exact-schema JSON extraction example; offline check by default."""
import json

from reader_core import finish_grade
from three_tasks import main

CUSTOMER_EMAIL = "mira@example.test"
ORDER_CODE = "A-104"
ITEM_COUNT = 3
SOURCE_TEXT = ("Customer: Mira. Email: mira@example.test. Order: A-104. "
               "Items: 3. Gift wrap: no.")

# Define reference fields from the independently specified source facts.
# Do not ask a model (or the grader) to produce the reference.
EXPECTED = {"email": CUSTOMER_EMAIL, "order_id": ORDER_CODE,
            "item_count": ITEM_COUNT, "gift_wrap": False}
TASKS = [{"id": "order-extraction", "grader": "exact-order-json-v1",
          "prompt": "Extract email, order_id, item_count and gift_wrap from this order. "
                    "Return a JSON object only, with exactly those keys. email and order_id "
                    "are strings; item_count is an integer; gift_wrap is a boolean. " + SOURCE_TEXT,
          "expected": EXPECTED}]

JSON syntax alone is insufficient. Reject duplicate keys and invalid number values, then validate the schema before comparing fields with the reference.

def strict_json(text):
    def pairs(items):
        result = {}
        for key, value in items:
            if key in result:
                raise ValueError("Duplicate JSON key")
            result[key] = value
        return result
    def nonfinite(value):
        raise ValueError("Nonfinite JSON number")
    return json.loads(text, object_pairs_hook=pairs, parse_constant=nonfinite)

def grade(task, text, finish):
    terminal = finish_grade(finish)
    if terminal:
        return terminal
    if not isinstance(text, str) or not text.strip():
        return {"passed": False, "classification": "empty"}
    try:
        actual = strict_json(text)
    except (ValueError, TypeError):
        return {"passed": False, "classification": "malformed"}
    types = {"email": str, "order_id": str, "item_count": int, "gift_wrap": bool}
    schema_ok = (type(actual) is dict and actual.keys() == types.keys()
                 and all(type(actual[key]) is kind for key, kind in types.items()))
    if not schema_ok:
        return {"passed": False, "classification": "malformed"}
    passed = actual == task["expected"]
    return {"passed": passed, "classification": "pass" if passed else "incorrect"}

Check both acceptance and rejection. These fixtures distinguish valid but wrong values from malformed output, and verify that filtering or truncation can’t count as a pass.

CORRECT = '{"order_id":"A-104","email":"mira@example.test","gift_wrap":false,"item_count":3}'
WRONG = '{"email":"other@example.test","order_id":"A-104","item_count":3,"gift_wrap":false}'
MALFORMED = ["not JSON", '```json\n' + CORRECT + '\n```',
             '{"email":"mira@example.test"}', CORRECT.replace('3}', 'true}'),
             CORRECT.replace('3}', '3.0}'), CORRECT.replace('3}', 'NaN}'),
             CORRECT[:-1] + ',"item_count":3}', CORRECT[:-1] + ',"extra":1}']

def offline():
    assert EXPECTED == {"email": "mira@example.test", "order_id": "A-104",
                        "item_count": 3, "gift_wrap": False}
    assert f"Email: {CUSTOMER_EMAIL}." in SOURCE_TEXT
    assert f"Order: {ORDER_CODE}." in SOURCE_TEXT
    assert f"Items: {ITEM_COUNT}." in SOURCE_TEXT and "Gift wrap: no." in SOURCE_TEXT
    task = TASKS[0]
    assert grade(task, CORRECT, "stop")["passed"]
    assert grade(task, json.dumps(EXPECTED, indent=2), "stop")["passed"]
    assert grade(task, WRONG, "stop")["classification"] == "incorrect"
    for text in MALFORMED:
        assert grade(task, text, "stop")["classification"] == "malformed", text
    assert grade(task, "", "stop")["classification"] == "empty"
    assert grade(task, CORRECT, "content_filter")["classification"] == "content_filtered"
    assert grade(task, CORRECT, "length")["classification"] == "truncated"
    print("OFFLINE JSON extraction: independent source facts, schema, correct/wrong/malformed fixtures passed; zero requests.")

if __name__ == "__main__":
    main(tasks=TASKS, grader=grade, offline_check=offline)

The final main(…) call connects tasks, grading, and reference checks to the runner. Adapt the source facts and expected fields, then run:

python json_extraction.py check
python json_extraction.py plan --config luna-config.json

Expected check output:

OFFLINE JSON extraction: independent source facts, schema, correct/wrong/malformed fixtures passed; zero requests.

Use separate folders. These allowances apply only to the shipped single-case plans.

python json_extraction.py run --config luna-config.json --run-dir runs/json-luna --live --max-usd 0.01
python json_extraction.py run --config sol-config.json --run-dir runs/json-sol --live --max-usd 0.03
python json_extraction.py reconcile --run-dir runs/json-luna --live
python json_extraction.py reconcile --run-dir runs/json-sol --live

Set equal repeat counts before starting. Three repeats of one case test variability, not workload coverage. Add representative boundaries and held-out cases before drawing broader conclusions.

Step 5. Compare saved runs and handle missing answers

For your saved JSON runs:

python compare_runs.py runs/json-luna runs/json-sol --output comparisons/json

This writes comparison.md and comparison.json without inference or regrading. It validates contracts, settings, and identities; incompatible runs require investigation rather than a silent ranking.

First rehearse it offline:

python synthetic_comparison.py runs/comparison-demo
python compare_runs.py runs/comparison-demo/luna runs/comparison-demo/sol --output comparisons/full

The table includes these synthetic fixture values, shown with abbreviated columns:

Sol’s fixture has a missing answer and an unverified bill. Its 5/5 score lacks complete coverage. These figures measure neither model.

Keep an uncertain trial and its reservation. Fix access or rate-limit problems, then resume with unchanged configuration, source, and allowance:

python json_extraction.py run --config luna-config.json --run-dir runs/json-luna --live --max-usd 0.01 --resume

Resume continues only never-attempted trials. A known bill can’t recover missing text. If we can’t find a generation record for a request, the report keeps it marked as unresolved. Don’t discard the run or delete attempts. The README covers stale locks after crashes.

To compare cases answered by every participant:

python compare_runs.py runs/comparison-demo/luna runs/comparison-demo/sol --common-completed --output comparisons/common

The explicit intersection retains original coverage and lists exclusions. It can select easier cases and doesn’t repair missing bills. Never compare different subsets without disclosure.

The optional 60-task runner and runtime setup are in the companion. Its historical results provide a larger worked example.

Which model did best on our benchmark?

Each model received 40 baseline and 20 hard tasks three times, giving 120 baseline and 60 hard grades. These results describe the 60 synthetic tasks under our fixed settings, completed on 20 October 2026. AA denotes the Artificial Analysis Intelligence Index from our dated public snapshot.

Cost per 100 task responses is the verified cost of the 180 scored answers, scaled to 100. It excludes operational overhead and unresolved bills. Median seconds measures local request elapsed time.

Three findings guide the choice on this set.

1. Highest pass rate. Sol passed 180/180 at about $0.2449 per 100 responses.

2. Lowest response cost. DeepSeek cost about $0.00394 per 100, passing 157/180 with 14 truncations. Luna passed 162/180 at about $0.01848. Choose against your required pass rate and failure consequences.

3. Output compliance affects the score. Sonnet’s 66 malformed responses cover 21 invoice, 30 date, and 15 refund trials. Raw answers include explanations where prompts required only a number or JSON. The score measures contract compliance as well as answer correctness, not pure reasoning ability.

Each model row totals 180. The final row totals all 1,800 responses. Categories record failed checks, not their underlying causes.

All 53 truncated responses remained failures.

The study requested 8,192 tokens and low reasoning where supported. Sixteen trials needed replacement requests because their earlier attempts returned no answer. One replacement also returned no answer and required another dispatch, giving 17 replacement dispatches for 16 selected answers. No returned answer was retried. The companion preserves every attempt.

Known study bills were $5.1032, plus $0.5383 reserved for unresolved bills. Reader plans estimate $0.97 for Luna’s 180 study trials and $19.46 for Sol’s. Routing, dates, and original concurrent/resumed scheduling limit comparisons with the serial reader runner.

How do public LLM benchmarks compare with your own?

We retrieved an Artificial Analysis snapshot through our benchmarks API on 20 October 2026 at 10:03 UTC, matching exact canonical model versions.

Spearman correlation compares ranking order, from −1 for reversed rankings to +1 for matching rankings. Across ten models:

Agreement was weak in this sample. Different tasks and settings prevent conclusions about general benchmark reliability. Use public scores for shortlisting, then evaluate your workload.

How many times should you run each task?

In 68 of 600 model-task pairs, the model passed some repeats and failed others. Retain every repeat. These are repeated observations of 60 cases, not 180 distinct cases per model. Close rankings need more held-out cases and repeats.

FAQ

How many test cases do I need?

Start with a small set covering one workflow’s common inputs, boundaries, and known failures. Ten carefully reviewed examples can support initial exploration, but they don’t provide a statistical guarantee. Expand with representative held-out cases when differences are close or failures costly. Repeats assess consistency; they don’t replace coverage.

Can I use an LLM judge?

Yes, for properties such as tone or explanation quality that resist exact checks. Validate its rubric against human-labeled examples and inspect disagreements. Keep code checks for calculable answers and required schemas. Our LLM-as-a-judge guide explains that approach.

Next steps

Build a representative set from your application using our golden eval dataset guide. Compare candidates, then gate pull requests on the checks.