Plune 发布 @plune-ai/cli：面向 LLM 应用的断言测试运行器

AI-powered assertion testing for LLM apps — a test runner for model behaviour.

Plune runs an assertion suite against an LLM provider and gives you a pass/fail report — locally, in CI, or as a regression diff between two runs. You describe the checks in one plune.yaml; Plune calls the model, evaluates each assertion, caches results, and reports token cost. Ten built-in assertion types cover plain text, JSON-schema, LLM-as-judge, and RAG metrics (faithfulness, answer-relevance, context-precision).

Links: plune.ai | npm | GitHub Action on Marketplace

Install

npm install -g @plune-ai/cli      # or: pnpm add -g @plune-ai/cli
plune --version

Or run it without installing:

npx -y @plune-ai/cli run

Requires Node.js ≥ 20.

Quickstart

# 1. Scaffold plune.yaml, an example dataset, and .env.example
plune init

# 2. Add your provider key (read from the environment / .env — never written to disk)
echo 'ANTHROPIC_API_KEY=sk-ant-...' >> .env

# 3. Run the assertions
plune run
# → 1/1 passed · 0 failed · 0 errored · $0.0008

# 4. Re-render the last run, or diff two runs to catch regressions
plune report --format markdown
plune diff baseline.json current.json --fail-on-regression

Each run writes its full result to .plune/last-run.json.

Configuration

Plune reads a single plune.yaml, discovered by walking up from the working directory (or passed with -c <path>). A minimal example:

version: 1
provider:
  type: anthropic            # anthropic | openai | openrouter
  model: claude-sonnet-5-5
evals:
  - id: example
    prompt: "Answer concisely. {{question}}"   # {{vars}} come from each dataset row
    dataset: datasets/example.jsonl            # a file path, or an inline `examples:` list
    assertions:
      - type: contains
        value: "Paris"

Datasets are JSONL, one row per line, shaped { "vars": { ... }, "expected"?: "..." }. The provider API key is read from the environment based on provider.type:

model is any id the provider serves; plune init starts you on claude-sonnet-5-5. A run's cost is priced from a built-in table of current models — a model it does not list costs 0, with a warning, until you add its rates under pricing: (input_per_1k_usd, output_per_1k_usd). Leave temperature out for Claude models released after Opus 4.6: they take none, and the API answers 400 to any value but the default. Plune sends no temperature unless the config sets one.

Where the CLI reads its settings

Every command reads them the same way: from the environment first — what your shell or CI job exported — then from a .env beside the config you named with -c, then from a .env in the current directory. The first place that sets a variable wins, so a .env never overrides what is already exported. That covers the provider keys above and every PLUNE_* variable (PLUNE_TOKEN, PLUNE_API_URL, PLUNE_RUN, …) for run, report, sync, ingest, login and run import, start and finish alike — no dotenv wrapper in your scripts. The token plune login saved is the last resort, used only when PLUNE_TOKEN is set nowhere else. Nothing writes a .env for you; plune init writes .env.example.

Assertion types

Commands

Global flags: -c, --config <path> · -v, --verbose · --no-color.

Exit codes: 0 everything passed · 1 an assertion failed · 2 configuration or execution error · 3 pull refused to overwrite uncommitted changes · 4 push had refusals.

Already running tests? Bring the results in

Everything above generates checks, which is why it needs a provider key. If you already have a suite, there is nothing to generate — the results exist and only have to arrive, and that route costs nothing beyond a Plune token.

plune login                           # once, on your machine; in CI export PLUNE_TOKEN instead
plune run import ./junit.xml          # Jest, Vitest, pytest, PHPUnit, Surefire, Cypress, …
plune run import ./playwright.json    # or Playwright's own JSON report
plune run import ./allure-results     # or the folder an Allure adapter wrote — see below

The format is read from the file, not from its name. Statuses go over as the report wrote them and Plune maps them per project, so error can mean something different to your team than to ours. A test Plune has no case for is counted, and --create offers it to the review queue instead — nothing becomes a test case until a person approves it.

For a Playwright suite there is also @plune-ai/playwright, which reports as the run happens and needs no second step:

// playwright.config.ts
reporter: [['list'], ['@plune-ai/playwright']],

Either way a Playwright suite brings its screenshots: every image an attempt kept as a file — PNG, JPEG or WebP, up to 2 MB each, 20 per result — is uploaded to that result, a passed test's too, so a run in Plune shows each test with what it looked at. plune run import finds them where the report says they are, so run it where the report was written; the summary counts any it could not find.

JSON and plain text come the same way — the answer of an API, a log — whether a test attached them with a path or as a body (a report holds a body itself, so plune run import needs no file for it). Only application/json and text/plain, as the attachment names them; UTF-8 without NUL bytes, up to 512 KiB each and 10 per result. The summary counts them apart from the screenshots.

A project that has connected its own storage bucket in Plune takes more. Every file an attempt kept — screenshots, text, a Playwright trace, a video — goes straight from this machine into that bucket over short-lived links Plune hands out per file, so a trace of up to 100 MB is streamed from disk, never read into memory. Up to four files go at once. The limits there are larger: 10 MiB for an image, 5 MiB for a text, 50 MiB for any other file and 100 MiB for a video or a trace, 30 files a result. The links are write keys to your bucket, so they are never printed — not in the log, not with --verbose, not in an error; a line names the file and a code. The first time a run finds the bucket it says so, and the summary reads N file(s) uploaded to your bucket, with the number skipped or failed when there are any. A file that does not arrive is a warning, never a failed import. Without a bucket, screenshots and text go as before, and a trace or a video is not stored — the log says so once and the summary counts them.

Files from a JUnit report

JUnit has no field for a file, so the runners that keep one print a line of their own in the test's output, as Jenkins and GitLab read it: [[ATTACHMENT|path]] on a line by itself, in system-out or system-err of the test. plune run import reads that line in every test case and sends the file the way the Playwright reporter sends one: into your bucket if the project has one, to the result if it is a PNG, JPEG, WebP, JSON or plain text; any other file (a trace, a video) needs the bucket and is counted as not stored without one. Any extension is read. A relative path counts from the folder of the report, so run the import where the report and its files were written.

pytest (junit_logging=system-out is what puts the output into the report):

def test_checkout(page):
    page.screenshot(path="artifacts/checkout.png")
    print("[[ATTACHMENT|artifacts/checkout.png]]")

pytest --junitxml=report.xml -o junit_logging=system-out
plune run import report.xml

Java, JUnit 5 (Maven Surefire writes the test's output into its testcase):

@Test
void checkout() {
    // ...
    System.out.println("[[ATTACHMENT|target/artifacts/checkout.png]]");
}

A file that is not there (a cleaned-up workspace, a typo, a screenshot that was never taken) does not fail the import: every result still goes to Plune. Each one gets a line, always, not only with --verbose, and the end of the output counts them:

plune: attachment not found — "shop.Cart › rejects a negative quantity": artifacts/missing-trace.zip
Imported 120 results · 34 files uploaded · 2 files not found

A CI step that should go red for it passes --fail-on-missing-attachments: exit code 1, with the results already sent. A path that leaves the folder of the report (../../etc/passwd, an absolute path elsewhere, a link that points outside) is refused with the same line, and the file is never read. Console output kept per suite rather than per test (jest-junit) names no test, so nothing is read from it. For Jest, use Allure below.

An allure-results folder

Every Allure adapter (pytest, JUnit 5, TestNG, Playwright, Cypress, Jest, Vitest, WebdriverIO, .NET) writes the same folder: a *-result.json per test, a *-container.json per suite, and the files the tests attached. Point plune run import at the folder (a folder is read as Allure, no flag needed). Each test arrives with its name, its status as Allure shows it (passed, failed, broken, skipped), its time, the failure message and trace, and its steps; a test run again (same historyId) is one test with attempts, the last one counting. The files come from the test, from its steps and from its fixtures (befores, afters), each with the type Allure wrote, and go the way a JUnit file goes: your bucket, or the result for a picture or a text. A source counts from the folder, and one that leaves it is refused. A missing file, --fail-on-missing-attachments and the summary line work as for JUnit above.

pytest (allure-pytest):

pip install allure-pytest
pytest --alluredir=allure-results
plune run import allure-results

import allure

def test_checkout(page):
    with allure.step("pay"):
        allure.attach.file("artifacts/checkout.png", name="checkout", attachment_type=allure.attachment_type.PNG)

Java, JUnit 5 (allure-junit5, with the AspectJ agent from Allure's setup; the results folder is target/allure-results unless allure.results.directory says otherwise):

mvn test
plune run import target/allure-results

@Test
void checkout() {
    Allure.addAttachment("checkout", "image/png", new FileInputStream("target/artifacts/checkout.png"), ".png");
}

Jest has no per-test output in JUnit; allure-jest gives it one (allure.attachment(...) in a test or a step). Tests are matched to Plune cases by the same key as a JUnit report's, classname#name, where the adapter's names allow it (pytest, JUnit 5, Playwright); for Vitest, Jest and Cypress the key is the runner, the file and the test's full name (vt:test/cart.test.js#cart adds an item, jest:…, cy:…). A file whose name starts with _ is not sent: the core hides it as Playwright's own reports do, and Allure has no such convention, so name attachments without a leading underscore.

Check that the output reaches the report: Gradle writes it only with reports.junitXml.outputPerTestCase = true, and Maven Surefire moves it out of the XML when redirectTestOutputToFile is true. As in the Playwright reporter, a file whose name starts with an underscore is skipped, and a file several tests name is uploaded once per test.

Several jobs, one run

Two suites, or a sharded matrix, report into a single run when every job shares a key and knows it is not the last one:

PLUNE_SHARED_RUN=1 plune run import ./junit.xml     --key "$GITHUB_RUN_ID"
PLUNE_SHARED_RUN=1 plune run import ./app/junit.xml --key "$GITHUB_RUN_ID"
plune run finish "$RUN_ID"                          # once, when they are all done

PLUNE_SHARED_RUN=1 is what stops a job closing a run its siblings are still reporting into; the key alone says where the results go, not who ends the run. Leave it unset in the last job and that job closes the run instead of the explicit finish.

plune run exec sets it for you — anything reporting inside it, this command included, joins without closing.

When a test is deleted

A test that vanishes from the code stays in Plune as an active case until something says otherwise. The run that covers the whole suite is what says it:

PLUNE_FULL_RUN=1 plune run import ./junit.xml      # or the same variable on the Playwright reporter's job

When that run finishes having reported everything it expected, every case its source used to report and did not this time is marked detached — not deleted, not retired: it keeps its history and its place, and the dashboard shows it with the run as the reason. The next result on it brings it back by itself. Set the variable only on the job that runs everything: a run of one file or a --grep also reports everything it expected, and with the flag it would detach the rest. A run that lost a shard (notRun is not empty) never detaches anything. plune run start has no list of tests to compare against, so the flag means nothing there.

What the run is called

A run nobody named is called after the directory, the minute it started and where it ran — plune · 2026-09-15 15:26 · ci — so a row in the list reads without being opened. The clock is the reporting machine's own; ci comes from the CI variable every hosted runner sets. To call it something else, set PLUNE_RUN_TITLE; PLUNE_ENV and PLUNE_LABELS mark where it ran and how, and those stay marks rather than becoming part of the name.

Optional: keep a history

Everything above works with no account, no network, and no token — that does not change. run, report, diff and init never open a socket, and that is the half this promise is about. If you also want run history, trends, and a shared dashboard, the commands below push your local runs to the Plune platform:

plune login          # paste the API token from your platform settings page
plune run            # exactly as before — the run is saved locally
plune sync           # upload .plune/last-run.json, print the read-back URL
plune pull           # the test cases as plune/cases.md — edit them anywhere
plune push --dry-run # what a push would change, before it changes anything

The token is stored at ~/.config/plune/credentials.json (mode 0600, honours XDG_CONFIG_HOME) and is never printed or logged. PLUNE_API_URL points sync, pull and push at a different server — useful for a self-hosted backend. The Markdown format pull writes and push reads is documented at docs.plune.ai/platform/cases/markdown.

Programmatic API

The same engine that powers plune run is exported for use from your own code. Unlike the CLI, the library does not parse argv or auto-load .env — set the provider key in process.env yourself.

import { run } from '@plune-ai/cli';
import type { RunResult } from '@plune-ai/cli';

const result: RunResult = await run({ dryRun: false, configPath: 'plune.yaml' });
console.log(result.summary); // { total, passed, failed, errored, ... }

Use in CI

Run Plune on every pull request and post a regression diff as a sticky comment with the companion GitHub Action, plune-ai/eval-action:

- uses: plune-ai/eval-action@v1
  with:
    config: plune.yaml
    fail-on-regression: true

Contributing

Bug reports and pull requests are welcome — see CONTRIBUTING.md. For security issues, see SECURITY.md.

License

MIT © Plune Contributors