AI Theresa 开源 Waggle：用 LLM 智能体做证据门控的代码与数据审计

Waggle audits a codebase (and, optionally, the data behind it) the way a careful reviewer would: it states falsifiable hypotheses, measures them against the code and data, lets an adversarial auditor attack every claim, and reports only what survives — with the evidence attached.

Website: aitheresa.ai/diagnosis

Contents: Overview · Method · Quick start · Usage · Command line · Expert lenses · Connectors · Privacy and telemetry · Limitations · Contributors · Citation

Overview

Context

• Language models can now read an entire repository. Asked to "review this code", they produce fluent, plausible findings — but it is hard to tell which claims were actually checked, which were inferred, and which are wrong.
• A single pass tends to mirror the question: models agree with the premise they are given, and there is no record of what was ruled out.
• Many of the questions that matter cross the boundary between code and data ("why do these two dashboards disagree?", "does this permission check hold for every tenant?"). Answering them needs measurements, not opinions.

Approach

Waggle treats an audit as hypothesis testing rather than summarization:

• Falsifiable problems, not opinions. Each expert lens poses problems that state what evidence would confirm or refute them, seeded by deterministic evidence (git history, repository structure, committed secrets).
• Measurement on read-only planes. Agents measure each problem against the cloned workspace and any connected data plane (warehouse SQL, key-value stores, MCP servers, vulnerability advisories) and return a verdict with the evidence used.
• Adversarial claim audit. A separate audit stage re-checks every claim against the workspace before it can reach a report.
• Evidence gate. A claim whose evidence does not resolve to the code or data becomes an open question (a coverage gap), not a finding. Checks that ran and came back negative are reported as ruled out.
• Two audiences, one record. A short Leadership brief (what matters, what to decide) and an Execution report (every finding with its evidence, fix, expected effect, "done when" and "how to verify", plus a REMEDIATION.md a coding agent can run).

What it is not

Not a linter, a SAST rule engine or a test runner, and not a replacement for them: Waggle reads their kind of evidence and asks the questions they cannot express. It does not modify your repositories or data, and it does not run your code.

Method

flowchart LR
  W["Workspace<br/>repos @ pinned SHAs,<br/>local folders"] --> C["Comprehend<br/>classify the system,<br/>choose lenses"]
  C --> B1["Lens 1<br/>Critique → Preflight → Expert"]
  C --> B2["Lens 2<br/>Critique → Preflight → Expert"]
  C --> B3["Lens n<br/>…"]
  P[("Read-only planes<br/>code · SQL · KV · MCP · OSV")] -.-> B1 & B2 & B3
  B1 & B2 & B3 --> A["Claim audit<br/>adversarial re-check"]
  A --> S["Synthesis<br/>themes, bottom line"]
  S --> F["Evidence-gated findings<br/>+ ruled out + open questions"]
  F --> R1["Leadership brief"]
  F --> R2["Execution report<br/>+ REMEDIATION.md"]

Loading

Engineering properties that make runs inspectable and repeatable:

• Read-only by construction — agents get a default-deny tool allowlist (read, search, and mounted read-only data tools), and file access is confined to the workspace.
• Bounded cost — a per-run ledger attributes every model call; past 80% of the cap no new work starts.
• Fail open, visibly — a failed agent or connector is recorded as a degraded stage on the run instead of silently shrinking coverage.
• Checkpoints — every stage boundary is checkpointed, so a stopped or failed run resumes without repeating finished work.
• Incremental re-scans — unchanged code replays prior findings at no cost; findings are carried forward as new / fixed / persisting, and a finding is only called fixed when its code actually changed.
• Memory as prior, not evidence — durable notes (metric definitions, known traps) are recalled as context to re-verify, never cited as proof.

See docs/ARCHITECTURE.md for the module-level design.

Quick start

Requirements: Node.js 22.7+ and git.

git clone https://github.com/ai-theresa-lab/ai-theresa-waggle.git && cd ai-theresa-waggle
npm ci
ANTHROPIC_API_KEY=sk-ant-... npm start

Open http://localhost:4317. (You can also leave the key out and paste it in Settings → API keys, or rely on an existing local claude login.)

With Docker:

docker build -t waggle .
docker run --rm -p 127.0.0.1:4317:4317 -v waggle-data:/data -e ANTHROPIC_API_KEY=sk-ant-... waggle

Usage

Command line

waggle runs the same pipeline without the browser — for scripts, CI and the Claude Code skill. It starts a local server in the background (data in ~/.waggle/data, or THERESA_DATA_DIR), so a run started here also appears in the console.

node bin/waggle.mjs scan --repo https://github.com/<owner>/<repo> --brief "is our auth safe?"   # → {"id": "rs_…"}
node bin/waggle.mjs wait rs_… --timeout 540        # poll until done (repeat while "timedOut": true)
node bin/waggle.mjs result rs_… --out ./waggle-out  # findings.json, REMEDIATION.md and both HTML reports
node bin/waggle.mjs ask --path . --question "Where is monthly revenue computed?"

Every command prints one JSON object. Other commands: doctor, status, stop, budget, telemetry, server start|stop|status (node bin/waggle.mjs help). Text that comes from scanned code is marked untrusted in the output.

API keys and cost

All model usage bills your own API keys. Keys are only ever sent to the provider's API.

Set them in the environment, in a .env file in the working directory, or in Settings → API keys (memory only, unless you tick Save keys on this machine, which writes <data dir>/keys.json with mode 0600).

Every run has a spend cap, $20 by default — change it in Settings → Spending or with THERESA_RUN_BUDGET. The run page shows live spend against the cap. The cap stops a run from starting new work; steps already in flight finish, and the final report-writing and audit steps always run, so the final cost can pass the cap — in our tests by up to about a third (for example $10.56 on an $8 cap). The run page shows how far over the cap a run went. Set the cap with that headroom in mind.

Your first scan

1. Connect → Public GitHub URL: paste https://github.com/<owner>/<repo> (no token needed for public repos).
2. Full Scan → New full scan: tick the repo, optionally write what you care about in the brief ("is our auth safe?", "why do these two numbers disagree?"), and press Run diagnosis.
3. Watch the pipeline live. When it completes, open the Leadership report first; the Execution report has the details and the REMEDIATION.md export.

Private code: connect a GitHub personal access token with read-only access, or a local folder on this machine (or upload one from the browser).

Expert lenses

Comprehend chooses among the built-in lenses (their IDs also appear in telemetry). Lenses are value-free: they describe methods and checks, never one organization's answers.

Connectors

All connectors are read-only.

Connection credentials (tokens, service-account keys, and credentials embedded in a connection URL) stay in memory unless you choose to remember them on this machine; they are never written to state.json or shown back in the console. Data planes are mounted only into runs you tick them for.

Privacy and telemetry

Your code, data, questions and reports stay on your machine; they are sent only to the model providers whose keys you configured, as part of the prompts the agents need.

Waggle sends anonymous usage telemetry so we can see how it is used. It is on by default, the console tells you so on first run, and you can turn it off at any time:

• THERESA_TELEMETRY=0 (or DO_NOT_TRACK=1) in the environment, or the switch in Settings → Telemetry;
• npm start -- --print-telemetry prints every payload exactly as it is sent.

What is sent (and nothing else):

Never sent: repository names or URLs, code, file paths, questions or briefs, finding text, reports, keys, emails, hostnames or usernames. The collector does not record IP addresses. Its source is in telemetry-collector/.

Security

The console has no login: it is a single-user app. It listens on 127.0.0.1 by default, and the Docker example publishes the port on 127.0.0.1 only. Anyone who can reach the port can run scans on your keys and read your reports, so do not expose it to a network without putting your own authentication in front of it. The server also refuses requests that a web page you visit could make on your behalf: a cross-site request (CSRF), or a request under another host name (DNS rebinding).

Reports are generated by language models from code you scan, so they are served under a strict Content-Security-Policy in a sandboxed frame. See SECURITY.md to report a vulnerability.

Configuration

Limitations and threats to validity

• Model judgement. Verdicts are produced by language models. The claim audit and the evidence gate reduce unsupported claims but do not eliminate wrong ones. Every finding carries its evidence and a "how to verify" step — check before you act.
• No public benchmark yet. We have not published a precision / recall evaluation against a labeled set of known issues. Contributions of labeled audit targets are very welcome.
• Coverage follows the lenses. An issue outside every chosen lens can be missed. The Execution report lists what was ruled out and what stayed open, so the absence of a finding is not evidence of absence.
• Run-to-run variation. Two scans of the same code can phrase, rank or group findings differently. Incremental re-scans reduce churn by carrying findings forward.
• Cost and scale. Very large monorepos are slow and expensive; select the repos or folders that matter. One heavy run at a time; further Full Scans queue.
• Providers. Requires an Anthropic key (or a Claude login); an OpenAI-only setup is not supported yet.

Documentation

• aitheresa.ai/diagnosis — product page
• docs/ARCHITECTURE.md — pipeline, planes, checkpoints, incremental re-scan, memory, reports
• SECURITY.md — threat model and how to report a vulnerability
• CONTRIBUTING.md — development setup and checks
• telemetry-collector/ — the telemetry schema and collector source

Contributing

Issues and pull requests are welcome — especially new expert lenses, read-only connectors, evaluation targets and documentation. See CONTRIBUTING.md. Run npm run typecheck, npm test and npm run check:english before opening a PR.

Contributors

Waggle grew out of accel-scope, an internal project at AI Theresa. These people built it:

New contributors are welcome — see Contributing.

Citation

If you use Waggle in research or a write-up, please cite:

@software{waggle_2026,
  title  = {{Waggle}: Evidence-Gated, Hypothesis-Driven Auditing of Code and Data with LLM Agents},
  author = {{AI Theresa}},
  year   = {2026},
  url    = {https://github.com/ai-theresa-lab/ai-theresa-waggle},
  note   = {Formerly accel-scope}
}

License

Waggle is licensed under the Apache License 2.0. It depends on the Claude Agent SDK, which is installed from npm and governed by Anthropic's own terms. See NOTICE for third-party components.