HonestCode 发布：为 AI 编码智能体提供确定性代码校验层

The deterministic verification layer for AI coding agents.

Catch invented APIs, undefined symbols, wrong calls, and other code hallucinations before your agent moves on.

The problem

AI coding agents hallucinate. They invent function names, fabricate library APIs, and call methods that do not exist.

HonestCode is the layer that checks what the agent just wrote against the repository it is supposed to be grounded in.

Agent
  ↓
writes code
  ↓
HonestCode
  ↓
evidence
  ↓
Agent fixes
  ↓
verified code

No LLM calls. No network. Pure AST + symbol resolution.

Demo

The agent was asked to "add authentication using the existing UserClient." It generated:

client.refresh_token()

The real project only has:

class UserClient:
    def refresh(self): ...
    def refresh_access_token(self): ...

HonestCode catches it deterministically:

✗ verification failed — 1 issue in app/login_broken.py

  [invented_api] app/login_broken.py:9
  `UserClient.refresh_token()` does not exist.
  did you mean: refresh_access_token()?
  available methods:
    - refresh()
    - refresh_access_token()
  confidence: deterministic · action: revise

Run it yourself:

cd demos/invented-api
python run.py

Why not just use Ruff / Pyright?

HonestCode is not a replacement — it is a verification layer that answers a specific question: did the agent's code actually come from this repository?

The key difference is repository grounding. Ruff checks the file. Pyright types the call. HonestCode checks whether the call resolves to a real symbol that exists in the codebase the agent is editing. (HonestCode does not do full type inference — its structural checks catch mismatched arities, iterating None, and calling constants, not general type errors.)

What a call site is checked against depends on how the receiver's type is known: an annotation (client: UserClient), a constructor assignment (client = UserClient()), self/cls, and an inline constructor (UserClient().refresh()) are all resolved. A receiver produced by a factory function (make_client().refresh()) is not judged — function signatures carry no return type in the surface model, and guessing there would trade a missed finding for a false one.

Install

pip install honestcode

Requires Python >= 3.10. 100% local.

30-second setup with Claude Code

# Add HonestCode as an MCP server
claude mcp add honestcode -- honestcode-mcp

Or add it manually to your Claude Code config (~/.claude/config.json on macOS/Linux, %LOCALAPPDATA%\Claude\config.json on Windows):

{
  "mcpServers": {
    "honestcode": {
      "command": "honestcode-mcp",
      "args": []
    }
  }
}

Restart Claude Code. You now have two verification tools available:

• scan_file(path) — legacy tool, returns issues plus the new evidence shape.
• verify_file(path) — agent-facing tool, returns status, findings with evidence, and a human-readable text summary.

Try prompting Claude:

Add a login endpoint using the existing UserClient, then run verify_file on the
file you just wrote and fix anything it reports.

HonestCode auto-discovers the project root from the file path, indexes the codebase, and returns grounded evidence.

Real example

# auth/client.py
class UserClient:
    def refresh_access_token(self): ...

# auth/login.py
from auth.client import UserClient

def login(client: UserClient):
    client.refresh_token()  # invented API

>>> from honestcode.mcp.tools import verify_file
>>> verify_file("auth/login.py")
{
  "status": "fail",
  "file": "auth/login.py",
  "findings": [
    {
      "line": 7,
      "kind": "invented_api",
      "symbol": "refresh_token",
      "owner": "UserClient",
      "message": "`UserClient.refresh_token()` does not exist.",
      "evidence": {
        "available_methods": ["refresh", "refresh_access_token"],
        "did_you_mean": "refresh_access_token"
      },
      "confidence": "deterministic",
      "action": "revise"
    }
  ],
  "text": "✗ verification failed — 1 issue in auth/login.py\n  ..."
}

Agent output protocol

verify_file returns evidence an agent can act on directly, not just an error message:

{
  "status": "fail",
  "file": "auth/client.py",
  "line": 42,
  "kind": "invented_api",
  "symbol": "refresh_token",
  "owner": "UserClient",
  "message": "UserClient.refresh_token() does not exist.",
  "evidence": {
    "available_methods": ["refresh", "refresh_access_token"]
  },
  "confidence": "deterministic",
  "action": "revise"
}

How it works

1. Auto-index the project (or reuse the cached symbol index).
2. Parse the target file with the standard-library ast module.
3. Resolve every call site to a concrete symbol:
   • infer the receiver type from annotations, constructors, and imports;
   • reconstruct the class's member surface from the repository AST;
   • mark the surface as unknown when a base class cannot be resolved or the class defines __getattr__.
4. Emit findings with kind, owner, evidence, confidence, and action.

The loop is deterministic and auditable.

MCP tools

By default only scan_file is exposed. Set HONESTCODE_TOOLS=all to enable the full set, including:

See the old tool reference below for the complete list.

Benchmark

HonestCode includes three benchmark suites that run automatically in CI on every push and pull request to main:

Agent-accuracy benchmark

benchmarks/agent_accuracy/ is a deterministic, LLM-free benchmark that measures how well HonestCode catches common agent hallucinations (recall).

Each task is a tiny agent episode: the agent writes a broken file, HonestCode verifies it, then the file is replaced with the fix and verified again.

cd benchmarks/agent_accuracy
python run.py

Current results (6 tasks, deterministic verification):

Summary: precision 1.0, recall 1.0, F1 1.0, false-positive rate 0.0, median verify time 33.7 ms.

The cross_file_new_symbol task pins the agent-session flow (write a module, then a file importing it — the index must be fresh, never stale); the dep_alias_and_pypi_name task pins dependency grounding through aliased submodules and PyPI-named requirements (PyYAML → yaml).

Real-world false-positive benchmark

benchmarks/realworld_fp/ measures precision on real code: it scans a vendored snapshot of psf/requests (19 files, ~6.4k LOC, Apache-2.0) and asserts HonestCode reports zero findings. Real code exercises the constructs synthetic tasks never do — stdlib imports, keyword-call style, @staticmethod members, classes inheriting stdlib ABCs, compat re-export modules — and any finding on this corpus is, by construction, a false positive.

python benchmarks/realworld_fp/run.py
# files: 19  findings: 0  elapsed: 5.4s (286 ms/file)

Performance benchmark

scripts/benchmark.py measures the latency of core operations (index, scan, graph, dead code detection, similarity search) on the repository itself.

python scripts/benchmark.py                          # text output
python scripts/benchmark.py --format markdown        # table for README
python scripts/benchmark.py --repo psf/requests      # benchmark a real-world repo

All three benchmarks run in CI (.github/workflows/ci.yml) as separate jobs: benchmark-accuracy, benchmark-realworld, and benchmark-performance. A benchmark failure blocks the build if precision or recall drops below 1.0.

See benchmarks/agent_accuracy/README.md for the dataset format and how to add new accuracy tasks.

Architecture

honestcode/
├── verify/        # Evidence protocol + repository-grounded verification
├── mcp/           # MCP server layer
├── honest/        # Symbol index + project binding
├── graph/         # Persistent call graph (SQLite)
├── structure/     # AST extraction
├── sandbox/       # Sandboxed execution
└── cli.py         # Command-line interface

verify_file is the agent interface. scan_file is the default MCP tool and returns both the new evidence shape and the legacy issues list.

Roadmap

v0.1 — Repository grounding (now)

• symbol / API / function / class / method / import / call verification
• invented API detection with structured evidence
• auto-index on scan_file

v0.2 — Semantic contract verification

• function expects UserID, agent passes User → suspicious

v0.3 — Execution verification

• static verification → tests → runtime evidence

v1.0 — Verification Runtime for Coding Agents

                 Coding Agent
                      │
              ┌───────▼───────┐
              │   HonestCode  │
              │ Verification  │
              │    Runtime    │
              └───────┬───────┘
                      │
       ┌──────────────┼──────────────┐
       ↓              ↓              ↓
   Static          Semantic       Runtime
 Verification     Verification   Verification
       │              │              │
       └──────────────┼──────────────┘
                      ↓
                   Evidence
                      ↓
                     Agent

Development

git clone https://github.com/Fengrru/honestcode.git
cd honestcode
python -m venv .venv
.venv\Scripts\activate  # Windows
# source .venv/bin/activate  # macOS/Linux
pip install -e ".[dev]"
pytest
ruff check honestcode tests scripts

Run benchmarks locally:

# Accuracy benchmark
python benchmarks/agent_accuracy/run.py

# Performance benchmark
python scripts/benchmark.py

See CONTRIBUTING.md for pull request guidelines.

Security

See SECURITY.md for vulnerability reporting.

Telemetry

HonestCode collects no telemetry. There are no analytics libraries, no background services, and no phone-home endpoints.

License

MIT - Copyright (c) 2026 HonestCode Team