跳到正文
GitHub·· 7 天前AI 评分38

hook-smith:带确定性幻觉红线过滤的冷邮件生成工具

yashsabbavarapu/hook-smith

AI 导读

hook-smith 是一款冷邮件生成工具,用不调用 LLM 的确定性校验器逐句复核生成内容,凡输入记录中不存在的金额、百分比、人数、日期、技术或实体,以及归属错误的真实数据,都会被标记 [REDLINED: Unverified claim] 并判定该 hook 失败。

正文

CI License: MIT Python 3.11+

Signal-to-copy outreach synthesizer with a deterministic hallucination redline filter.

Most AI SDR tools invent case studies, metrics and funding events, which burns domain reputation. hook-smith ingests a structured lea lead-enrichment record, generates personalized cold email angles, and then re-parses every generated sentence with a deterministic validator that never calls an LLM. Any dollar figure, percentage, headcount, date, technology or named entity that is not present in the input record (or that is real but attributed to the wrong signal) is tagged [REDLINED: Unverified claim], and the hook fails.

The generator is replaceable. The redline filter is the product.

Setup

python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
export GEMINI_API_KEY=...   # optional; without it the offline grounded composer is used

Usage

Generate for one lead, several, or a whole directory, leads run concurrently:

hooksmith generate --input fixtures/sample_lead.json
hooksmith generate --input fixtures/ --concurrency 8 --out results/ --strict
hooksmith generate --input fixtures/sample_lead.json --count 5 --json

Redline a draft a human (or another tool) wrote. No API key needed. This is the guardrail on its own:

hooksmith validate --input fixtures/sample_lead.json \
  --draft fixtures/hallucinated_draft.txt --subject "Your Series C" --strict

--strict exits 1 if anything is redlined, so it drops into CI or a send-gate. Exit codes: 0 clean, 1 redlined under --strict, 2 a lead failed to generate. A bad lead in a batch is reported and skipped; the rest still run.

Every verified claim carries the URL that backs it:

[Funding Growth] PASS
  Subject: Linear's $35,000,000 Series B round

  Linear raised a $35,000,000 Series B round with backing from Accel and Sequoia Capital.

  ✓ $35,000,000      https://linear.app/blog/series-b
  ✓ Series B         https://linear.app/blog/series-b
  ✓ Accel            https://linear.app/blog/series-b

How the redline engine works

validator.py runs the same claim extractor over both the draft and the lead record, then checks set membership. Claims are canonicalised, so $35M, $35,000,000 and $35 million all reduce to 35000000.00 and compare equal.

Claim type Caught example
money $50M when the record says $35,000,000
percent a 40% drop in cycle time
funding round Series C when the record says Series B
year / date 2019 when the record says 2022-06-14
headcount 340 engineers, 12 to 340 engineers, 90 days
technology Kubernetes, Next.js when not in the stack
named entity Andreessen Horowitz, Datadog, fabricated investors and logos
misattribution 4 open Infrastructure Engineer roles when 4 belongs to a different role
citation a cited_signals entry that does not resolve to the record

Claims are bound to the signal they name

A number is only true of something. The Linear fixture holds 4 Product Engineer roles and 2 Infrastructure roles, so 4 open Infrastructure Engineer roles is false even though 4 appears in the record. Each funding round and job posting is a SignalScope with its own facts and the claim kinds it owns; when a sentence names a signal, claims of those kinds are checked against that signal alone. Sentences that name no signal fall back to the whole record. Misattribution is reported separately from invention, because "real number, wrong subject" is the subtler failure.

The vocabulary adapts to the lead

Technology detection uses hooksmith/data/tech_lexicon.json plus the lead's own tech_stack, so a tool the bundled lexicon has never heard of is still validated correctly. Extend the lexicon with HOOKSMITH_TECH_LEXICON=/path/to/file.json.

The model is resolved at runtime

There is no pinned model. The client lists models that currently support generateContent, picks the newest general-purpose Flash, and keeps the runners-up. It then handles three real failures seen against the live API:

  • Retirement. A model can still be listed after generateContent is retired for it (gemini-2.0-flash and gemini-2.5-flash both 404 this way). The 404 body names the replacement, so the client follows it.
  • Quota. Limits are per model, so a 429 falls through to the next candidate.
  • Load, transient 5xx is retried with backoff.

HOOKSMITH_MODEL pins a specific model and skips discovery.

Known limitations

Stated plainly, because a guardrail you trust incorrectly is worse than none:

  • Sentence-initial names are skipped. The first word of a sentence is capitalised by grammar, so it cannot be judged as a name. Gartner named you a leader. slips through, while You were named by Gartner. is caught.
  • A sentence naming two signals accepts either signal's numbers. Attribution narrows to the union of the signals named, not to clause-level parsing.
  • Ambiguous lowercase brands are not flagged. stripe and figma only count as claims when capitalised, to avoid false positives on ordinary English.
  • Semantic hallucinations are out of scope. "Your team is drowning in tickets" asserts no checkable quantity. This filter verifies facts, not insinuation.

Development

pytest -v        # 72 tests
ruff check .

CI runs the suite on Python 3.11–3.13, plus the CLI against the fixtures with --strict, so the guardrail itself is exercised on every push.

来源:GitHub · github.com