GC AI 用 12 小时 38 个项目实测 TypeSafe 分类模型 Jev

上周我们在 GC AI 举办了一场内部黑客松，测试 TypeSafe 的新分类模型 Jev。我们暂停了正常的冲刺工作，给团队一整天时间，并说：去构建任何你想构建的东西，尽可能压榨这个模型，看看它会出什么问题。

如果你还没了解过 Jev，它与标准 LLM 的主要区别在于它根本不生成文本。它不会在几秒钟内流式返回 token，而是你传入一些上下文以及一组范围狭窄的问题，比如布尔检查、多项选择选项或有界分数，它会在 0.1 到 0.5 秒内返回结构化评估，成本仅为标准模型调用的十分之一到百分之一。

我们的大部分生产栈都围绕生成式工作流，比如起草法律备忘录、修订合同和交互式聊天，其中延迟以秒计，提示格式始终面临漂移的风险。从模型中获得亚秒级、强类型的答案起初感觉完全陌生，所以我们想看看它在真实产品功能、内部开发者工具和黑客松玩具中表现如何。

到当天结束时，我们有了 38 个项目和实验，其中 23 个提交到三个赛道。得票最多的项目是一只名叫 Jeventus 的公共 Slack 宠物树懒，但这一天给了我们一个出奇清晰的蓝图，指导我们如何围绕快速、轻量的分类模型进行架构设计。

把模型压榨到崩溃

我们的一位工程师没有构建一个精致的产品，而是花了一整天进行压力测试，以摸清 Jev 在哪里成功、在哪里彻底崩溃。

用 Jev 做实验：极限、游戏和产品工作

他的一些发现出奇地令人印象深刻。Jev 能够从 30,000 个 token 的样板文本中挑出一句关键句，并且在 253 个选项中，根据剧情摘要正确分类电影标题 50 次中的 46 次。但一旦他问它基本的日历算术，比如计算某个日期是星期几，它五次中有四次失败。它没有可靠的时间推理或数学能力，如果你把它当作通用知识库，它很快就会迷失。

他最关键的收获是关于如何为批量评估构建上下文。如果你给 Jev 一份文档并问它 20 个关于该文档的不同问题，它会几乎不增加额外延迟地并发评估所有问题。然而，如果你试图将许多相似项目打包到提示中的数组里，并让 Jev 评估该数组，准确率会从 96% 下降到 72%，因为模型开始在相邻行之间交叉污染答案。对于我们的架构，这立即确立了一条规则：分散到单独的并发 API 调用，而不是将表格数据打包到提示数组中。

实时服务条款扫描

在产品方面，最干净的演示之一是 ToS Trap Spotter，这是一个 Chrome 扩展程序，旨在你浏览时扫描消费者服务条款协议。

Fig. 1 - ToS Trap Spotter evaluating paragraphs in real time as the user scrolls. — 图 1 - ToS Trap Spotter 在用户滚动时实时评估段落。

当你打开一个条款页面并触发该扩展程序时，它会将文档分块成段落，并并发地通过 Jev 发送它们，每个块运行八项并行检查，检查诸如自动续订、强制仲裁、集体诉讼豁免、单方面修改权和广泛责任免除等内容。

Fig. 2 - High-confidence traps highlighted in red, with secondary warnings in amber. — 图 2 - 高置信度陷阱以红色高亮显示，次级警告以琥珀色显示。

在Spotify长达110段的用户协议中，该扩展程序在1.9秒内评估了整个页面。当针对一组在提示调优期间未曾见过的合成条款进行留出集评估时，Jev取得了89%的F1分数，而关键词正则基线仅为57%。

Hovering a clause shows each trap and Jev's confidence — 悬停条款可显示每个陷阱及Jev的置信度

最初的失败模式尤其具有教育意义。在早期对实时网站的测试中，每一个数据共享警报最终都被证明是误报。原来，DOM抓取器捕获了页面顶部的cookie同意横幅，而Jev忠实地将其作为数据共享条款进行评估。将模态对话框排除在提取范围之外立即解决了该问题。

另一个微妙的bug出现在责任豁免方面：早期的提示询问某条款是否限制了公司的责任，导致Jev将标准退款政策标记出来，因为退款条款在技术上界定了供应商对客户的义务边界。将提示缩小到专门关注用户遭受的损害、损失或伤害后，误报减少了一半，同时没有漏掉任何实际的责任保护条款。

聊天流式传输过程中的内联引用验证

在法律研究中，无依据的引用或幻觉案例参考比几乎任何其他事情都更快地摧毁用户信任。

我们的一位工程师构建了一个流式引用验证器，在我们的聊天助手正在生成回复时，对照源文档检查各项主张。当每个被引用的句子流式传入时，后台调用会评估所引用的段落是否真正支持该主张。如果来源核实无误，文本保持干净；但如果该主张存疑或缺乏支持，在助手甚至还未完成下一段流式传输之前，内联就会出现琥珀色或红色下划线。

Fig. 4 - Streaming citation validation flagging weak claims in real time. — 图4 - 流式引用验证实时标记薄弱主张。

The claim card shows the verdict first, then each source — 主张卡片先显示判定结果，然后显示每个来源

Fig. 5 - Detailed claim breakdown showing the underlying source passage and the exact discrepancy. — 图5 - 详细的主张分解，显示底层来源段落和确切差异。

这里最棘手的边缘情况归结为情态动词，如“may”与“must”。在一个测试案例中，源文档声明一方可以（may）做某事，而模型生成的摘要则断言该方必须（must）做此事。一个询问来源是否支持该陈述的通用提示完全漏掉了这个错误，但当检查添加了三个针对性的是/否问题，涵盖明确支持、细节不匹配和夸大表述时，Jev立即捕捉到了情态动词的转变。

警报分类、PR审批，以及分类失效之处

几位工程师专注于内部开发者工作流，这产生了一些关于Jev在何处有效、在何处完全失效的最清晰数据。

我们的一位工程师构建了jev-pager，一个值班分诊机器人，它监控生产告警流，以判断是否需要呼叫工程师。它并非仅根据原始告警文本进行呼叫，而是将每条告警与代码计算出的信息（如计数、时钟以及其他正在告警的内容）一起评估。回放整整一周的生产告警，共136条告警，它正确识别了所有需要人工干预的事件，仅出现一次误报。主要挑战是告警模板化用语：原始监控模板经常使用诸如“用户请求出错”之类的惊悚措辞，这导致 Jev 过度升级琐碎的超时问题，直到该机器人先询问告警家族，并将该答案传入呼叫判断。将观测到的 CPU 值交给 Jev 也将呼叫精确率从 0.63 提升到 0.92。

另一位工程师构建了 PR Stamper，它会在人工添加 stamp 标签后，自动审查并批准机械性、低风险的拉取请求。它从不合并。它不依赖宽泛的提示来判断拉取请求的整体安全性，而是将硬编码的策略门禁（阻止对迁移、认证或计费等敏感区域的任何修改）与七项狭窄的、文件级别的 Jev 检查相结合，这些检查涵盖变更类型、是否改变行为或共享契约，以及缺陷可能扩散的范围。

另一方面，我们尝试使用 Jev 进行开放式代码清理，恰好暴露了该模型的局限所在。当我们构建一个机器人来识别无用的单元测试，并向 Jev 提出宽泛问题“这个测试无用吗？”时，它的表现并不比抛硬币好，AUC 为 0.54。然而，当我们不再要求主观价值判断，转而询问关于测试结构的八个事实性问题，例如它是否仅对 mock 调用次数进行断言，或者是否执行应用逻辑时，AUC 跃升至 0.72。该模型擅长验证离散的事实属性，但无法自行做出整体质量判断。

Slack 宠物树懒与类型化 switch 语句模式

我们得票最多的项目与法律 AI 毫无关系。Jeventus 纯粹是为了好玩而构建的，但它最终展示了本次黑客松中最有效的单一架构模式。

Fig. 6 - Jeventus reacting to a team member offering him food in Slack, with Jev's classified outputs updating his state in Postgres. — 图 6 - Jeventus 对团队成员在 Slack 中给它喂食做出反应，Jev 的分类输出更新其在 Postgres 中的状态。

人们初次接触 Jev 时犯的核心错误是试图把它当作聊天模型来使用。构建 Jeventus 的工程师恰恰相反，让 Jev 完全与文本生成隔离。每当有人在 Slack 中给 Jeventus 发消息时，Jev 会从几个特定维度评估该消息：用户是在喂食、玩耍还是责骂这只树懒，以及一个友善度评分。当有人发布牛油果三明治时，Jev 将该动作标记为喂食，置信度为 1.0，友善度高。随后，标准的 Postgres 查询更新树懒的饥饿和情绪变量，由确定性代码选择回复。

将 Jev 视为一个超快速、类型化的 switch 语句，结果成为几乎所有成功项目的制胜设计。当团队试图让 Jev 处理开放式推理或细微差别时，它表现挣扎，但当他们用它把离散输入路由到确定性应用代码时，其延迟和可靠性感觉如同魔法。

我们学到了什么

Jev 不是更便宜的 LLM，也不是更聪明的正则表达式。它是一个通用分类器，你可以用它来塑造几乎任何对代码来说太模糊、对 LLM 来说太耗时的难题。

这个差距比听起来要大得多。因为 Jev 能在几分之一秒内给出答案，并且几乎能泛化到任何狭窄的问题，它出现的场景远比我们预期的要多：确定性代码前的类型化 switch 语句、昂贵模型前的筛选层、与流式响应并行的验证器、agent 循环内工具调用的守卫，以及人工审批前的关卡。在 38 个项目中，我们归纳出了 16 种可复用模式。

关键在于，只有按照它的规则来使用它，它才能发挥作用。Jev 不是生成式模型的替代品，也不是用来处理模糊、主观问题的自动裁判。给它狭窄、客观的标准和明确的否定选项，其余的交给代码和 LLM 处理。

黑客松之后，我们的一位工程师将这些发现和架构模式整理成了一套内部开发者技能，名为 jev-design。现在，只要工程师开始设计分类工作流，我们的编码 agent 就会自动加载这项技能。

完整的内部技能指南附在下方，其后是一个交互式浏览器，记录了黑客松中全部 38 个项目的实现。

---
name: jev-design
description: Design helper for features that use Jev (TypeSafe's structured-judgment model, `typesafe-ai/jev`). Walks through whether a task is Jev-shaped, which question shape and composition to use, where to place it, and how to pick thresholds, using patterns and results from the Sep 2026 Jev Hack Day. Includes an interactive pattern explorer (`jev-patterns.html`). Use when someone plans, builds, or reviews a Jev call, asks "should I use Jev for this", compares Jev to an LLM classifier, or wants Jev examples, patterns, or lessons. Also a primer for people new to Jev, including R&D attorneys.
---

# Jev design helper

Jev answers narrow, typed questions about some context. It returns a probability (yes/no, which TypeSafe calls a Noul), a pick from a list (Choice), or a level on a scale you define (Score). It never writes text. Typical calls take 0.1 to 0.5 s and cost 10x to 100x less than the LLM they replace.

This skill has two parts:

- This file: lessons and a design walk-through.
- `jev-patterns.html`: an interactive explorer of 38 hack day projects, 16 reusable patterns, where Jev fits, and where it did well or struggled, with links to each PR, write-up, and recording. Open it in a browser: `open .agents/skills/jev-design/jev-patterns.html`.

## How to use this skill

1. If the user is new to Jev, give the four-line summary in [What Jev is](#what-jev-is) and point them to the HTML explorer.
2. If the user has a feature in mind, walk through [Design decisions](#design-decisions) one step at a time. Ask one question per step. Recommend an answer with the evidence behind it. Do not dump the whole list.
3. Before code review or shipping, run the [Checklist](#checklist).
4. When a decision is close, cite the matching pattern or project from `jev-patterns.html` so the user can read the real result.

## What Jev is

- Request: `state` (string or JSON, up to about 32k tokens) plus a map of named `questions`. Up to 255 options per Choice.
- Answer types: Noul `probability`, Choice `choice` + `probabilities`, Score `score` + `probabilities`.
- Price: about $0.042 per million input tokens. Output is free.
- Not deterministic: the same request can move by about ±0.03. Choice picks are more stable than probabilities.

Call shape (AI SDK 7 through the Vercel AI Gateway):

```ts
import { experimental_evaluate as evaluate } from 'ai-v7';

const { answers } = await evaluate({
  model: 'typesafe-ai/jev',
  state: { claim, source },
  questions: {
    relation: {
      type: 'choice',
      instructions: 'Does `source` support `claim`? Judge only from `source`.',
      criteria: {
        supports: 'The source states the claim or directly implies it.',
        contradicts:
          'The source states something that conflicts with the claim.',
        silent: 'The source does not address the claim.',
      },
    },
  },
  providerOptions: {
    gateway: { zeroDataRetention: true, only: ['typesafe-ai'] },
  },
});
```

`packages/ai` already depends on `ai-v7` and `@typesafe-ai/sdk`. There is no shared Jev wrapper on `staging` yet. Several hack day PRs each added their own. Before you add another one, check `packages/ai/src/jev/`.

**Customer data:** send it only through the Vercel AI Gateway with `AI_GATEWAY_API_KEY` and `typesafe-ai/jev`. Set `zeroDataRetention` and `only: ['typesafe-ai']`. `TYPESAFE_API_KEY` (direct to TypeSafe) is for public or synthetic data only.

## Design decisions

Work through these in order. Each step lists the options, the default, and the evidence.

### 1. Is the task Jev-shaped?

Yes when a knowledgeable person could answer it in about a second with the right context, and the answer is a fixed set.

| Strong fit                                                                            | Weak fit (use code, an LLM, or a fallback)                                       |
| ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| Pick from a list: skill, email bucket, alert family, agent route, document type       | Holistic calls: "is this sound advice" (83%), "is this test useless" (coin flip) |
| "Does this text do X?" on concrete text: ToS traps 89% F1 vs 57% regex; SMS spam 100% | World knowledge and trivia: missed "Nike" = goddess of victory; band trivia 57%  |
| Same entity: 95.5% agreement with Opus at $0.00011 per probe                          | Counting, dates, math: weekday of a date 20%; counts off by one                  |
| Claim vs. source: 0 of 294 good cites hidden                                          | Criteria about files or tool output: 59% vs 96%                                  |
| Pick a candidate code found: 9/9 fields in one call                                   | Near-synonym labels, two-hop cross-references, security judgments                |

If the task needs written output, Jev can still screen or route in front of the LLM (step 4).

### 2. Question shape

- **One narrow question per decision.** Write it by hand. AI-generated question lists gave no signal. Check the wording on 20 to 50 labeled items before scaling.
- **Fuzzy judgment:** split it into 6 to 10 concrete yes/no patterns, mined from real history, with per-pattern thresholds. One broad question scored AUC 0.54. Narrow patterns plus a code-side count scored 0.72.
- **Clean category:** use one Choice with a short description per option. Document status scored 88% as one Choice vs 82% split into yes/nos. A described 5-option Choice scored 1.00 where two vague yes/nos disagreed.
- **Extraction:** code finds candidates (dates, sentences, fields) and Jev picks one. Always add `none` and a separate "is it present?" yes/no. Without an escape option, Jev picked a wrong date at 0.92 confidence.
- **Ties possible:** add a "is there any real basis to decide?" yes/no. With no context, Jev still picked a winner at 0.81 to 0.89.
- Ask each decision one way. A question and its negation do not sum to 1.

### 3. State and batching

- Send only relevant context, but a relevant full document is fine. A 12k-token document matched focused excerpts, and full-contract context caught buried risks. Misleading or boilerplate text hurts: Jev believes what it reads.
- Strip template text. Put facts code already knows in the state (counts, computed dates, the CPU value). That took one router's page precision from 0.63 to 0.92.
- Many questions about one item in one call: nearly free (1 to 150 questions went from 0.33 s to 0.76 s).
- Many similar items in one array: Jev loses its place (primes 72% in an array, 88% keyed, 96% one per request). Fan out one item per request, or key items by name (`cite_4`, not `citations[4]`).
- Many pick-from-list questions in one call also hurt (83% at 5 sentences, 60% at 40).
- Separate trusted from untrusted text in the state (user ask vs. pasted material vs. tool output).

### 4. Composition: who decides

| Option                                                                 | Use when                                    | Example                                                   |
| ---------------------------------------------------------------------- | ------------------------------------------- | --------------------------------------------------------- |
| Jev + code                                                             | Mistakes are cheap or reversible            | Skill pills, tool card layout                             |
| Confidence bands: Jev on the clear share, stronger model on the middle | Quality must match an LLM at lower cost     | Relationship detection: same 91.9% accuracy, 43% cheaper  |
| Jev screens, LLM finishes                                              | Positives are rare, or text must be written | Preference memory: $245 vs $2,830 a month                 |
| Classify first, then decide                                            | Misleading text sways a single call         | Alert pager: false pages fixed by asking the family first |
| Profile → policy file                                                  | Several signals feed one action             | Inbox triage: 26/26 vs Sonnet 25/26                       |
| LLM writes the Jev question                                            | The request is messy and open-ended         | Caesar bot. Add a basis question.                         |
| Jev, then a person                                                     | Approvals, merges, CI gates                 | PR stamper, same-buyer checker                            |

Default for anything that matters: confidence bands. Make them asymmetric when one error costs more (0.40 / 0.70 for relationships).

### 5. Placement

- **Around the chat UI** (most common): as you type, after a message, while streaming, when a panel opens. The agent never waits.
- **Inside the agent loop:** only for tool-call guards. Adds about 200 ms per call. Plug into `needsApproval` or block. Decide fail-open vs. fail-to-approval, and make replays safe.
- **Backend pipelines:** ingestion, routing, document intelligence. Volume is high, so bands and fan-out matter most.
- **Internal tooling:** CI checks, approvals, alert routing. Keep hard rules (security paths, permissions) in code.

### 6. Thresholds and evaluation

- Build a labeled set first. Split by file or PR so near-duplicates do not leak across the split.
- Pick the cutoff on a tuning split and check it on held-out data. Cutoffs ranged from 0.25 to 0.95 across projects. Do not assume 0.8.
- Jev's probabilities are usable for routing. In one test, every wrong answer from the comparison LLM was at 0.85 confidence or higher.
- Do not trust LLM auto-labels on borderline items without a human check.

### 7. Failure handling

- Fail closed for auto-actions and suggestions (approve, merge, show a card). Fail open for checks that would block users (citation chips, guards, lint hooks).
- Give timeouts room for the SDK retry. A 1.5 s budget aborted inside the retry wait; 3 s worked.
- Cache verdicts so the UI does not flicker. Judge only new items instead of re-asking.
- Pin the model version and hash the questions so tuned thresholds fail loudly when either changes.
- On untrusted input, add an explicit injection yes/no and let it win. Do not auto-act on Jev alone below the calibrated bar.

## Checklist

- [ ] The task passes step 1, or Jev only screens or routes.
- [ ] Each question is narrow, hand-written, and checked on labeled items.
- [ ] Every Choice has `none` or "not applicable" when the answer may be absent.
- [ ] Code does counting, dates, math, and candidate extraction.
- [ ] Similar items are fanned out or keyed, never indexed in a long array.
- [ ] Customer data goes through the gateway with zero data retention.
- [ ] Thresholds come from a held-out split and live in one policy file.
- [ ] Low-confidence answers go to a stronger model or a person.
- [ ] Untrusted input has an injection question and trusted/untrusted fields are separate.
- [ ] Timeout, retry, cache, and fail-open or fail-closed choices are explicit.

## Sources

- Interactive explorer: `jev-patterns.html` in this folder.
- Hack day hub and submissions: [GC AI Jev Hack Day](https://app.notion.com/p/3e257c014bec80ae82e3f5a109f9ff57).
- TypeSafe docs: [primitives](https://docs.typesafe.ai/primitives), [how to build](https://docs.typesafe.ai/concepts/how-to-build-with-system-one), [failure modes](https://docs.typesafe.ai/model-jaggedness/jev-1.13).

Numbers come from small, mostly synthetic sets, usually one run. Treat them as direction, and rerun on real data before a Jev call gates anything.

我们还整理了一个包含全部 38 个项目的交互式矩阵，其中包括哪些有效、哪些失败，以及完整的延迟和准确率数据。

在新标签页中打开完整模式矩阵

顺便说一句，Jeventus 在他的 Slack 频道里依然活跃。如果花一天时间用新模型构建快速原型听起来是你的乐趣所在，我们正在整个团队积极招聘。