AI Architecture & Systems Engineering Vault：Transformer 到生产级 AI 系统的开源笔记

AI Architecture & Systems Engineering Vault 🧠⚡

Open-Source First-Principles Notes & Production Systems Engineering Breakdowns Maintained by @ai.transition

Welcome to the AI Architecture & Systems Engineering Vault. This repository holds the material behind each @ai.transition episode — the core formula, whiteboard schematic, and teleprompter script for foundation model mechanics and distributed AI serving infrastructure. Some parts go deeper into production trade-offs and failure modes; all of them are first-principles, short, and built to be read alongside the video.

📚 Curriculum Syllabus

Volume I: Transformer Foundations & Macro Architecture

• Part 01: Self-Attention Mechanism
• Part 02: Transformer Block Lifecycle

Volume II: Scaling Laws, Alignment & Inference Mechanics

• Part 03: Chinchilla Scaling Laws
• Part 04: Post-Training Alignment
• Part 05: Inference Sampling & Temperature

Volume III: Serving, Memory & Applied Production Systems

• Part 06: KV Caching & The Memory Wall

• Part 07: PagedAttention & vLLM Memory Paging

• Part 08: Remote MCP Architecture & Measure.sh Teardown

• Part 09: The Context Distraction Paradox & RAG

  • Keywords: Softmax Attention Dilution, "Lost in the Middle" Degradation, Parametric Memory Collisions, Cross-Encoder Re-ranking, Context Budgeting.

• Part 10: Mobile Systems → AI Systems

  • Keywords: LMK ↔ KV Cache Preemption, Virtual Memory ↔ PagedAttention, 16ms Frame Budget ↔ TTFT, Reactive Streams ↔ Token Streaming, On-Device Inference (ExecuTorch / CoreML / NPUs).

• Part 11: Prompt Injection — No Patch, Just Blast Radius

  • Keywords: One-String Context (No Privilege Boundary), SQL Parameterization vs. No-Grammar LLMs, Blacklist Futility, Indirect Injection, Least-Privilege Tools, Human-in-the-Loop Confirmation, Dual-LLM (Privileged / Quarantined) Pattern.

• Part 12: Summarising Your Agent's History Can Triple Your Bill

  • Keywords: Stateless API & Quadratic Conversation Cost, Cache Read vs. Write Economics, Prefix-Structured Invalidation, Rolling-Summary Trap (Re-compression), Breakpoint Placement, Boundary Compaction.

• Part 13: Your Agent's Errors Return 200

  • Keywords: Success-Shaped Failure (No Exception to Catch), Inert Retry Semantics, Detection Cost Floor (Check ≈ Produce), Truth vs. Provenance, Grounding Assertions for Invention, Schema vs. Semantic Checks, Irreversibility as the Verification Budget, Mutation Testing Non-Deterministic Output.

• Part 15: $33,000 to Say Yes

  • Stands on: Part 13 (detection cost — the check costs about as much as the work).
  • Keywords: LLM-as-Judge Cost ($33,000 vs. $160 per 1M Checks), Jev (TypeSafe AI) as the First System One Model, Typed Boxes with a Probability per Option, Confidence-Threshold Routing (Ship / Human Review), Frontier vs. Cheap Open Model vs. Decision Model, Forced-Choice Failure (0 of 30 Out-of-Scope Flagged), The "None of These" Box.

• Part 16: Paid Twice, Key and All

  • Keywords: Refund AI Agent, Two Tickets → Two Agent Runs, Per-Call Idempotency Key vs. Repeated Decision, Pending Status Looks Like No Refund, Key on the Thing It Changes (Order Line + Action), Ledger Before the Side Effect, One Automatic Run per Irreversible Action, Human Queue for a Second Refund. Includes runnable reference code: refund_ledger.py.

• Part 17: Saved Tokens. Wrong City.

  • Keywords: AI Agent Memory, Running Summary vs. Memory Graph, Rolling Summary Drops Facts Before the Question Exists, Stale Value Survives a Rewrite (Pune vs. Chennai), Entity · Property · Value + Timestamp, Newest Value on a Key Wins, Question Reads Only the Node It Names, Loose Keys (deliver_to vs. shipping_city), Fixed Property List Before the First Write, Short Summary for the Thread. Includes runnable reference code: memory_store.py.

• Part 18: Thinking Off. Flight Missed.

  • Keywords: Making an AI Agent Smarter Without Training, One Pass per Token (Writing More = Thinking More), Thinking Mode as Hidden Steps, Thinking Tokens Billed as Output (300 + 6,000 = 21x), Thinking On Only Where the Answer Needs Steps, Thinking Cap per Route (Not the Model Maximum), Reasoning Field Before the Answer Field When Thinking Is Off, Thinking Adds Steps Not Facts. Includes runnable reference code: thinking_router.py.

Track 2 — AI Product (runs alongside Track 1; part numbers stay cumulative)

Product seen from the room where "what gets built" is decided. Same rule: one visual blueprint per part. Test for the track: if acting on the takeaway changes code inside a system already decided on, it's engineering; if it changes what gets built, or whether, it's product. Same gates as Track 1 — an aha, something countable, a drawable mechanism, survives model progress — and every product part names the engineering part it stands on.

• Part 14: Four Placements, One Label
  • Stands on: Part 11 (prompt injection — same mechanism, commercial payload, operator as injector).
  • Keywords: Slot vs. Sentence (a Label Needs a Thing), Four Placements (UI Slot, Recommendation List, Retrieved Page, System Prompt), Label Efficacy (61 → 55 vs. 22), Prompt-Level Commercial Steering (18 of 23 Models), Reader Undetectability, Counterfactual Diff as the Only Control (Build It, or Ask Twice), The "Never Paid to Say" Contract.

🛠️ Directory Standard

Every topic directory contains:

1. script.md — First-principles deep dive, formulas, production trade-offs, and teleprompter script.
2. meta.json — Publishing title, badge, caption, and semantic keywords.
3. master_cut.mov / master_cut.mp4 — Final rendered reference video cut.

Where available: canvas.png / canvas.jpg (vertical iPad whiteboard diagram) and captions.srt (styled subtitle track).