swyx 访谈 Mihail Eric：斯坦福 CS146S 如何教授 AI 原生软件工程
swyx 访谈 Mihail Eric：Stanford CS146S 如何教授 AI 原生软件工程
Latent Space 发布 swyx 与斯坦福 CS146S 授课教师、Monaco AI 负责人 Mihail Eric 的访谈，讨论软件工程师当下该学什么。
译文尚不完整，完整内容请切换到原文。
Latent Space· swyx .ai·2026-11-09 05:41· 40 分钟前AI 评分60
斯坦福如何教授 AI 原生工程 —— Mihail Eric，Monaco
AI 导读
Latent Space 发布 swyx 与斯坦福 CS146S 授课教师、Monaco AI 负责人 Mihail Eric 的访谈。
正文 · 原文
AI Engineer YouTube 频道刚刚突破一百万订阅者。感谢每一位观看演讲、把演讲转发给同事，或将所学投入生产的人。为这期特别节目，我们想请一位经常要在工作中回答我们许多人都关心的问题的人来聊聊：软件工程师现在究竟应该学什么？
Mihail Eric 有两个地方可以检验他的答案。他在斯坦福教授现代软件开发者课程，并在 Monaco 领导 AI 工作，他的同事们在那里使用编码智能体交付软件并排查故障。我请他带我们过一遍课程大纲，然后告诉我们哪些部分经受住了他自己工程团队的实践检验。
这也算是对下周来旧金山参加 AIE Code 的各位的一次预告，作为你们即将与演讲嘉宾、赞助商和参会者展开的所有对话的必备知识。如果你是 LS 订阅者且能看到这篇文章但还没买票，可以使用此优惠码享受 30% 折扣（抱歉，不可退款）
去年的作业只需一个提示词
CS146S 的第一轮有九次作业和一个毕业项目。其中一项练习中，学生在代码仓库上运行 AI 代码审查器并评估其反馈。到下一轮时，Mihail 认为学生可以一次性完成其中很多练习。当他问一屋子斯坦福大二和大三学生如何写代码时，他调查到的每个人都已经在使用智能体了。
所以这一次，部分作业是为一个开源项目做贡献。仓库维护者会审查学生的补丁。我喜欢这个作业带来的效果。你可以让智能体在一个下午实现一个功能。但你仍然需要弄清楚维护者是否想要它、代码为何如此运行，以及你的补丁还可能破坏什么。这种对话也是初级工程师向资深工程师学习的方式。
本期节目里有一个练习，你可以借用到下一次 PR 中。Geoffrey Litt 的 explain-diff 技能会把一次变更变成可视化演示，然后就此向你提问。拿你智能体刚写的补丁试试。哪个问题是你必须重新打开代码才能回答的？观看讨论。
Monaco 买什么，建什么
在 Monaco，团队评估了代码审查产品，并继续使用 Devin。智能体工作区则走了另一条路。后台智能体需要有个地方来运行产品、复现故障并测试补丁。具体细节取决于 Monaco 的云环境如何搭建，所以团队自建了工作区。Mihail 的经验法则是：“买来学习，自建来保留。”
一个成果是 Hal。工程师可以让它调查某个出故障的东西并开一个 PR。销售人员可以问同一个系统产品如何运作。在一种情况下，有用的答案是一个可供审查的变更。在另一种情况下，是一个可以讲给客户的解释。给这两个人同一堵调试输出墙，你只服务了其中一个人。
我不断回到的问题是：PR 的另一端会发生什么。智能体看到的是当前代码。它该去哪里查明团队六个月前为何否决了同样的改动？这就把我们引向 Basis 的 Authority Map，以及 Mihail 的预测：开源仓库将变成由智能体运营的软件工厂。这也是如今把学生送到维护者身边的一个好理由。总得有人来决定哪项改动属于这个项目。
如果你正在用编码智能体做开发，欢迎来参加 AI Engineer CODE。如果你一直在自己教这些内容，我很想知道你最先不得不重写的是哪份作业。
节目笔记
Stanford CS146S —— 十周的教学大纲、作业和开源合作伙伴。你也可以切换到上一年的课程进行对比。
构建一个编码智能体和深入 Claude Code 内部 —— Mihail 对工具循环及其外围 harness 的讲解。
理解才是新的瓶颈 —— Geoffrey Litt 关于如何弄懂 AI 编写的 diff 中改了什么的方法。
Monaco 的智能体开发者工作区 —— 该团队构建一个让智能体可以运行和测试应用的地方的理由。
如何终结代码审查和其背后的 Faros 研究 —— AI 采用率高的团队合并的 PR 多出 98%，而他们的 PR 审查时间却长了 91%。
Basis 的 Authority Map —— 智能体应该去哪里查找当前行为、早先的决策和意图。
Stripe 的 Minions —— 一个生产环境案例：为后台编码智能体提供环境，并打通通往经工程师审查的 PR 的路径。
Boris Cherny 的 AI 采用步骤 —— 我们在节目接近尾声时投到屏幕上的五个阶段。
Mihail 谈 The Thinking Mode —— EO Studio 的采访，我们在节目开头做了节选。
时间戳
0:00 · 一百万 AI Engineer 订阅者与 CODE 峰会
0:51 · Mihail Eric 与 Stanford CS146S
5:26 · 初级工程师现在需要学什么
10:02 · 开源维护者审查学生的代码
15:04 · 十周教学大纲巡览
24:04 · 编码智能体内部的循环、工具与 harness
32:18 · 面向大型代码库的上下文工程
38:15 · 让你看懂 diff 的技能
46:42 · Monaco.com 之争与飞机横幅
50:23 · Monaco 为销售团队做什么
59:21 · 走进 Monaco 的工程团队
1:01:40 · “买来是为了学习，自建是为了留住”
1:05:24 · 让仓库为智能体做好准备，并闭合 on-call 循环
1:12:37 · 当代码审查成为瓶颈
1:20:37 · 安全、依赖与权限
1:30:13 · 后台智能体与 Stripe 的 Minions
1:34:16 · MCP 门户与基础设施为何趋于收敛
1:41:22 · 构建软件工厂
1:45:24 · 还留给开发者什么工作？
1:52:16 · 从个人采用到团队自主
文字稿
在下方下载完整文本。在网络版中，你还可以阅读文字稿并点击任意时间戳跳转到采访对应位置。
| transcript.txt153.06 KB • PLAIN 文件 |
自动转录文本，已与最终剧集对齐。时间戳可打开 YouTube 上对应的时刻。
0:00 swyx · 开场 — 好，这是一期非常特别的节目，因为我们正在庆祝 AI Engineer YouTube 频道达到 100 万订阅者。所以如果你还没听说过，就去那边订阅吧。嘉宾是 Mihail Eric，他不仅在斯坦福大学教授《现代软件开发者》这门课——这门课现在正在走红，基本上涵盖了所有 AI 原生开发的最佳实践——而且他还是 Monaco 的 AI 负责人。所以他不仅是老师，也是实践者。
0:24 下周来参加 AI Engineer CODE Summit 的朋友们，这将是对过去一年你在编程领域所见到的所有极端进展的庆祝。我对所有主题都感到兴奋：对抗 slop、系统一模型，以及提升你对基本上所有代码中你所见到的东西进行大规模重写的雄心。做 AI 工程师从未有过比现在更激动人心的时刻。而有了 100 万 AI 工程师，我觉得这才刚刚开始。
0:49 那么让我们进入采访。
0:51 好，我们在演播室里和 Mihail Eric 在一起，他是斯坦福大学的兼职教授，也是 Monaco 的 AI 负责人。欢迎。谢谢。很高兴来到这里。你引起了我们的注意，是因为你的课程，你正在第三次更新它，第三次？
1:03 第二次。第二次。是的。CS146，或者说 146S，《现代软件开发者》，在斯坦福大学。
1:09 Mihail Eric · EO Studio 片段 — 这一代初级开发者、初级工程师，也就是现在进入职场的人，我认为将成为这种新转变的第一代。
1:18 我看了教学大纲，我就想，这就是我们涵盖的一切。所以时不时我喜欢邀请像你这样的人，你涵盖广泛的主题，然后我们可以 sort of 一个主题一个主题地聊，给大家每个主题的最新技术水平，我猜你和所有嘉宾都是这么做的。当然。给我们讲讲这门课的历史。
1:36 是什么让你想推出 CS 146？是的，这是第二次教授这门课，我第一次教是去年秋天，也就是 2025 年秋季。那是在 4.5、4.6 之前的日子。甚至在 2025 年秋季之前，这门课已经酝酿了大约一年。这门课的起源大概是 2024 年 11 月。我当时有这样一个想法：我们看到编码智能体领域发生了所有这些事情，是的，它们还是编码智能体的早期版本，但已经感觉开发者的工作方式正在发生某种剧烈的变化，我们很快就能完成工作，所有这些有趣的东西。
2:08 所以当时，一切都只是零散的博客文章，如果你想了解做这些事情的方法论，你能在网上找到的零散东西。所以我有了这个想法，嘿，有没有办法围绕这个构建一个更正式一点的课程体系？
2:23 有了这个想法，我的母校是 Stanford，我一直想回去教点什么，把这份回馈传递下去。我一直深耕于教育领域。我和 Chip 聊过，她也教过一门课，我的好朋友 Chip Yen，对，就是 Chip，我和她聊过，她教的是设计机器学习系统，所以我问了她那个过程是怎样的，从她设计课程的方式中汲取了很多灵感，思考着让学校把课程组织起来，花了大约九到十个月，从 2024 年秋季一直到 2025 年中后期，让课程通过了委员会审批，
2:58 为这门课的存在做了论证，得准备一份提案等等。这把我带到了 2025 年秋季，我第一次听到它的时候，反响非常好。学生们对它真的很感兴趣。去年甚至还有一篇 Business Insider 的文章写了这门课。然后，是的，今年很多事都变了。我想说尤其是 agent，自主编码 agent，真的以我们前所未见的方式发展起来了。
3:23 所以感觉是时候再做一次了，更新一下内容，然后就有了现在。与此同时，你还是 Monaco 的 AI 负责人，这也是一家非常热门的初创公司。是的，是的。我们现在正在节目里，玩得很开心。是的，我挺喜欢你同时也是实践者这一点。你们显然在 Monaco 落地了所有这些技术。是的。从高层面上看，从去年到今年，有什么原则或观察发生了变化吗？
3:53 是的，是的。变化太多了。我觉得当我回顾去年那版课程时，我的意思是，编码 agent 已经存在了，但我们仍处在这样一个阶段：不断有公司涌现出来，去攻克 AI SDLC 的每一个环节。有做代码审查的编码 agent，有做规格设计的编码 agent，有做 SRE 的编码 agent。当然，其中一些公司现在仍然存在。
4:16 但感觉去年更多的是广泛地巡览一遍，告诉你编码 agent 可以用在哪些地方。某种程度上，去年很多课程内容都偏向于曝光和了解。这种概览，这种有点像奥德赛式的旅程，带你了解，嘿，你熟悉的整个 SDLC 现在都因为 AI 而改变了。今年，我觉得有一些原则正在固化，对吧？
4:38 比如非常细致的上下文工程这个理念，比如真正深入思考设计和架构，以此作为处理非常复杂任务的方式。所以我的课的性质是，它不是为初学者程序员准备的。它也不是传统计算机科学中算法那类课程的替代品。事实上，那些课程中的很多都是我在这里所做内容的前置课程。
4:59 但我知道你们有很好的基础，现在要朝着应用这些基础、把那些原则运用到真实世界的生产环境这个方向前进，这一点我认为现在正处于一个合适的阶段，你知道，在 Monarcha 我们要处理庞大的代码库。你知道，我也帮助很多公司思考他们庞大代码库的设计之类的事情。所以这更像是一门面向实践者的课程，针对的是学生毕业或即将进入职场后，如何成为最有生产力的员工。
5:25 你说你对教育充满热情。我觉得处于大学阶段的学生正处在一个相当有意思的位置。对于现在人们是否应该学习计算机科学，你有什么建议和看法吗？
5:40 Claude Code 和 Codex 就是我们所需要的全部了吗？比如，我想听听你的想法。是的，你知道，学生们有很多焦虑，你大概能想象得到。我觉得这次我想传达的很多内容是，这种焦虑，其中很多依然存在。去年也有焦虑。但我想论证的是，一个训练有素的 AI 原生工程师——我知道这是个流行词，但我们不妨深入探讨一下其中哪些方面最有趣。
6:10 一个训练有素的 AI 原生工程师可以成为任何公司最有用的资产之一。
6:17 这其实是开课第一周。第一周，第一堂课，我就说，嘿，我真的相信每家公司都需要你们，但门槛高得多。你是作为毕业后的第一份工作进入的，但如果用谷歌那种职级体系来说，仅仅达到 L3 是不够的。你通常得比接近高级工程师的水平还要更好才行。
6:39 那么，我们怎样才能更快地达到那个水平？对吧？所以我想说的是，仍然有一部分核心的基础软件工程很重要。一些关于软件系统如何构建以及这些系统权衡取舍的核心价值。然后在此之上，我们再去，你知道，这就是我认为你们作为人类仍然应该有价值的地方，而其余的实现细节你可以更多地交给别人。
7:02 嗯。是的。这是我之前说过的话，几乎可以说是那种有争议的、为了显得辛辣而说的话，但我几乎真的有点相信它，那就是大多数学生都完了，其中一些人会非常有价值。我不觉得这是个很受欢迎的说法。斯坦福的孩子会没事的，对吧，他们来自斯坦福，但美国其他地方的人会怎样呢？这是个过度的问题。这是个奇怪的情况，因为我觉得很多教育都行动缓慢，我甚至好奇，在一门课程中，你如何确定课程的截止点在哪里？
7:39 你在斯坦福教一门现代软件课程。那正在改变一切。你作弊了。你直接请来那些做编码智能体的人来讲课，对吧？
7:48 哦，对。让他们来当，对。这挺有意思的。新老客座讲座。嗯，对。我是说，那是，好吧。所以关于客座讲座这一点，我觉得那是，我是说，那也是去年的事了。去年也是这种情况。所以那是刻意设计的，因为我，你看，我来了，对吧，我说，我是个实践者，正如你指出的，swyx。就像，我是个实践者。我每天都在做这件事，你知道，我的日常。
8:09 我设计这些系统。你知道，在 Monaco 构建这些东西，以前见过很多次了。话虽如此，我不想声称我今天拥有所有答案。那样会，你知道，非常自大。所以这门课的结构一直是：这里是一位实践者从实地工作中、从与众多业内人士交谈中汇总的一套见解。
8:33 但还有些人真正处在最前沿，你知道，看到的东西甚至超越这些，邀请他们来提出自己的主张、自己对世界的看法。这就是客座讲座存在的意义。我觉得让学生看到大名字、真正在做这件事的人，直接听他们讲，能向他们提问，这非常有影响力。当然，你知道，他们想实习，想找工作。
8:56 有客座讲师是好事。不，我觉得他说的意思是。所以，好吧，我换个说法。不。所以，好吧，你知道，我们也采访过 Ange，他教 Coachella 课程。对吧。对吧。那门课就像是，这里全是名字。只有名字。对吧。而他实际上也在产出，你知道，这里是 200 行代码的 cloud code。对吧？
9:15 那是，没人让你那么做。没人会来跟你讲那个。但就像，对，当你想教基础知识和一个你能用的框架、一种工作理解时，这就是你会做的事。对，对，对，对，正是如此。我确实认为这类材料存在一种平衡，正如你所说，就是有一种，你不能，它不是 100% 固化的。
9:34 我是说，从设计上讲，这是一个特性，也许某种程度是个缺陷，但课程很大一部分确实从去年发生了变化，对吧？
9:42 所以整个课程的开发有一种动态性。但我至少想提出，有些东西我相信是会留下来的。是的，提示编码智能体，它们变得更聪明了。这在应对它们的新方式方面意味着什么？
9:53 但话说回来，Boris 也会知道一些我不知道的东西，对吧？所以也让他来桌边分享他的心得。这次我注意到的一件有趣的事是，相当一部分内容是关于开源贡献的。这背后的思路是什么？
10:11 你们有这些人……对对对。对，说实话，这是我最引以为豪、也最兴奋的事情之一。我是说，这门课马上就要进入第二周了。对，所以这些是非常慷慨的开源合作伙伴，我……这真是非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、
10:28 非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、非常、
10:34 非常、非常具有教学意义，说实话还挺简单的。我觉得对很多人来说，他们可能一次就能搞定，我就想，好吧，这其实没多大帮助。到了这个阶段，智能体已经普及了。我是说，我周二调查过的每一个学生，你知道，没有一个人还真的手打代码了。他们都说，大家都在用编码智能体。没有，而且这，你知道，就像斯坦福的大二、大三学生都是这样。
11:10 你调查他们了吗？市场份额是多少？是问他们用什么工具吗？好。我还没做，这主意不错。你应该做一下。对。对。我会做的。挺劲爆的，斯坦福更偏爱。不不。嗯，我觉得大学，我觉得他们会有一些，就像大多数供应商都有教育项目，对吧？
11:27 我是说，对，他们在学生套餐上非常慷慨。对对。所以我觉得默认情况下，甚至在开课之前，就有，你知道，Claude 许可证，我猜还有 Cursor 许可证。大概还有 Cognition 的许可证。我不确定，你可能比我更清楚。但除此之外，我还能额外争取到的是，嘿，如果我不想让大家被每月 20 美元的订阅卡住。
11:49 我希望你们去，如果你们想把 token 用到极致，看看能做到什么程度，那就去做。所以，我回头再说合作伙伴的事，但我能额外争取到的另一件事，不是去跟人谈，而是给学生额外的额度。比如 Factory 也参与进来了，Cognition 的人也参与进来了。OpenAI 也是。但不管怎样，关于开源合作伙伴，我大概是这样想的，好吧，在教育学生这件事上，我最担心的之一就是，如果大家都像 Suix 说的那样，说他们完蛋了，他们会怎么样？
12:19 几乎有一种他们被落下的感觉。而我认为导师制是提升所有人水平、确保不会出现巨大鸿沟的非常重要的一环——这个鸿沟存在于那些在编码智能体出现之前就已经在行业里、了解情况并采用编码智能体后变得非常出色的资深人士，和那些只能说‘好吧，现在我们该怎么办’的初级人士之间。
12:39 所以从某种意义上说，目标之一就是：嘿，编程智能体真的很强。我想把你们放到处理复杂代码库、大规模代码库的场景中。我们会和一些我认为是顶尖的 AI 原生仓库合作，你知道，它们都在做以 AI 为核心的事情，要么本身就是编程智能体，要么是在为智能体构建工具，你知道，在当今世界里。
13:00 我希望你们参与到这些项目中。我希望你们的作业有一部分是真正为这些项目做贡献。在参与所有这些项目的过程中，你们实际上会与维护者交流。他们会审查你们的代码。他们会帮助你们建立对优秀软件的思维模型。这样一来，实际上就实现了真正优秀软件设计精神的传承，我认为大多数学生没有机会做到这一点。
13:20 当然还有好处，我确信其中一些人会因为在这些作业中表现出色而被挖走、被录用。是的。顺便说一句，有意思的是，如果我不说这个会招来一堆骂。我觉得你说 cooked 和 cracked 是为了戏剧效果。你知道，就是喜欢把话说得极端一点。当然，当然。是的。所以也许我们应该深入讲讲课程的具体细节。
13:46 是的。在顶层上你还有什么想探讨的吗？我是说，这已经很有帮助了。是的。都在那儿了。好的。我有一些关于学生使用工具、最佳实践的高层问题。你支持吗？
13:59 比如个人导师。我猜这也会被学生采用。所以，你知道，在我们直接深入课程之前，你在这方面有什么指导吗。比如在什么方面。你说你对教育充满热情，对吧？
14:13 是的，还有一个污名是，老师、教授如何使用它？然后学生如何最好地使用这些工具？不是简单地把作业扔进去，一次性搞定。如果有什么建议的话，你如何还能保持学习？
14:28 是的，是的。我认为，还是会有一些作业放进去，我认为它们不是要成为世界上最难的东西，而是更关乎概念，更像概念性的东西。比如，你知道，skill 是什么意思？
14:39 也许更多是这些工具、技能的一些战术层面，比如智能体代码库，比如 agents MDs 之类的。所以有一定量的内容就是，嘿，如果你还没听过这些术语，先跟上进度。不过我认为大部分学习，在把工具付诸实践方面，会发生在开源项目和最终项目中，那个非常开放。
15:00 他们只需要去探索一些有趣的东西。酷。是的。所以也许，是的，我只想谈谈，好吧，让我们过一遍一、二、三、四，呃，到 10。是的。呃，然后我们再深入。所以基本上是广度优先。当然。当然。是的。所以我们从编程智能体的内部原理开始。没错。你教人们如何编写自己的 Claude Code 克隆。它不开源这一点很有帮助。
15:23 然后呢？对。对。所以，呃，第一周，全是，呃，harness 的内部机制，你知道，就是从零构建一个编码智能体。呃，第二周更多是，好，现在假设已经有了，我们怎么开始思考大规模下的上下文工程。所以超越单纯的提示词。然后我觉得与此相伴的是，在做这件事的过程中，思考工具设计、MCP。我觉得有一段时间 MCP 招来了很多骂声。
15:46 我仍然支持 MCP，我到处都能看到它。所以我觉得，在这一点上，它已经是一个存在的标准，而且我认为它会继续存在。所以谈论这个，有效的工具设计，那是第二周。第三周，我会说更多是关于技能，以及从简单的专用工具扩展到更像流程的东西意味着什么。那就是我们会谈论技能的地方。
16:06 然后这开始引向智能体的自动化。第四周大量是关于如何定制智能体的基础组件。这包括配置文件，比如 AgentsMD、CloudMD、钩子、子智能体。现代 harness 里有很多机制，你可以用于不同目的。所以我想让学生接触这些。智能体就绪的代码库。这大概是我最喜欢的话题之一。
16:33 就是，这对要进入行业并思考规模化的人来说尤其相关，你知道，本质上就是，拥有一个能被智能体有效消费的仓库意味着什么？
16:45 所以大量思考，你知道，设计服务级别边界的权衡，还有文档和可验证性之类的东西。所以那是第五周。对，第六周，算是智能体代码审查。这里我觉得有趣的是，也思考从第五周延伸出来，第五周围绕代码库，比如智能体就绪的代码库。到第六周更多是，好，为了，代码审查现在正成为瓶颈，我不知道学生们，除非他们，你知道，做过很多，他们中有些是硕士生，所以他们在行业里待过，但很多人没见过这个。
17:17 但对今天大多数使用代码智能体的人来说，我们都在经历代码审查中的某种新瓶颈。那么我们如何解除这个阻塞，或者稍微疏通一下这个瓶颈？
17:27 那里有哪些机制？还有可验证性之类的东西，以及我们如何把它扩展到代码审查流程中。第七周，安全。我的意思是，显然，我觉得有一个点。所以这在去年的课上也有，这个我想强调。我的意思是，任何自主系统采取大量行动，我们今年看到很多，尤其是很多东西被黑。
17:48 你知道，其中很重要的部分就包括 Vercel 的一些核心软件，比如网关，完全是 LM 网关。所以我想花点时间聊聊安全领域存在哪些新的攻击向量，以及我们如何缓解其中一些。第八周是后台代理。所以我想从这里开始谈的是，你如何基本上在许多不同的产品服务领域运行代理，不仅仅是在传统的 IDE 中，还包括基于代码库和仓库中在 CICD 期间触发的功能，甚至基于事件触发的功能，比如 SRE 风格的机器人。
18:31 你甚至如何构建这样的系统，我想过去这一年是很多公司开始发布他们的文章的时候，比如 ramp inspect。Spotify 有自己的版本，我想还有 Stripe Minions，所以所有这些公司都在写非常酷的文章，介绍他们为此构建的内部基础设施。我想谈谈这个，让大家了解那是什么。是的，这个更多是关于在现代世界中身处团队时会发生什么，以及团队级基础设施的一些核心部分。
19:00 所以从 LM 网关到 MCP 门户，也就是规模化的 MCP，然后还有如何思考不断演变的角色，我想说。所以这也许有点像软性的东西。就像现在是什么，你知道，而那种老套的、也许不那么细致的说法是，每个人都是构建者，但我想再深入一点，超越这一点，比如什么东西实际上正在被吸纳？
19:22 哪些角色实际上正在演变或新兴，而以前并不存在？学生应该思考哪些可防御的技能组合，而这些以前可能并不成立？
19:31 所以更多是团队动态类型的考量。然后，是的，这一切都在某种程度上朝着工厂发展，我认为这是现在很多人最关心的事情。一旦我们在构建工厂，人类还存在吗，诸如此类？
19:46 围绕人类参与，有意设置的新瓶颈是什么？比如意图，比如建立可验证的系统。以及在极限情况下会发生什么，当也许 80%、90% 的代码完全由代理驱动时，我们如何填补这一切。所以，是的。不错，扎实的 10 周。我本来想说，你知道，其中一些是比其他更成熟的模式。而且，我认为人们需要知道，并非所有这些都是已定论的科学。
20:18 当然。但它就是一切，我会认可这一切都是人们谈论的事情，你应该知道。我们知道开放问题，我们可能不知道答案。完全正确。是的。我是说，你可能对此有自己的版本。我很好奇你怎么看什么仍然不那么可防御，或者不是不可防御，而是不那么板上钉钉。但甚至就像软件工厂，我们是在谈论黑暗工厂吗？
20:39 我们说的是工厂里那种半自动化的灯光吗？我觉得那里也存在一个光谱。是的，而且你知道，那些处于最前沿的人，比如 warps 和那些像建工厂并卖服务给它们的工厂。而且，你知道，即使他们也只声称，即使是那种完全由智能体驱动的，也只有大约 30% 多。那么，在极限情况下这意味着什么？
21:00 很难说。有些人走得非常非常远，搞了黑灯工厂，然后出来说，我错了，伙计们。是的。对。所以，我们在 AIE 上讨论过这个。只要意识到不是每个人，他们当时可能相信自己说的话，但他们有权改变主意，而且他们确实改变了，甚至 Boris 最近也发推说，是的，我经常犯错，大家都说，因为他创造了 Clock Code，就像把他的的话当作福音，而我说不，有时他不同意自己，这没关系，这不是政治，对吧，我们仍然是科学家，我们是工程师，我们受制于，我不，我不声称自己是教条主义的，但我只是希望人们去实验。
21:40 我认为这也是一条原则。你之前问过原则。就是实验。我的意思是，你只需要准备好……而且通常，当我做这些讲座或跟人们谈论这些事情时，他们会说，有这么多事情在发生。我如何跟上？
21:52 有件很简单的事，就是每天实验 10 分钟。就花 10 分钟做一件你以前没做过的事。如果那是你过去解决问题的传统方式，现在有一个你想尝试的新工具或新框架之类的。就花点时间，甚至可能只是用那种新东西做完全相同的问题，就能弥补大部分差距。是的。
22:10 我要提一下，比如你的某些，比如你在谈论代码审查的那周有 Cognition。是的。我实际上和 Silas 一起做过 Devon Review。是的，是的。相当不错，我会说。这些东西很难基准测试。有一个 ReviewBench。没人用。但定性地讲，Devon Review 相当不错。嗯，我们用你们。哦，真的吗？
22:30 是的，所以 Monaco 确实用 Devon。而且 Silas 去年实际上讲过话，所以 Silas 很棒。就是超级超级聪明。但是是的，我们用你们。所以我就想，好吧，在我们用的那些代码审查工具提供商中，我们抽样了很多。我们实际上在很多用例上确实收敛到了 Devon。Silas 现在实际上在模型方面，然后实际上是你想要的 Walden。
22:51 是的，是的，是的。我也和 Walden 谈过。我还要提一下，你知道，来自 Factory 的，因为，你知道，谈到智能体就绪度，他们一直专注于此。他们构建信号、工具。你们用那个吗？
23:02 是的，没错。所以我们确实使用我们自己那套 agent readiness。我的意思是，Eno 也很棒。我觉得他们在这条路上起步非常早。我想当他在演讲中把它系统化的时候，我猜是在 AIE，那是什么时候，我忘了具体时间，但那是他的某场演讲。欧洲，对。对，欧洲，那场演讲非常精彩。我觉得它有点超前于时代。所以我就想，我能想到的最适合来谈这个话题的人只有一个，我真的很想把聚光灯打在他身上，非常期待。
23:27 是的，挺有意思的。我当时在 Cog。我们正在讨论这些东西，我们说，好吧，我们需要形成一些观点。然后就在我们讨论的那一周，Factory 就基于它发布了一个完整产品。我当时就想，好吧，你赢了。我的意思是，大家基本上都认同同一件事。问题只是，你想不想把它做成一个仪表盘？
23:45 因为它就是一个仪表盘。对对对对对对。但你知道，这是有价值的。而且我觉得他们从营销角度做得很好，抓住了 writings and report 这个点。我确实跟人聊过，你知道，我会经常向别人推销这个想法，说，对，你应该跑你自己那套 writings and report。当然，人们有它的开源版本，但我认为这些原则非常重要。
24:04 是的。你想一条一条来，还是你还有别的事情？我可以一条一条来。好，行。那第一条，对吧？
24:11 你重点讲 cloud code，但你同时还有 Pi 作为开源赞助。这是怎么回事？好，这一条可能稍微偏营销一点，但没关系。我确实想强调一点，我去年也说过，也讲清楚了，我并不为任何工具背书，好吗？
24:26 我不，我觉得那样对待这类事情是错误的方式。工具有来有去，而且说实话，去年这门课开讲的时候，Cloud Code 还没那么流行。它还没那么被真正接受。Boris 当时只有 4,000 个 Twitter 粉丝，现在他有，你知道，几百万吧。我甚至不知道是不是。他涨了很多粉丝。Cloud Code 真的，算是横空出世了。
24:49 我觉得。是的，我觉得我们请 Boris 和 Kat 上播客的时候，也低估了 Claude Code。对对。它当时才两个月大。我当时真没怎么用过。一次查询花了我 50 美分。我当时就想，谁会他妈用这玩意儿？
25:00 完全同意。它还在 beta 阶段待了一阵子。但我觉得他们能在模型还没准备好之前就把它做出来，这很了不起。是的。你会听到很多“为模型将要去的地方而构建”。完全同意。超级有野心。当时模型还没那么好。你感受不到。然后，啊，我们看到了，我们看到了。它就是从那儿来的。是的，挺有意思，因为去年他做演讲的时候，我记得那是一套非常高层级的幻灯片，但我记得其中一页就是字面意义上为六个月后的模型而构建的。
25:23 那大概是去年十月的事，我能理解如果他当时就知道六个月后 Opus 会达到什么水平，为什么会提出那个观点，因为我认为那也是另一个巨大的突破点，我甚至会说这是课程内容变化如此之大的原因之一，对吧？
25:37 当时围绕这些智能体真正的自主性，确实存在一场巨大的军备竞赛，比如长时间运行的智能体任务。但回到你的观点，这不太关乎——我并不是在给 Cloud Code 背书。我的意思是，Cloud Code 我当然在用，也用其他东西，但这里我更多是在谈，一个现代编程框架应该是什么样子？
25:54 其实我写过一篇博客文章，对，就是这篇，是今年早些时候写的，我在里面拆解了一些类似的原则。TLDR 是，你基本上只要具备入门级算法课程的理解水平，就能找到一个智能体循环，完成你所谓编程智能体的大部分工作。当然还有各种花哨的东西，没有子智能体，有 MCP 之类的，那些都没有。
26:18 但如果你想看到一个东西，你跟它对话，然后它真的会编辑你的文件，你还能用它做相当复杂的事情，那不需要超过 200 行代码。所以我在教育中尝试做的很多事情，就是剥掉那些像是秘方的东西，展示出它的核心其实——我的意思是，这些还是学生，你想教的是概念性原则。
26:37 这背后没有更多东西了，在斯坦福有一门课叫 106B，就是入门算法课。你不需要超过 106B 的水平就能构建这样的东西。所以你就去做。就是这样。既然你已经教过这门课，那有没有反对意见、问题，或者你从学生身上观察到的奇怪之处？
26:54 他们没搞懂什么？他们的问题似乎集中在哪些方面？你是指现在这个特定概念，还是一般情况？
27:02 这个概念。就是从零构建，然后做你做过的事。因为这基本上就是 Pi 所需要的东西。对对对，完全正确，是的是的。所以其实并不算。我觉得他们——我完全没有收到对这个想法的反对意见。我觉得他们其实更多只是——他们大多数是消费者，对吧？
27:21 大多数学生最终确实只是在使用这些工具。我不认为他们有很多接触机会，如果有的话，尤其是在课程作业中，去把这层洋葱剥开。我觉得，所以那一周是关于框架的。周二第一节课，后勤安排之类的，那些东西。但然后前半部分是，课程内容真正开始的第一块就是 200 行的框架。然后昨天，我做了后续那节课，我们实际上拿了 Cloud Code，然后你有这些代理工具，你需要代理工具来拦截请求。
27:53 MIT 和代理？对，完全正确。完全正确。所以用那个。用那个工具。拦截请求和响应。很好。看看底层，看看那里发生了什么。然后就像，好吧，我确实说过你可以在 200 行里实现 Claude Code。但真的，好吧，这里是真正的 Claude Code。所以我实际做的是，我拿了 Opus 5.5，这个才发布两天的模型，然后就像，这是——我拦截了那些请求。
28:13 这就是具体的情况。这是提示词。这是系统提示词。这是计划提示词。这是压缩提示词。然后我们就把这些全部过了一遍，并展示给了学生。所以现在他们真的拥有了一个可以理解的最先进的 harness。是的。能拿到这三个层次挺好的。学生很大程度上是消费者。这是循环中的基线 agent，然后是真正的那个。
28:32 我喜欢你博客文章还有视频讲解这一点。你觉得学生是不是处于某种极端，有些人是很厉害的高级用户。他们吸收了很多内容。对有些人来说，如果你了解 agent 是怎么工作的，你可能会觉得这很直截了当。你在课堂环境中是怎么平衡这一点的？
28:53 比如你会针对平均水平来定制吗？是的，我觉得这有点自我筛选的性质，因为课程内容都在这里，对吧？所以课程从，我是说，你知道，至少几周前就已经可用了，我猜我几周前宣布了它，但它大概至少已经筹备了一两个月。所以如果你去找，你大概能找到。
29:12 而且我觉得现实是，如果学生对此不感兴趣，他们就不会来，他们就会选择不来。我觉得有意思的是，确实有学生会来，但也有很多来自斯坦福更广泛范围的人，其他院系的博士后。有很多跨学科的兴趣，比如在病理学或类似医学领域的人，甚至还有其他非纯 CS 的工程领域的人也会来，也对接触这个感兴趣。
29:41 Well, also a lot of, uh, GSB students, MBAs, um, that also, uh, want to kind of learn how to not have a technical co -founder or whatever the motivation there is. But so I think, by the way, if anyone hasn't done the MITM proxy exercise, you should do it. Yes. You'll be shocked. Cloud Code has the longest system prompt of all the coding agents. Something like 20 ,000 tokens. Yeah. Yeah. Although, yeah, that is true.
30:07 That is true. Although they've also reduced the size. They cut it by 80%. Yeah. Yeah, I know. But then they grow again. And then they cut and grow again. It's a seesaw. Yeah. Yeah. No, I think it's interesting. You'll be surprised what goes in there. That actually sometimes you don't use. Sometimes it actually hints at the internal workings of some of the tools. Anything that you found there that could be cool to talk about?
30:28 Yeah, no, I mean, in fact, yesterday when I was talking about this, it was a little bit of this like, one of the things that I thought was most interesting about Cloud Code back in the day, and I think part of why it really blew up in terms of popularity was their focus on simplicity, right?
30:41 So in the early days, they're among the first to really lean into this, lets fewer tools do more, right? Lean into SQL, lean into Bash, lean into the things that are powerful, like Swiss Army knife tools, and have that be a lot of the driver of capability. I think they did a fantastic job leaning to that. I think that's why they were as popular as they did. I think that when I presented something like the system prompt, it was like, well, there was a different system prompt about two months ago.
31:05 It is now, they have reduced in complexity, and it turns out that now what was once part of the system prompt, things like how to use even Git, right?
31:13 or how to set up tasks or things of that nature are no longer part of the system prompt. It's actually expected to be more captured in the models, like the model weights itself. So they've kind of baked in a lot of those learnings more at the model level. And that just raises interesting questions around the model versus the harness, right?
31:26 And how do you kind of think through that trade -off? What do companies invest in? What should you invest in to really push the frontier of your own capabilities?
31:34 And so I think we still, even as a community, we see saw between the two a lot. I think you don't want to fault them for growing a system prompt. And then cutting it. At some level, it's also how the model was, right?
31:47 Weaker, older models, you need to give them examples of this is good, this is bad. And then obviously you cut that away at some point, right?
31:54 And that's just part of learning what has been done, right? You can apply the same towards if you're using a local coding agent with not the strongest model, you might need to add these examples, right?
32:05 That's right. Is that correct? Yeah. Yeah. And then, of course, there's interesting AI learnings to follow out of that, which is why, you know, as the models get better, when are you over constraining it by adding those examples, right?
32:16 And then how do you pull away the over constraining? And that's, I think, part of the learning as well. Okay. Week two, you're going to talk about context engineering. Repit, I haven't seen, but there's many versions of Spectre of a Development. What would you, would you highlight to a broader audience about what you're putting in here that people should know?
32:35 Yeah. This is like the, you know, one hour version of the course. Yeah, exactly. Please condense everything. I think that's where this is going. That's fair, yeah. Well, no, it's also a trailer, right?
32:47 It's like, well, if you want more, go to the source. If signing you're up next year or next fall for courses, you know, this is your one hour. This is like, this is like, oh, it was like the Quibi of the class. You know what I mean?
32:56 It's too long to be a trailer. It's too short to be the class. Is this like, what I guess people will double click. If it's a trailer, by the next time you teach it, it will change again. That's true. That's true. That's true. That's true. Yeah. Well, yeah. So it's okay. So good call on that. It's funny. I think that the main thing that I just want to reinforce here is, and I see this even when I'm like interviewing folks, very few people in industry even progress beyond like just vanilla prompting, right?
33:21 There's like a form of like, hey, do me the thing, right? Like do the thing. And that certainly as the models get good, that can carry you a certain distance. But inevitably, any folks that I speak to that come out of a Google or someplace where there's a massive code base, the reality is you have to be a lot more thoughtful about how you're ingesting context. So thinking through what are some of those principles and those techniques around how to point ...
33:46 The rapid framework that I'm describing here is functionally similar to like DEX's RPI, right, or something of that nature, where you have this like research, kind of proposal, sort of a spec, and then you do like your implementation, and then you kind of tie it together with some testing and whatnot. So what I just wanted to reinforce here is that like you should have this step, when you're entering unfamiliar code bases, when you're trying to sort of infuse into the context something that an agent needs to understand about a very large code base, there's value in something that is a research phase.
34:16 And then in the case of this particular sort of framework that I'm putting forward, it's like also then suggest some proposal where there's a few alternative ways of solving a problem once you've done the research. And so this is where I sort of also invite the human in the loop to sort of be the, decision maker on one approach versus the other. And then you enter the plan, which is kind of the spec.
34:35 All to say that the, the TLDR is like, there is a way to do this that actually does scale out to code bases that are, you know, millions of lines long. Um, and it's not just the vanilla prompt. Uh, and I think that's what I want to highlight. if you're going to go working these big code bases, you should use something a little bit more involved. Anything to add on context engineering?
34:54 I think that probably the weirdest thing about how I think about context right now is when you split out to a subagent because subagents can benefit from inheriting your entire context of the main thread or not. Sometimes there's a regular disagreement on that. And then sometimes you want to condense what the sub -agent does and researchers put it back onto the main thread. And it's very hacky. I don't like anything I'm doing.
35:24 So I don't have any answers. All I know is when you want to fan out work, it is not just about you have a big spec and you RPI, you do Repit, whatever. It is not one agent anymore. It is about fanning out work and making sure they all have the right context. And they all have the right guidance. If you have a review agent reviewing all the other works, you have to make that guy have everything.
35:48 And you always go through this phase of expanding contracts. Like, oh my God, these guys don't have enough context. I'm going to give them more context. Okay, great. And then they're going to over to you into some random one -off comment that you made a while ago and then make some really stupid mistakes because you didn't mean it that way. But like, they took it as the law. Something like that. So I don't know what the word is I'm saying.
36:09 It's just like multi -agent management for context. Yeah, yeah, yeah. Yeah, I think that's absolutely right. There's like some aspect of this is maybe the part that's more maybe among the most art than science, right?
36:21 It's like some combination, I think, to your point. Yes, the fan out for the exploration, the fan out for like the subtasks, and then the compaction, whatever you might want to call it. I think being thoughtful about refreshing of the context, understanding when things are getting a little bit too bloated. I think Dex had this concept of like the dumb zone and sort of like being a little bit more thoughtful about the dumb zone.
36:39 He's good at these equating things. Oh, yeah, yeah. The little diagrams. Yeah. Oh, yeah. Why do you have him on? Yeah. Well, I think he's, I mean, he helped definitely informed a lot of the curriculum here. And so he's a fantastic thinker in all this. No, I think the diagrams are huge. That's useful. I would add breadcrumbs for people that are trying to look into this. It's good to see best context engineering preferences, like on your sub -agent inheriting.
37:02 Cloud managed agents has a version where sessions are separate and it's not inherited. This used to be a big topic before reasoning models when agents didn't run as long we had shorter context windows everyone was very very managing that precious context window right link chain used to write a lot of blogs about this in good context engineering all this stuff is good for people to know i think in the student context as well you can expect the average student to take your class because it's a great class and they'll know all these things.
37:35 So you should know them, know them better as well. This is a fun space for research as well. One of the papers, research papers presented at AIE, they looked at the delta in the cloud code system prompt, what got changed. And they plotted a whole bunch of changes in performance. They tried to reverse engineer it. It's just good research that you can do. There was another one that also looked at context window management and how it affects performance downstream.
38:03 They weren't the best papers, but it's quite accessible and it puts out good blog posts. So people that are trying to work on it, I just recommend it. It's a good area you can easily poke into. Yeah, good shout out.
38:16 Agent skills, CLI. Yeah, yeah. I think all this is a little bit of an extension from a single kind of function tool calling, like MCP, interacting with third party external services, and then extending into what you would call procedures. How do you automate some set of actions over and over again that you then formalize in the form of a skill MD or something like that?
38:38 think just making that a little bit more, giving people exercise in codifying some of those procedures. I think what I'll also include here is a few fun skills that I think are good to use. think Matt Peckock's grill me is a good one. And then there's ones that I also like more from the educational perspective, like Jeffrey Litz. He has this explain diff, which I think becomes especially interesting once you start talking about code review.
39:02 Because, you know, the whole... Understand the code. Understand the code, exactly. Yeah, you've got to understand the code. And I don't know if you guys have used that, but effectively, like, you run, you know, you have your your, your kind of, the, the, the diff you've put in your code and then you run the skill and at the end of it, you have this, not only like kind of a visual representation of what's happened, so you can, you know, kind of skim it and understand what's, what's actually at the core of it.
39:22 But then what I really liked was this almost like quiz at the end of, uh, at the end of it where you go through and sort of like really reinforce whether you understand what's happening. And I think that, that level of interactive sort of back and forth I think is, is quite nice. Yeah, there's a couple blog posts in this domain that I really like to bring up. There's one, I think, recently, which is like kind of, there's no situation in which turning your brain off is a good idea.
39:45 And I think these skills actually force you to turn your brain on. And when everyone wants to use coding, you just turn your brain off. They're just like, make me an app and read my mind and get everything correct without me telling you about it. That's right. But a lot of these skills are mostly in the genre of like, no, no, ask me more questions, please. I think there's a lot more output than people are used to.
40:07 Like I can type in a one line prompt and get tens of thousands of lines of code and a whole structure. It's like, look at my productivity, guys. But, you know, more effectively, you should know what you're working on. You can really be put out of touch, out of the loop. You should spend more time thinking about what you're actually doing. Doing the work and getting the code written is much easier than it was.
40:33 Yeah, the understanding there's no excuse for, for sure. Sorry, what is a web skill? Yeah, so I mean effectively like skills that interact with the web itself. Computer use. Computer use, yeah, things like this. Anything else?
40:45 I mean mainly things that involve like pulling information from the web. Are there best practices on skills that will interact with the web versus?
40:53 I think this is more of like a bit of a survey or kind of a demo of stuff that I found that was kind of interesting. I think it might have been out of, was it like a Vercel?
41:02 I'm trying to remember now. I'd have to look back at the full reading materials, but there was one site that I found where I think they, maybe it was, it might have been BrowserBase, actually. They might have done. Do you remember this?
41:12 Well, it would be them that coined this thing. It's just, I was like, did I miss something? This is a big part of the skills community. He's good. He's being educated. Yeah, I think they made some sort of, because I guess they were, yeah, I mean, BrowserBase, obviously. They like browsers. They like browsers. But yeah, they had a good, I think, website where they serve some of these interesting capabilities. Seems legit.
41:34 CLI is a fun one. Do you have a lot of students where this is their first time in a terminal in a CLI, or is it kind of the norm now?
41:41 No, I don't think so. I think that given that most of them, again, last year I would say that some of them had probably still remembered the days of finger -on -keyboard typing.
41:54 Most, I think, students have interacted with Cloud Code at this point. I mean, I'm curious the split of people that are now just using the apps, right?
42:02 Cloud Code also has a desktop app. So does Codex. Codex is primarily an app now. You're kind of not in the CLI, not in the terminal. It's a thing that puts off a lot of average people, but just random note. Yeah, I would say, know, I think there's also another thing about skills that are bundles of skills, where they sort of cross -reference each other. Yeah, like the plugins, right?
42:26 like plugins. then there's also, I guess, the art of writing a skill, which is a thing that I've also focused on for AIE. I think a lot of these are focusing on skill consuming. And you have all these influencers like Matt Polcock and what have you, they're like, yeah, this skill will save, it will solve all your problems. But no, you should write your skills yourself. And actually, when you try to write them naively, they're usually bad.
42:51 And actually, it takes a little bit of skill to write a skill. That's absolutely right. That's absolutely right. What are swyx's best tips on writing skills?
42:58 I have a public skills repo. I went through definitely a phase of psychosis of skill everything. No, no, no. Don't give us skills. How do you write good skills?
43:06 Yeah, exactly. So I have a skill for cutting skills. So every cloud code and codex, they all have skill creation skills, right? So then I have a skill cutter skill that cuts it down because it just documents every way in which those default skills that are created are bad. So you always run this after any skill is created to cut it down. Nice. Right?
43:30 I sound like this. Yeah. really annoying because sometimes there is slop, but there's good in the slop. And then you have to sort of whittle down what there is. Right. Oh, interesting. As you're saying, even the slop, it's still in the language of the agent. So the slop means something. It's like a, yeah, okay, got it. It's fun. Everyone has their own version. I used, first, there's very easy to like vibe check a skill, right?
43:54 You just call it, run it a few times. Yeah. And then I distill these preferences into a skill that's like, this is what I like, this what he doesn't like. And it adjusts over time. And then, you know, it can automate. So the problem that, yeah, you're calling it, but the actual, the worst part is when a skill triggers when you're not expecting it to. Like, you wrote it one time to fix the thing, and then, like, now you're working on a completely different thing, and it triggers.
44:18 Why? Because it's barely written, because the trigger says, like, oh, you call me whenever blah, blah, blah happens. And you're like, okay, read's good. And then you do a different project, and it triggers, and you're like, oh, goddammit. Yeah, what the hell should happen?
44:29 There's a few people that are just like, okay, I have a bunch of skills I keep adding on, sometimes run this up. And I know some people, like Eugene Yan, they are pretty particular about how many skills they have. They prune them. If not used, they clean them every few weeks. Yes. So, yeah, this is a huge problem if you're in that skills world. I think this is actually why I suspect they're very smart people at OpenAI and Anthropic that want skills to die.
45:00 which is not a popular statement. They won't say this publicly, but just feels sloppy. This feels sloppy. I see my own skills people being sloppy. It doesn't feel AI -pilled. Do know what you I mean?
45:14 Yeah. It's not what AGI was supposed to look like. It's like everyone having a giant bunch of markdown files. I mean, I think part of it is like... There's a gardening problem, yeah, in some sense. A lot of people actually don't look at their Markdown files, right?
45:28 Like the UX of it, of just chatting what you want, creating a skill in chat, using it, seeing output. There's a lot of value you can get without ever looking at your Markdown. Don't overcomplicate it at some level for yourself. You can get a lot of value. If you do stuff repeatedly, just be like, look at the last five times we did this, make me a skill. And you can iterate it with chat.
45:53 There's a good side of it as well, right? There's another thing. There's another way to look at it as well. There's the use of the skill as just procedure that the agent runs whenever to do a code review or whatever. But then in the case of something like Devon, where we've codified skills as a way of representing our best practices and then exposing that to the agent or the code review system to then institute the APIs the way that we like to define them, institute the service level bound is the way we define them.
46:22 That's like an alternative way of sort of representing like org level knowledge and maybe not so much this like arbitrary thing you want the agent to do so you don't have to write the prompt each time. And so I like that. I feel like that positioning might, that purpose might exist longer than just, you know, these kinds of random procedures.
46:42 I just Googled Monaco AI skills and it's giving me the country.
46:48 one day one day well we're going to beat them on AEO that's for sure can you just buy the country well here's the thing okay on this we we're about to no we're not about to buy the country but but but you know we did we did get into a little bit of flack when we bought the monaco.com domain I will say that we did get sued by them so what yeah there's there's like a whole public there's a whole public thing about this where yeah so they didn't like that but here we are and we won.
47:17 We won. Okay, then we're keeping it in. I was about to say this is spicy, but you won. Yeah, yeah, no, no, no. Is that why we see the flags flying around?
47:27 Because it's like a victory. It's because it's a victory. It's our victory. That's funny. For those that are in SF, just look up and, you know. Yeah, so for those who are on audio, we're showing this on YouTube, you fly a plane around. I think you're still one of the few that actually do this on a regular basis. Sam Blond came up with it, $6 ,000 a day. I don't necessarily think it directly causes me to Google what Monaco is, but all of Monaco's ads are just Monaco dollar sign.
47:58 Right. We stand for money. It starts with dollar sign and then X number. It starts the conversation. The whole point of the ad is, what the hell is Monaco?
48:07 Yeah. That's a debate to be had, but in some sense, what is the purpose of marketing if not to get attention? right and and yes maybe there's one level beyond this which is if somebody so you know i'll go to events and people like oh you guys are the ones with the planes you guys are the ones with like the big number on the billboard blah blah 193 you know million whatever it is um and and that but they're like but what do you do and then but that's that's but that's but maybe that's the whole point right is that i just needed someone to come and use it as an icebreaker i will say it's what i said as you walked in as well it's like ah the planes yeah and now what do you guys Well,
48:42 let me tell you about what we do. As someone who has been in those meetings deciding what we do on the Devin billboards, we spend so much time on the words. And maybe we should just be like, Devin. I think you do it. Which is so fucking dumb. But like, okay. You know, look, I think Sam had another post this past week where he was just talking about like, I mean, Sam is effectively like our, you know, marketing sort of mastermind here.
49:07 And, uh, yeah, I think there's a little bit of history behind the planes as well. But, but the TLDR is that, his whole shtick is like, I don't have all the answers. You just gotta try things. And I think a lot of this is, we just try things. And, and if it sticks, great. If it, like, did we know this was gonna blow up as much as it did?
49:21 I mean, not really. But, but now that it did, it's an idea. Yeah, exactly. And minimalist start small, go high, instead of start big, go small. Right. Way easier to start small, go big. Yeah, exactly. Also, you know, it's 6K a day. Like, it's very much just try things. I think people... Cheaper than a billboard. Cheaper than a billboard, for sure. Yeah. And it's fun. Not much harm done. But, you know, the kind of simplicity here that's worth mentioning is then when we're trying to, like, one -up the plane, we're like, you know what's better than one plane?
49:50 Two planes. Two planes? What? My way to rocket. Well, okay. I also had that idea. I would like for that to happen as well at some point. But we'll see. This is the next SpaceX launch. We just see Monaco. Yeah, there you go. For the shoot with the stars. There you go. There you go. Got to warm up to our SpaceX friends. I'm happy. You don't want to?
50:15 Well, I mean, we're on Monaco. Is it worth talking a little bit about Monaco? Do we leave it to the end? Yeah, yeah, yeah. Because there's just like an intermission between the topics. You decide. Okay. We mentioned Monaco, right?
50:26 Like, let's give... I think it's a little bit annoying to mention what the hell is Monaco and then not answer it? Sure. Can you answer it?
50:33 Sure. Well, I hope so. Yeah. So this is the country. So it's a small little, I guess. There you go. There we go. There you go. Okay. This is Monaco. This is the Monaco that I know and love. Yeah. So we're a Series B company here at Basin SF. We are building effectively like an all -in -one sales platform. Fully AI native agents doing all the fun stuff. or at least the stuff that humans don't like to do but that agents like to do because they don't really have opinions do a less air -prone fashion so that salespeople and sales teams can focus on doing what humans are good at which is closing deals.
51:12 So presumably Mark Benioff doesn't like you guys. Probably not. But he's also honestly a very AI -native person, very aggressive in agent force and everything. What are you doing that he doesn't do or he cannot do?
51:26 Well, okay, so right now we are, we started the company in a sort of like, the market that we went after was not enterprise from the get go, right?
51:35 We sort of said like, we're going to start by, like if our goal is to build the all in one, then we need to kind of build this in an irritative fashion with a group of customers. And we need to grow with them into this. It's much harder to pull someone away from Salesforce than to say, hey, if we meet you at the point when you are still just a Google spreadsheet tracking pipeline and we show you a more AI -appealed answer to the problem, then we can grow with you.
51:59 In a way, it kind of follows the same model of Brex or all the YC companies where you kind of sell to your batchmates, you kind of grow with them, and then you go upmarket. And so I would argue, are the, you know, I would claim that among the best, if not the best sort of like startup focused sales platform. And so all of our customers are like early stage startups. At this point, probably through like series A is kind of where we've primarily been focused, kind of seed to series A.
52:24 And then now we're going more upmarket. We've kind of learned a lot about the all in one in the last year and a half of doing that. And you are the head of AI?
52:32 Yeah, that's right. What's some cool AI stuff you guys have built? Yeah, I mean, we have built our own Agenda harness for sales use cases. So all this, I mean, to call back to the coding harnesses, taking a lot of inspiration from the systems like the codexes and the clods of the world and sort of building bespoke tools that are very sales focused, bespoke workflows that are all kind of geared toward GTM use cases.
52:55 So we built our own version of that that powers pretty much every single thing that powers the monocle platform. So from scoring of TAMs, scoring of ICPs, and defining of ICPs, through to pipeline evolution, through to all the mundane kind of things that maybe a salesperson doesn't want to do, which is like, oh, update my meeting, pre -meeting notes, update my summaries, all these things. We have agents powering all those workflows.
53:22 So we've done that. It kind of comes out of the box. And yeah, so long horizon agents and short horizon agents in some cases for GTM use cases. And it's pretty domain -specific, built on foundation models. Would you do any of your own models for this?
53:42 Yeah, it's a good call -out because so far, I mean, I think this is generally true of a lot of like vertical AI companies. It's like you kind of start where you sort of use the frontier models like as third -party services to see what's possible. because they are like the very frontier. And then I think we've seen this, I guess, in the past week with the whole like sort of where they like the plus 50 % to minus 50 % margin kind of question.
54:03 I mean, you know, I think in irrespective of the numbers, I think what is interesting is like at a certain point, things do become interesting from a cost perspective. You, you start to wonder, can you do it faster on, you know, open weight models?
54:14 And then that's a lot of that. Then a lot of what I think the purpose of, of the AI pieces are is just to posturing those models, build environments for, you know, kind of to posturing using RL and things of that nature. So, um, we're kind of in that phase right now. Ooh, fun. Yeah, I've been calling this playbook the Agent Lab playbook, but that's kind of my coining of like, you know, agents start from the user first and then they use whatever models you need to build products.
54:38 But then you can backfill the model side once you have the user base. And then the model labs kind of go the other way. They start with the big P trains and then they try to find users. And it's like, you know, it's hopefully working as a split. The monocle would be on the sort of revenue GTM side. That's right. Yeah. And I think it's interesting. They would want to serve you while also try to compete with you as monolabs.
55:04 That is correct. Of the things that I've been, I dabble in some of this. I think one of the hard problems is actually enrichment.
55:14 I would say I spend a lot of my time doing that for basically SEO and research and what have you. And I imagine for you, it's actually a core part of that. That's right. Do you find that models are good at it?
55:29 Do you have any vendors that you like working with and you want to give them a shout out? Yeah, absolutely. So we were adopters of Parallel. And I think they're very strong. So just general search. Yeah. Not like Zoom Info or some one of those guys. No, no, no. But yeah, we use Parallel. We also use Exa. So we'll give a shout out to both of those folks. Can you compare them?
55:50 They're both friends. They're both friends. They're both folks that we like share. to You'll be very interested if you like one of the other. Flip a coin. I'll keep it. I'll keep it neutral. I'll stay Switzerland in this today. Yeah, and then of course we do our own version of enrichment as well. So it's like most of our data systems, I mean, part of like one of the most valuable pieces of software that we built or most valuable systems is like our world database.
56:13 Like the database of companies, enriched companies, enriched accounts, enriched contacts. So like that whenever we are constructing ICPs for folks, we have the most up -to -date information about who you should reach out to at which company at this point in time. Like I would say that one of our, like I think where we are like world -class hands down is just demand. Like we are a very good demand company. Like we help companies increase their meetings booked in a way that actually.
56:36 So find buyer at? Find buyer at the right time, right? And it's kind of like a combination of those two pieces. What do mean you at the right time?
56:41 Like there's times when, like here's an example. Like if you are targeting someone that just, are selling to VPs of sale, let's say. The right time to reach out to them is not when they are in the middle of some, like they've been at the same place for the last five years. If they are now in the middle of a job, a good signal is if they've gone from one job to a new job, they're now entering a new place where they're more likely to be to purchase or be open to a purchase of a new sales platform because they're in the middle of that job transition.
57:11 So going from company to company B for VPs of sales, a good signal of like, hey, should you reach out to them at this point in time is did you detect that their LinkedIn changed or something like that?
57:19 How much of this is your expertise put into model and harness versus models are good, there's generic best practices? Like, is there, there's obviously some value in both. Yeah. Is it just the glue getting it reliable, doing it all in one platform, doing it well?
57:36 It seems like there's also stuff that we wouldn't know that you guys are also domain experts in that's baked in. Yeah, I think without the former, it would be like a fully commoditized. I mean, not fully, because there's still a lot of value in the all -in -one. I think that is something that we definitely pioneered. Most companies in the sales space started as just point solutions, and I think they tried more of the YC approach of you kind of land and expand.
57:59 You kind of pick a niche, and then you expand from there. And we, from the get -go, I mean, that's why we stayed in stealth for the greater part of a year and some months, was just building, because we knew we wanted to be the all -in -one from the start. But that alone is powerful. It's meaningful. But I think the real power is the fact that today, Monaco as Monaco has a world -class go -to -market team.
58:16 Like a truly world -class go -to -market team, selling Monaco of course. But we also have the same go -to -market people that sell Monaco are the ones that are what we call forward deploy account executives. They're FDAEs. They're effectively like human salespeople that are like consultants of sorts that help onboard all of our customers, get them building agentic workflows, and are effectively there to help guide them through every part of their own sales go -to -market lifecycle.
58:39 And so your question, the all -in -one yes, but it's also the fact that we have these like, you know, Sam and we have brought on a lot of like former Sam's members and stuff like that who were like the best people in GTM at Brex and other places. So we're codifying all of that in the platform. Yeah. Interesting to have you talk very differently than a normal head of AI that I might also talk to because you're also a former founder, right?
59:03 And like you're, you're also think, used to thinking about these things in this way. And it's like interesting to, it's important to have that, uh, in your role. Yeah, cool. Uh, I, I, I think you've, you've introduced Monaco enough there. And then, and obviously I think as the Monaco, the story develops, we can, revisit. Maybe the bridge back is also what is coding or agent -native coding or modern software development like inside of Monaco?
59:31 I think when you're teaching, you're maybe trying to be, here's a lay of the land, but maybe inside of Monaco, you guys are more opinionated on one thing versus the other?
59:40 Yeah. So we've made investments in coding agents from basically like day one and we've like learned a lot. We have our own like scars that we sort of like picked up and learnings. And I think we've now been getting much deeper sort of sharing some of our learnings. So like in our blog, which might be worth sharing here, we've been increasingly posting some of what, uh, maybe not that. So that one is more of like a product for the one at the very top of this stuff.
1:00:03 I would say like, yeah, when AI gets desperate and before like scaling contributions, build or buy SRE. So, you know, we spent a lot of time looking at different providers and vendors and men are of building a lot of things in -house. So for us, something like a background agent platform, we have our own version of like a ramp inspect that we call Monocoder, which is built on the Coder platform, but it is what we use to essentially scale up our engineering product.
1:00:26 Coder, like Coder.com? Coder.com, yeah. Oh, OK. We actually have sort of adapted it to some of our use cases. And it's where a lot of our agent development happens. It's just this really well -designed piece of software for essentially isolating of sort of development environments, scaling out to multi -agents, you know, being able to run these like long, long horizon agentic coding agent tasks. Monocoder, shout out to John Tracy and co.
1:00:50 for building that out. And then, yeah, like from there, we've also built our own versions of like a, you know, like a Slack bot, which is HAL. But effectively Hal is like this part AISRE, part customer success, part QA engineer, because it lives in our Slack, but it serves all the purposes of helping us debug things and put up PRs when things are broken, but also used by our own salespeople to find out how the product works, find out how to get to the bottom of something.
1:01:21 So there's interesting questions we've had to answer around the different personalities that Hal has, depending on whether it's talking to a salesperson versus a developer. And so systems like this that we built out over time a lot in the last year or so. Shout out to Yunja for Hal and John Tracy as well.
1:01:41 So you're building a lot in -house. Yeah. You also use other tools. We do use other tools, yeah. So what is the line? Yeah, so. Build versus pi. What was that?
1:01:49 Yeah, the Build versus pi. It's a good question. And we, I think there's this, like a thing, a general principle that I generally, which is like build to learn and you buy to keep. Or sorry, buy to learn and you build to keep. Yeah, the other way around there. So you buy to learn and you build to keep. And I think that's generally true here, where in the early days when we were still trying to figure out like, you know, get our feet under us a little bit, we were, you know, code reviews, an example one of them, where we spent a lot of time looking at different code review platforms.
1:02:18 It turns out today we still use Devon because we've actually still find a lot of value in how the system's been built. So, you know, shout out to the cognition team. But then along the way, we did try out some like SRE tools and we tried some sort of like off the shelf, maybe more like background agent platforms, decided that it was just given our infrastructure, given how we had set up our own cloud platform, it just made more sense to kind of build this out internally.
1:02:37 Of course, we're not dogmatic. We are open to revisiting them if something comes out. But here, I felt, I think usually we ended up building in -house when we felt that there was no like mature offering out there. So CodeReview is maybe like an interesting example because we did do a whole survey of CodeReview tools like maybe over a year ago. And at the time - And I'll shout out like CodeRabbit, right?
1:02:59 Yeah, CodeRabbit and them. And there's also like CodeRabbit, Graphite, Graphite for a while before they got acquired. Like Ellipsis, which like a YC company before the fact. But there's a bunch, right?
1:03:09 I mean, so there were a lot of folks that we did try. And they were okay, but I don't think they had crossed the threshold of like they're immediately useful yet. And so then we kind of like, we sort of like put it aside a little bit. This was, you know, I'm saying like well over a year ago. Then we kind of went back to this and sort of organically, I mean, again, shout out to the Devon team and Cognition folks where we tried it again.
1:03:28 I think I had sort of gotten it that somebody told me, like, hey, you should try this. And it seems like it's gotten a lot better. And I've been actually doing Silas' talk during the class last year. And it was just good. And it was picked up organically. Like, like, that's I think. So purely vibes, right?
1:03:41 Like, this is there's no way that you know this. There was. Yeah. This was like, I mean, okay, okay, here's what I'll say. when we did that first evaluation, the first time around, where we didn't really pick any vendor, um, and we were like, there's no vendor that we're super happy with. I did do like an analysis where I looked at our PRs, basically did sort of like a, you know, set of pie diagrams, like here's what this is solving.
1:03:59 Are we, are we happy with the kind of the tradeoff of verbosity and how often it's posting with utility. Um, and so there was, it was more statistically driven than just vibes. The second time around, I wasn't going to push hard on it. The first time I was like, hey, we should adopt an AI code tool. Second time I was like, no, I'm not going to do that again. But then organically people were just like, hey, Devin is good.
1:04:17 Can you add me to this? And so today people are just getting organically added. So is a vendor one of the cases where we're still leaning to the vendor. We'll see in the long term what that looks like. But everything outside of that, we just didn't feel like maturity was there. Yeah. And it's not painful enough that you would revolve Devin and write you on. That's right. That's interesting. Yeah. It's a good good talk for devin i didn't even know uh you know for the record i'm only an advisor i'm not that involved anymore but like it's interesting to to hear about it externally because internally uh there is a certain amount of let's call it um the the bubble where like on slack you're you're like you only hear positive things because people only want to circulate positive things uh but like you you have no dog in the fight like you just choose I mean,
1:05:04 if you want to get us free credits, that's totally that'd be cool too. I wouldn't say no to that. I think Silas will hook you up. Okay, so there's that, right?
1:05:14 And that informs all your teaching work. Yes, that's right. I think that's really good. And we can bring in more of this as you like, because I think this lived experience is very good.
1:05:25 I'm happy to go back to agent -ready code bases. So week four and five, kind of matching. Yeah, they blend a little bit. Yeah. Blend a little bit. That's right. Whatever strong beliefs you have introduced in Monaco and recommend people to start with. Yeah, no, for sure. Yeah, week four and five, I would say they are a little bit, four is maybe a little bit more like conceptual foundations and ways to do this, because there are a lot of ways to customize agents.
1:05:53 And then I would say week five is where I really kind of lean into, all right, let's think about this and apply this in practice on a true code base. And so here I will definitely give a shout out to the factory team and want to just highlight the value of something like a readiness report. I think that the main piece that I want to reinforce, something like agent -ready code bases is to me like a precursor and a bit of a building block toward everything to come afterwards.
1:06:18 When I say everything to come afterwards, I mean like people always got, you know, for a while, and I think people still are like super hyped on like multi -agents, like paralyzing agents, like tens of agents doing all kinds of things. It's like you actually need to set a foundation in order to do that well. And the foundation is how verifiable is your code base, how well -structured is your code base, how understandable is it by an agent.
1:06:36 And so I think agent readiness to me is almost like piece one of the next four to five weeks where maybe until like week eight or nine where spend more time talking about broadly AI software teams and the kind of org -level structure. But definitely building up the building blocks for background agents, the software factory. I think this is where it all kind of gets interesting from that perspective. Yeah. One thing that I observed you talking about for Monaco, but it's not here, is the SRE side.
1:07:05 When you close the loop, you actually want to have the agent read your logs for builds and then ideally read your analytics for live production. That's right. That's right. Yeah. Not explicitly, but it's a really good shout out. So I think that I'm going to weave that in in like two places. One is in the background agents week and then also in the software factory week. So absolutely right. It's a really good callout.
1:07:27 This idea of trigger slash event -based closing of the loop, which now is one of the probably two or three entry points when people talk about agents kicking off tasks in the software factory. That's absolutely right. That'll come in there. Yeah. It's interesting that people call it trigger or event -based, which I guess that is the actual technical implementation of it. Yeah. It just seems too low level almost. So the trigger is usually a cron job, right?
1:07:54 Let's be real. Sure, sure, sure. Event -based is usually a cron job that surfaces an event and then it's event -based, but it's all crons. Yeah, yeah, that's right, that's right. Yeah, yeah, yeah, or like web hoax or something like that, but yeah, that's absolutely right. Yes, and the technical term, of course, because at the end of the day, yeah, being sort of like a CS person by trade, and yeah, we call those things, and so I guess I fall back on that terminology.
1:08:19 Yeah, and have any AISRE companies made an impression on you? So, okay, I mean, like, I think the Resolve folks are doing well there. The Resolve is doing well. I would say Night Shift is also doing quite a good job there in terms of like, background agents. I'm a big fan of kind of, like, they're Greylock backed. I think they're doing, doing well in this area as well. Yeah, I, I would say that, where I think SRE is an interesting use case, and this might back to the build versus buy question.
1:08:47 I think what's interesting about it is like, where do, when does it become, like, where does, where does an SRE, an SRE like really see the most value?
1:08:55 It's typically when you have like a tremendous number of like service dependencies and these like massive sort of like systems and tons of Kubernetes and how be. going it's to Helm files, just a tremendously large Kubernetes infrastructure Helm that's files strewn everywhere around. I think that's why Resolve has done really well in the enterprise, where you go and you sell to a Ford, you go and sell to a Walmart, and you say, hey, your service level dependencies are crazy.
1:09:19 You see these diagrams of how things are set up. Then they go in and they're like, we'll try and make sense of all of this. think that at smaller scales, it's usually still manageable. You can roll your own. But I will say that for the enterprise, I think they've done a good job. Yeah. For people who have never worked at this scale, because it took me a while to see this, it is on the order of, let's call it tens of thousands of repos, more repos than developers.
1:09:44 These are the situations where you don't know what code is running anywhere in the company, because probably the guy who wrote it is no longer here. That's absurd. And by the way, it's still low -bearing. That's the scariest thing. That's absolutely right. And then you're the person who's on call, and you're interacting with a system that goes down where that person still isn't there. And there's like old out -of -date runbooks, and you're like, what the hell is going on here?
1:10:10 It's a scary, scary thing. yes, at the enterprise level, it's a whole different beast. Yeah, so the kind of college hacker, one person per repo, they've never, no appreciation. But I think if you just give that idea, you drop them in a simulation of, okay, day one on your job, this happens. What now?
1:10:32 Yeah. Yeah. So another shout out I'll give here. So Resolve folks, actually last year, they made an assignment for the class where they effectively created a sandbox for students to go. And within the sandbox, there was many services that were doing their various things. And there was even bugs in the services within that sandbox. And then as part of the assignment, the students had to go and actually use the Resolve agent and kind of probe and play the human on -call person probing the system, describing what's happening.
1:11:06 They've got an interesting website. Yeah, it's an interesting website. I do not. It's been a while since I've seen this. This is very interactive. Okay, they're being a pager duty now. Yeah, but also this video. I was, uh, you know, you were shouting out. Interesting. Interesting. But okay. Yeah. So, okay. Well, you know, we never get a chance to talk about this side of the world, except for conversations like this. any other segments?
1:11:32 Because I think AI SRE is maybe too broad or even too narrow, uh, depending on how you look at it. Right. Like too broad, meaning why there's several subdivisions of AI SRE that's interesting. Um, or too narrow being like, well, actually that's not even the biggest problem that we have. So take it however you like. Yeah. Uh, yeah. You mean specifically for Monaco or just broadly for organizations thinking about this?
1:11:53 Broadly, because that is our audience. Yeah, I think that, again, it's a function of how we just scale. I think that for the enterprise, there are a number of issues that aren't always... I almost feel like the answer is not even always just more agents. And since we're still on the week five agent -ready code bases, when I talk to companies of a certain scale, that the things that are holding them back are really more just like low -level DevOps, CI, CD stuff, right?
1:12:21 Their merge queues are getting backed up. It takes too long to stand up an app. There's just like actually more foundational things that have almost nothing to do with that like throwing in one more SRE bot or a bot wouldn't be the answer to the problem because there's like more systemic things that they need to figure out. And so...
1:12:37 So that brings us to week 6. Yeah, exactly right. So that's where we get into... Yeah, exactly right. There we go. So the merge queue and beyond. Yeah, this is a love letter of sorts to modern Gentic code review. Yeah, I think that the biggest thing that maybe happened between last year and this year was just it suddenly became very clear that code review is a bottleneck, especially once people got really, really uphilled and the velocity on generation became so high that you could, in theory, put up 30 plus PRs a day.
1:13:08 And then what? At what point do you decide that you either have to institute yourself and put yourself in the middle as some way to gate and review things in the traditional way, but then you're literally the constraint, you're the choke point of the velocity. And that's not good, right?
1:13:24 Then what was the point of generating at scale? Or you say, well,
1:13:33 And I think a good that's thing. I think a good that's thing. I think that's a good thing. I a that's think good thing. I think that's I a good thing. think that's I a good thing. think that's I a good thing. think that's a I good thing. think that's I a a good that's thing. think good thing. I a good that's think thing. I a good that's think thing. I think that's a good thing.
1:13:45 I think that's a good thing. I think that's a good thing. I think that's a good thing. some percentage of the time, essentially like auto -approving?
1:13:58 And what has to be true from a kind of a design perspective, from like an architecture perspective to just auto -approve? And when do you have to insert yourself and say, actually, here, I still want to review this carefully?
1:14:08 And even if you do review it carefully, are you still doing line -by -line review or are you doing something else, right? So yeah, some of this one. One of our hottest topics of this year, we have a guest writer program for Latent Space. This is from Ankit from Aviator. Nice, nice. And he has a whole thing on killing the code review because I, know, That is maybe the final boss. That's a bottleneck.
1:14:28 He has a few gates by which he does this. I don't know if you've talked to Fanket. Yeah, I've read this post. I haven't spoken to him personally, but I've definitely read this. Mostly right?
1:14:38 Any criticisms? No, I think this is right. Broadly right. I guess you mentioned one thing, the last boss. I think two choke points. There's the review, and there's probably the intent or the thing at the top. There's some stats as well that I like to pull up. Everyone's stats is different. For sure, for sure. But clearly, more people are merging. Clearly, the number of reviews are not increasing in line with merge.
1:15:02 Yeah, that's right. And if you look at any modern sort of company that's really invested in this and thought about this carefully, and some people have already done posts about this, and I'm sure we'll do posts about this as well, but I think recently, like the Amplitudes and I think Cloudflare has done a good one. I think Augment has also done a good one they talk about the design of their AI code review system.
1:15:23 These are a bunch of really good, you want to share some of those links, I think they're fantastically written posts. Yeah, so shout out to, I actually, shout out to Cloudflare because I think that very, like, kind of, you know, in a sense, like, legacy company or, like, you know, bigger company, but I think they've done a fantastic job in terms of just their content around the AI, the AIS DLC. So I think they have a good post there.
1:15:43 But TLDR, yes, I think this is true. think that there's, we're still finding that sweet spot. What is auto -approvable? What is not really auto -approvable?
1:15:53 And if it is not auto -approvable, is even in that world of something that has to be somewhat human reviewed, what do we review?
1:16:00 Are we still doing the line -by -line? Are we doing some combination of the spec? And, you know, I think this, yeah, again, a whole separate question there as well. Are people keeping, are you guys keeping the spec up to date at all times?
1:16:15 There's no way, right? Yeah, well, I mean, okay, a few different places, right? So a spec that as it exists in a linear issue, and there's a spec as it exists in the context of a PR description, right?
1:16:25 So you can have your PR description be the place where you describe more or less the entirety of the implementation in a more condensed format. And as long as that's the most up -to -date thing representing this point in time of the code base, I think that's sufficient, actually. So we're thoughtful about our PR descriptions. We make them be not just one paragraph. It's more like it is effectively like a spec of what has happened in the PR.
1:16:46 And there's even like kind of line file level pointers to things where things have changed. So you not only get the high level of what's happened, but even the kind of the construction of the files and sort of the service level boundaries that you're touching. So, yeah, I think that's where we've invested quite a bit. So I push back on that because to me, of course, it's, you know, bare minimum, it should be internally consistent with the PR spec.
1:17:07 Sure. Yeah. No, what I meant is the spec for the feature, code base, because that is the summation of all PRs. That is like if your PR spec got it wrong or missed something, it's going to be the delta between what the whole code base or the whole history of the code base is supposed to do versus what your PR is trying to do. So I mean, I don't expect you to have an answer because I really stress about this myself.
1:17:30 try to instruct models to do it. They're bad at it. That's it. So the humans are the spec gods of like, well, five weeks ago I did this other thing and now you're reversing it. Have you seen the post from Basis?
1:17:44 Like they had a tweet, tweet little thing, tweet article. I thought they did a great job talking about like sort of file artifacts. They track via Git and some of them are like the specs, the intent of what is to be. And the other is like what they call like the canonical state of the code, which is like a little bit more maybe what you're describing, which is implicit knowledge, implicit intent, that also has to be codified somewhere.
1:18:07 Right? So that part, that's all version controlled. And I think they found that to work for them. So might be another thing to share as well. Yeah, Viboo is trying to pull that. This is Basis, the accounting company. The accounting company, exactly right. Yeah, yeah. We're having Mitchell at my New York conference soon. Talk to me about that. I like that they're, coming up with this. My solution for this is, um, I don't know if like how ready I am to talk about it because I may never ship it, uh, is something that is replacing Git in order to capture all of this.
1:18:36 Uh, so I call it a program database, uh, where it captures all traces and all, and, and reifies specs to something that, that we, that we can ignore and keep. And it also controls the skills that are attached to it as well. Because I think basically Git is only focused on code then you just need a source of truth that is not only everything else outside the code, including the implicit, what did you call Yeah, it?
1:19:00 like implicit assumptions or constraints, the intent. Other stuff. Yeah. But also highly concurrent. Right. Because Git, by design, introduces mutexes everywhere, which sucks. Yeah. So you need like super concurrency. Have you seen, is it Kid Butler, is it?
1:19:20 The guy who used to be CTO. Ian, I think is his name. I forget his name, but yeah. Scott, Scott. So yeah, so I think there's a lot of people going after this. And I think there's the question of like, really what is the ideal representation of tasks?
1:19:36 And I think the winner, probably not me, Linear. Really? Because that's where the tasks are. And clearly they have decided to become a coding agent company. And they have all the sorts of truth there as long as you buy into linear. Yeah, that is true. I wonder, I've had this somewhat held belief that I think a lot of companies are going to evolve into some version of a coding agent company, which is any company that has an interaction with B2B software where you're collecting something about the user or some product insight, I think you can also close the loop.
1:20:12 Because ultimately, the purpose of coding agent is to act on some adventure thing that's happening where users are involved or customers are involved and then use that to fix ideally some asset to the product. I think what Linear is doing, I imagine the same thing Amplitude could do as well, because they're also getting insights about user analytics, user problems. I imagine even if you want to really stretch it, you could even have Decagon or Sierra and do something similar where, you know, here's what's broken, go close the loop.
1:20:37 Yep, all right. So set all those things up for agent -ready code base set up code review. Security is week seven, it's not week one. Students. What do you mean security is not important?
1:20:48 I make fun of it because I always say, like, you know, at Amazon, job zero is security. And they always say this. Yes, yes. Anyway, but yeah, security review at the end. I was at Amazon and I don't remember that, but I believe. This is a thing Andy Jesse says. Okay, okay, okay. I see, I see. I guess he ... I was there partly during Jeff's time and then partly when he was transitioning to Andy.
1:21:08 But yes, I think this is a topic last year as well, and it's also one this year. I think that there ... Isaac also spoke last year. I find that he's an incredibly thoughtful thinker ... Thoughtful thinker. A thoughtful leader in this space. I think Semgrup is an interesting company because they've been around even before agents. I mean, the OSS repo was what exists well before, like, kind of traditional SAST kind of defense mechanisms.
1:21:36 But I think they've done a good job of pivoting, or not pivoting, or augmenting their offering into this space. You know, I think it's an interesting time, right?
1:21:45 There's just a lot going on where these attack vectors are now kind of coming to the surface. There's a lot more slop and AI -generated code. And I think maybe one of the core that I want to drive here is just the idea of dependency tracking and supply management, basically, like supply chain. You have to be careful about your dependencies and pin things appropriately and don't just arbitrarily hit npm install and expect the chain to go.
1:22:10 Yeah. So potentially, the most secure code base is one with no dependencies. Yes. That's correct. You look at the dependency. You're like, cool implementation, bro, but I'm just going to make my own. That's absolutely right. That's absolutely right. And that's an interesting insight because it's sort of to the limit. And, you know, I think it's probably easier said than done. But if you believe that with code now being commoditized, code being easy and cheap, why not just build everything in -house, right?
1:22:36 Like, of course, and that would be the safest thing because you've seen everything. You control everything instead of, you know, depending on Axios or whatever. But, you know, we'll see how many people actually act on that. That is the safest thing. I agree with you. No, so that statement is dangerous because then you would just make your mistakes instead of relying on other people to discover mistakes. The whole point of open source is that, you know, what is it?
1:22:57 Like sunlight is the best disinfected. It's like Linus Torvalds thing. It's like, well, yeah, you will be very slow to discover your own custom software dependencies because you only have your users and they won't report it to you. But if this thing has 100 ,000 users worldwide, someone will find something. And then you can depend on it more, right?
1:23:17 Hmm. Yeah. This is, there's no, basically, I don't want to be, I'm trying to be a voice of reason. Right, right. And you cannot be absolutist in both. And you, you, I kind of trapped you there. I talked, I talked to you into the talk track of vendor everything. Right, right. And there is a bug there. Yeah, yeah, yeah. It's an interesting one. I wonder how much that, how much is that, buy some aspects of that.
1:23:40 But how much of that is impacted by the fact that everything... I have this belief that most OSS repos, if not all of them, will just become software factories that are effectively agent run in the future. What happens then where... I mean, even if you look at Warp, right?
1:23:54 Warp, which is a modern coding agent, of course, they're also selling a software factory product, but they are running their open source warp terminal as a software factory. And that almost suggests that there is a part where yes, humans are still involved, but increasingly less so. So certainly not every single person is reviewing every single thing that comes in to maybe the same exact level that we did in the past. Does that make this code base more or less vulnerable to security?
1:24:17 It's a bit hard to say, right? I think that's just where there's a little bit of nuance that I think we're still... Even the folks that I know that work in OSS are trying to figure out exactly what that stable point is. Yeah. There's a couple of related arguments and then I'm happy to keep going. Security is like a nonstop, huge, huge, huge surface area thing. one argument from Malta of Vercel who's like, okay, let's take every single highly dependent on software that's written in C, C++, port it to Rust.
1:24:48 And it's like, we'll be faster, more secure. And it won't probably take like $6 million, but you'll get it done. And then there's another argument for, this also links to pacing the frontier from the open source, from the model labs, who are like, okay, let's slow down so that we, or we'll do Daybreak, we'll do Glasswing, we'll make sure everyone's patched, and then we can let loose on rampant acceleration. So I think there's an inherent belief that the models will fix all security, and I think it's just unproven to me because what if they introduce new bugs and they're just different bugs?
1:25:26 That's right. That's right. So, I mean, well, I'm looking for pushback. I'm looking for like, well, no, you're an idiot. Like, obviously, models are way better at this.
1:25:37 Yeah. I guess like that's almost like in the limit. The question is like, is a model, do we believe a model will create perfect software, right?
1:25:46 If a model is creating perfect software, then you're done. Is there such a thing? Then that's done, right? Then we're done here at all. Like, then we're just going to go home and chill on the beach for the rest of our lives. I think there's also the question of, is it good enough to first give it for security?
1:26:00 So let people patch, then just delay the release and do it after? Or is there a problem of, oh, no, it'll be much better at finding vulnerabilities. You can full set and test it for longer, even if you've patched. How long, right?
1:26:14 You can give it for three months. Someone can try to exploit it for six months. The problem is not solved, right? Right, right. Yeah. My whole problem with, and again, this is a whole rant that we don't need to go into, You name 100 less organizations that, okay, these are priority. They have access to a cyber model that you've just created 100 attack vectors for me to go compromise. And by the way, they're pretty easy to compromise.
1:26:40 Yeah. I don't know. Anyway, above our pay grade. It's always fun to get juicy in security. I mean, security is right now very juicy, right?
1:26:52 I think actually the security folks love it because they usually don't get that much attention and now they have a ton of attention. Is that true?
1:26:58 What do you mean? I mean that they don't get attention. I find that at least from a VC perspective, VCs love investing in security companies and have pretty good, like a whiz or something, has done very, very well. Whiz and Socket, I literally had this conversation with VC two days ago. Apparently, Whiz and Socket are the only security companies in the last five years that have worked. What about abnormal?
1:27:21 I don't know. It worked for a VC point of view. I see from an economics perspective. Interesting. Not for me to say. Obviously, many great players in that field, but a tough industry, I think. So basically, the best time, I think it's not controversial, best time for any security company is when a high profile incident just happened. Yeah. Right?
1:27:44 Obviously. Then obviously you're going to sell a bunch of security. Totally. Yeah, so I think that's a fair thing. People should be aware. should know the basic mitigations. I do think code scanning is important, but I just want to give people the idea that models introduce bugs that you wouldn't make as a human. And if you're not checking those bugs, like... then It's not even that they make bugs. It's that you have less awareness of what's being written, right?
1:28:13 Like when humans. Yeah, because you know, you're reviewing everything. Yeah, yeah. Forget bugs. Models are pretty good. They'll have, on average, I would say less bugs. The problem is also that you don't even know what you're putting out. Like, yeah. There's a lot of that. Yeah. And that's, yeah, I think we have to be thoughtful of mitigations and, and like the, and there's sort of a combination of security and, and CRD as part of code review as well, right?
1:28:35 I think this is also doing those checks, I think, is important. Yeah, so I think maybe I would say one more thing here, or check my intuition on this thing, is that not all parts of the code base are created equal. The parts of the code base where security is very important is data storage and billing. So let's call it data billing auth. And auth is obviously related to all those things.
1:29:03 And those basically don't Vibe code and then everything else go nuts. Yeah, I think that's absolutely right. That's absolutely right. And it's really funny how like, okay, well, if you don't know what the difference between off N and off Z is, please learn it. And then, know, if you don't know, if you have never looked at AWS IAM, which you have, please learn it. Like these are basic like conceptual things to understand that you can prompt for.
1:29:28 Once you have the language, you can prompt for it. If you don't have the language, you're fucked. Totally, totally. Yeah, to an extent, I would extend that to just general, certain cloud platform of things, right?
1:29:38 I mean, there was the, I forget which company had that huge incident where it was just like a DNS issue that brought down, was it?
1:29:44 Do you remember this? DNS issues? You're going to need to narrow it down. Yeah, I know. There was a particularly prominent one. Was it?
1:29:52 There have also been many prominent DNS issues. I don't remember now. We'll list them off. It's always DNS. So DNS is more of like a reliability uptime. Sure, sure. You know, security, I think people are like, did you lose money?
1:30:06 Did you leak customer information? Yeah, yeah, yeah. I would probably put that above that for sure. Only DNS because of Yeah, general uptime. Yeah. Background agents. Now you got your buddy from Qualcomm here. But, I mean, your guests aren't necessarily correlated with that week, right?
1:30:21 Yeah, no. I mean, like the nine one was a little bit of a scheduling thing where I've thrown in, you know, Ilaad and I'm just, I guess I'm just isn't with every week nine. Ilaad, I was trying to get him at the end, but I think it was just timing wise. Couldn't work it out. So yeah, it's a, it's a bit of a weird one where I don't, I guess I'm not lecturing at all week nine, but, we're coming back to those topics in week 10.
1:30:39 So it's a little bit staggered in a bit of a bizarre way there, but yeah, week eight, Rajesh who has been leading a lot of like the infra stuff at cloud infra, dev exits and beyond at cloud flare, which as I earlier said was, I think they're doing a great job here and definitely. their adoption for a company of their size and a public company of their size is fantastic. And then their content I think is also really good.
1:31:00 mean, this is fundamentally about all the ways you can use asynchronous background agents. I think I'm also going to spend quite a bit of time talking about the Stripe Minions, the Spotify, I forget the name of their version of this, the Ramp Inspects, the things of that nature that I think are all kind of showing what does it look like when you scale this out. Cloud Tag. Instinct. Yeah. Yes. Muse?
1:31:23 Muse? Yeah, I mean, I guess less on consumer, more on software teams using them at scale. They're all coding agents under the hood. It's true. Everything is a coding agent. Yeah, like GrokBot is one of those interesting things where it's very much presented as consumer -y. It's Grok. It's Cursor under the hood. Yeah, yeah, yeah. It's the Cursor team. No, no, for sure. So I've seen the Cursor working in the dot, dot, dots.
1:31:49 But they don't have a code -based integration, right? They do. Okay, I haven't used that. I've only been using it for open table reservations. Exactly. I think that's the point of what they're trying to do. I see. They're trying to be like, well, you can use it for consumer stuff, but here's the extra power if you know how to use it well, because it's actually Chris around the hood. I think this goes to all coding agents, right?
1:32:14 Like cloud code, cloud design, people using codecs, not knowing they're using Codex, right? A random code interpreter window pops up and you're just asking JetGPT a question. At some level, there's coding agent under the hood, right?
1:32:29 Yeah, so I'll give you another sort of practical example. So I run all my SEO and AEO of Codex it has a code base, even though like a traditional marketer would just check a address or something. Because I write all these scripts for doing all sorts of crazy stuff. And also I need to store data. And also I need to visualize data. And pretty soon you go, as long as you're doing anything on an ongoing basis that has maybe a lot of steps or a lot of data, you need a code base.
1:32:58 Hmm, that is an interesting take. I think models also like building code bases. No, but that could be a temporary thing. I guess maybe my point is more on the interface, right?
1:33:09 Like, will, are you going to, like, are you going to go so far to say that a Grokbot will have a place where, like, the power user will be able to see the code under the hood?
1:33:16 Is that, or is it, will that be extracted? Maybe they'll just jump to cursorize, I guess. Is that what you're saying? Well, exactly, because I'm like, otherwise, then you basically cannibalize cursor. Like if in the limit, what you're saying is GrokBot is the cursor, then you don't even need like a cursor if all under the hood is just becoming... It's the same team. So they... Right now they don't care. But yeah, it's all GrokBot.
1:33:34 Yeah, yeah. I buy that. I mean, I do accept that even things that seem outwardly just like non -technical use cases under the hood are technical, can be like written and designed and implemented in code. Like that much is absolutely true, right?
1:33:47 The question is just more like, is a developer, is that their single point of entry? Yeah. I think it's not even developer, right? Like you think of a regular person doing marketing that is using Cloud Code or just Cloud in the app now is working in a code base that they would never know, never touch, but there's a code base manager. I think Cloud Tag is a good one for background agents.
1:34:09 It's a big way Anthropic works and, you know, they make it, you should see how they work. Yeah, yeah. Take a shot up. Okay, AI Native team. We talked a lot about these topics of like gateway. We just put a routing episode out today and MCP Portal I would say MCP Portal is like the least established bullish on MCPs there's not bullish or bearish it just is part of the landscape now right
1:34:37 That's a statement there was a whole MCP is so back MCP is dead
1:34:42 MCP Portals is an unproven business I thought about starting one I actually pitched a guide and topic on on potentially starting one, and then I lost interest. I lose interest in a lot of things. But what's in there?
1:34:55 So what I'm thinking here is if this is the week more geared toward scaling this out to what happens when you take some principles, and a lot of this was informed by the conversation I had with folks that were leading A -native teams at respective companies. And so here, I think that something at MCP Portal really becomes interesting once you're at a certain scale, where you effectively have to have a single point of entry to do things like governance, security, auditing, cost optimization.
1:35:25 And so I'd be curious to hear what you see as future or maybe lack thereof of something like this. But to me, if you believe in an open router or a single point of entry for many things to happen and connect to and be effectively governed, I feel like there's a world where this also can easily exist. Yeah. Yeah, my only tweak on this is that this folds into the LLM gateway.
1:35:48 They are the exact same platform. That I will give you. That I will give you. I can accept that. That actually you can agree with. And then, by the way, it also folds into the other thing of, what did I say?
1:36:00 It was off Z, permissioning and all these things and billing. So to the limit, all infra converges to the same exact problems where they have the box in the middle and then you have the box in two sides. Godbox will solve everything, right?
1:36:14 For the enterprise. It's true only if you understand the job of the Godbox. And so separating MCP portals and own gateways means you don't accept that they are the same layer of problem. In a very similar way, I also find that people don't understand database querying to have a lot of the similar things where they will separate filtering from ranking or just like general recommendation. They're the same problem. And so then you just come, because when you take all these things, you combine them in the same layer, then you can sort of cross -depend on things in useful ways.
1:36:50 If there are different layers, then they cannot. Yeah. I actually will not disagree with this. I think maybe the, yeah, so slightly pedagogical, I think that this breaking up of separate terms versus one, I think it is absolutely correct that the underlying architecture is the god box where one thing goes in and there's a bunch of other stuff that is some third party, some first party that all kind of exists on the other side of the god box.
1:37:15 So yes, I agree with this actually. I have a spicy tweet. Oh god, where is god box tweet? This is the MXN to M plus N architecture. So anyone working at Infra needs to understand. Congratulations you now have a job in infrastructure or you want to be an infra engineer, you need to understand that your entire job is making God boxes and then taking them apart. So either your problem is either you don't have enough, you don't have a good God box or your problem is someone else's God box you don't like.
1:37:50 So now you have to take it apart and make your own. That's funny. That's very funny. I like that. That's very funny. Into the heat death of the universe. Wow. That's funny. Which is that this is a, a restatement of like, there's only two ways to make software, which is you bundle them, you unbundle them. Right. But like, what are you bundling?
1:38:07 It is the things that go together, hopefully. And then like, maybe over time, people's opinions shift as what, what goes together, what should not go together. Right. Right. So, so like, you know, there, there is that, that, that whole thing, uh, you know, maybe, uh, Monaco should be more of a point solution, maybe Monaco should own everything, right?
1:38:25 And there's the question of, in some ways, you're bundling for the kind of startup segment that you serve, in some ways, you're unbundling from Salesforce, because Salesforce, that's how you win market share from Salesforce, you're unbundled from Salesforce. That's interesting. That's an even broader statement, I mean, connecting it to broadly all of vertical AI. And now, like, bundling, unbundling, I mean, that's like the whole Craigslist and Airbnb, the Tinder and all those are separate.
1:38:53 Tinder. Well, you know, like personals, basically. Yeah, yeah, yeah. I did. Never thought about it that way. But yes, you know, one of those things, so we just shipped the OpenRouter pod today, right?
1:39:05 And OpenRouter, huge success. Lots of people tried to build gateways. None of them had success with OpenRouter. OpenRouter only builds gateways. So there's an example of like this. They disagree with me that the router layer should also do ops because Arise is doing ops and observability and what have you. And the MCP gateway, they don't have that. They only handle model routing.
1:39:31 I love being wrong like that because they're like, we are the point solution and we only do this one thing really, really well. And that's our engineering focus, which is good. But it's also the marketing focus, which is nice. I think what you've described of the one -stop shop for all the things is ideologically what makes sense. But I think what ends up happening in practice, and part of it is probably also marketing, is that especially when you look at the enterprise, this is broadly under infra, broadly under platform, but then there's separate teams owning different parts of the stack.
1:40:04 And I think part of that consolidation at the scale of an enterprise, if we're just MCP portal and gateway, someone has to come together and say, now there's a single endpoint that you're hitting for both of these things. And that fundamentally is a hard thing to get everyone on board with. So oftentimes it's more political and even people problem rather than from a pure software design perspective, this makes a ton of sense.
1:40:25 But that's not it. Software systems sometimes are brought on by people problems more than pure software. My framing of this is you know how you ship your org chart?
1:40:34 I forget what the name of this principle is. Conway's law, you ship your org chart, you buy your org chart. When you start a startup or you make a tool, who in the company are you best friends for?
1:40:47 And if it's for everyone, it's for no one. That's interesting. That is interesting. So a conscious part of latent space in AIE is like, okay, there's this category of software engineer that really, really, really cares a lot about AI. Is it going to make it their day job?
1:41:05 Let's only make it for them. Right, right, right, right. Not a bundling of engineering. Never going to be the dev everyone. Yeah, look, we talk about databases, talk about security, but there's an AI angle to it. We'll see. That makes sense.
1:41:22 Software factory. One thing I'll just shout out as well, related to the point, is I think role of the future, that is increasingly important. platform engineer. I think this is a huge one. You know, okay, more lore. Temporal, job before all this, would have, yeah, so we wanted to own platform engineer. If I hadn't started AIE, I would have done platform engineer. Because that is the role that decides all of this mission critical stuff, and then the kids over in applications can do whatever they want on top of our platform.
1:41:51 But, you know, the big boys do platform. Yeah, and I think in the modern day, they own cloud, they own DevEx, they own security so I think it's all the things. Everything where you can actually lose the company or make or break platform guys. And then product guys can adjust the CSS.
1:42:14 That's saying. You're starting a fight here. I'm not the CSS adjuster, but I appreciate everyone. I mean, it's a fun time. What's the future?
1:42:24 I feel like all the guys can adjust all the things. Yeah, yeah, yeah. So there's a few aspects here. Okay, so there's a little spillover from the previous week. So yeah, coding agents, big teams, some of the concepts, but then, you know, we'll touch on the software factory. I want people to really start thinking about world of this machine that builds the machine where, if at all, are people still in the loop there?
1:42:44 Today, I think the answer is some combination combination of intent and kind of review or maybe the places like sort of intent where you're defining the specs and what needs to be built and review where you're, you know, doing some amount of your own, checking of the system. And of course this also, this is, I think that bottleneck is going to be loosened up before even the intent piece does. But for now, those are the two choke points I think in, in a fully automated system.
1:43:07 Um, so yeah, I just want to get people exposed to that, for one of the USS partners actually, warp. They're doing this thing as part of their contribution to the OSS program where they're going to essentially make a version of warp that is running the factory and students will just be like building the factory, contributing to the factory and keep it running. Like work in our factory. Work in our factory, exactly.
1:43:26 So that's going to be one of the ways. And then they're going to find things that are interesting there and pull it back in. This is one of those things where I'm not sure, obviously factory itself is very keen on this. I feel like everyone wants to build their own. Yeah, I think that's, is, it is a good question. And I think they're, of course, entitled to kind of try and sell the products they do.
1:43:48 I think it's a hard system to own its entirety. It's a lot. Decide everything for me, please. Yeah, it is a lot. And I'm trying to, I haven't gone too deep in understanding how they approach their GTM here. But I suspect you would have to do a piecemeal, right?
1:44:02 You'd have to be like, hey, we first are your coding agent. Oh, then also, like you basically land and expand, right? I mean, we talked about this bundling and unbundling piece. You don't start by saying, here's your whole factory. There's no way you could get like a Ford on board with that, you know what I mean?
1:44:14 Where you're gonna go on and just plop the whole thing on their desk. That's at least how I would think about it, is you first be their coding agent of choice. You build out some background agent for them. You own some of their SRE pieces. And then you're like, well, now we can just weave it all together if you define everything in code or something like that. Or like, you know, configs.
1:44:32 There was a fun blog post on this from OpenAI. Our buddy, Ryan. We had Ryan on the podcast. Basically, they spent five months making an internal tool with zero lines of manually written code. They kind of walk through. One, it's called Harness Engineering, so you can learn Harness Engineering. But the learnings of there was kind of a slump or a slower at first, and then you offload and you had PR review.
1:44:56 At this point, it's from February, but it's a vision of a software factory, right? Like you are full sending, letting agents fully write everything and how to do that well. Yeah, absolutely. Yeah, I mean, some version of this is going to exist. To swyx's point, I do wonder about the sales piece here, like the actual business of this. That's an interesting question. But from a principal's perspective, I think this has been a version of the future.
1:45:25 And then beyond that, I think there's going to be some amount of dabbling into and speaking about, okay, what does it mean to be a developer in the future?
1:45:34 People talk about taste. People talk about understanding of systems and architecture and software taste. I want to go a little bit deeper here on maybe starting to at least put some answers to the question around this. Yeah. So this is your answer to where AI software engineering goes next. Yeah. The factory is the tactical, but I think beyond that, there's a question around the role. What does the role evolve into?
1:45:57 If the factory is running and full steam ahead doing its thing, the conveyor belt is churning out code, what do we do in all this?
1:46:05 And I think you and I, we had this discussion with Jerry at the dinner once upon a time a few weeks back with Dex and Jerry around the factory. Yeah, I think that there was this part of, at least the discussion points there, which I'm pretty much on board with, is increasing levels of abstraction and then somehow there's someone still, even a CEO of a company has to have enough oversight and understanding of different constituent parts to be able to make a judgment on direction, to be able to course correct, to be able to set metrics and things of that nature.
1:46:34 I think there's a version of that translates of that. There's a version of that analogy that translates to someone who oversees the technical. We call it a manager of agents. We call it different terms. They're all buzzwordy. But in some respect, someone who understands product requirements, constraints, technical constraints, kind of loops through all of those and knows how to adjudicate. Yeah. I will say that in the chair before you, we just had Anthropics head of products and head of engineering for the platform.
1:47:03 And I was like, when are your jobs going to come by? And they're like, not yet. And so if Anthropics is not there yet, you should not feel like it's imminent. Good, good, good. Good. Did they say how long or are they just not yet?
1:47:18 But not yet, but who knows? And I definitely, there was definitely a lot of like, okay, yeah, I'm doing her job. She's doing my job. Like, yeah. I think it's also interesting perspective for students, right?
1:47:28 You said there's a lot of sophomore undergrads. Like you got two, three years to go to plan out what you want to be doing. There's different ways to look at it from that, right?
1:47:37 You're not straight in industry. The thing that I care, I'm thinking a lot about is, well, so one part of this discussion that resonates more with me is the loop, uh, stuff where like, um, okay, you, you started writing prompts. Now you no longer write prompts. You write prompts, they write prompts. That's right. Right. And then they just keep going up. And I think to me, the sort of recursive, concentric circles nature, uh, going outward, but also going inward is something that, builds the factory.
1:48:05 I think build up to the factory, not build down. But I think a lot of people will eventually start building down. Like you start from the top level, like what is my goal?
1:48:14 And then you figure out the systems. That currently creates a ton of slop and you should not do that. But someday you will.
1:48:22 Yeah. So instead of, you know, that would be my question to prompt you.
1:48:31 is what is the sort of prime question that people exist to solve? That is your thing, right? So let's say engineer today, you are measured by your PR merge rate, PR contribution rate. Well, two years from now, that's going to be ridiculous. Okay, well, what's your main thing?
1:48:54 Money? People always go to money too quickly. It's something in between money and PR. Customer satisfaction, CSAT scores from your features. Even that's like a product decision, not really a code decision, right?
1:49:05 Like, I don't know. But if the engineering is evolving into like more of a PM engineer, like product engineers, you know, hybrid, then I think that's one answer to the question of like, what does the role evolve into?
1:49:17 I think it goes without saying that I would today recommend to students spend more time thinking about holistic systems, not just your like single part of the engineering puzzle, but how that ties in with, you know, And this is like in practices of happening, like students for their capstone projects, I encourage them to go out and like, in fact, what I give them as the seed of ideas for final projects is like the YC request for startups.
1:49:39 I'm almost like, go build a company, you know what I mean? Like, go, go build something interesting. What did people build last year? I mean, a variety of things. Like someone was like, someone came in the E department, was in the E school, and was studying EE. And they were like, they built effectively a translator for like parsing of like manuals to that have like, like memory addresses in C and then converting that into like, like code that they could then like interpret and even like visualize.
1:50:05 So something specific to their use case, someone was like a designer and is now actually at Vercel. And she like had, you know, like take a picture of my like design notebook and generate me an app based on what I take a picture of. Right. So literally she was like for her demo, I was like, here you go. Snaps a picture of like some random like scribbles that she was doing. And she got a fully fleshed out Vercel app from that.
1:50:23 And I think because of that, probably got a job at Vercel doing that, doing some version of this. Wow, nice. That's a high bar actually. Vercel, very known for their design engineers. Yeah, yeah. And I think she's now working on Eve, if I remember. Ah, no Eve in your?
1:50:37 Eve is an OSS, or like OSS partner. I mean, one of the ways we're selling is partnering with us is that they are having students build with Eve. Yeah. So this is the concept of meta -harnesses, right?
1:50:49 Harnesses of harnesses. You can always go one level deeper. But I do think it makes sense, at least for me, in the same way that model routers wrap model APIs, you need to wrap harnesses as well. That makes sense. I think at some level, students should also stay grounded. And there's a lot of stuff happening at the edge. There's also a lot of companies that are still like single call chat gpt in production playing around haven't tried it um you know there there is a lot of that so there's there's still time for what to work but i think your point on taste is good have taste try stuff like uh whether each company has their own version of a software factory uh you're building tools to help companies do it,
1:51:34 staying on top of it, actually using the tools, playing with it, you'll figure it out along the way. Yeah, and I think 100 % agree with all that. I mean, first to your point that are a lot of companies still in the prompt chat GPT or some version of chat GPT?
1:51:49 Maybe they just got a cursor license and are just now getting people onboarded. That is way more true than we believe, certainly here in the Valley. That is, I can say, 100 % confidence. But yes I think for the students being exposed to this it's just at a certain point the fact that they know this and are on the edge is what will eventually allow them to also be some of the more interesting and more valuable contributors when they go to maybe places that maybe aren't up to snuff in terms of being on the frontier one of this kind of progression type talk still very influential about a year old, Yeggy talking about the levels of psychosis he's all the way through level 10 now
1:52:27 where he's given up and he's moved to a farm. But yeah, you want to say, sorry, I'm making fun of him. He built Gastown. Yeah, I remember. Yeah, yeah, yeah. All these things, moved to Gas City. Now he's burned down the city and said, I'm building my game. I'm no more Gastown. I thought that, I mean, I, I remember talking to him at, at the engineer fair, uh, earlier this year and, uh, wasn't he, his whole thing, like the only useful thing that was built at Gastown was Gastown.
1:52:53 Yeah. Right. That's a horrible failure. Horrible. Totally. Totally. Um, and, and so like people should be aware that like psychosis is real. if you're not holding yourself accountable to delivering real products, you're just screwing well with LLMs. Yeah. This is like a maybe, I mean, relevant to it, but I wonder how much of, I mean, I played a little bit with Gastown, very involved system, maybe a little ahead of its time, but I'd also wonder if they were like bottlenecked a little bit by the marketing that it was a very difficult thing to grok from a terminology perspective.
1:53:24 Like there's just a lot going on. It made sense to him. I think the Gastown marketing also is one of the things that propelled it, right?
1:53:30 Like, what was right before Gastown? Beats. Yeah, Beats. Unless you're talking about someone else's stuff. No, no, there was the whole open claw, there was something else, and there was Gastown. And the whole manifesto also really helped spur it into the conceptual. The people that would never use Gastown, but learned about Gastown, because you can read the manifesto. There's a marketing angle to it. Yeah, I mean, so Steve is necessarily someone to take literally, but directionally correct, you know?
1:54:04 And so I think he's directionally correct on this one. And so if you're at the IDE stage, it's time to move into the power user stage. If you're a single agent, it's time to move to multi -agent and blah, blah, blah. And you probably should try to build your own orchestrator, right?
1:54:16 And that's all that. Arguably, this is kind of out of date now. There is more to the software factory beyond this. But this article predates the software factory. do think of those things where I'm open to, as an employer, I have this problem in my company today which is I do not want to hire anyone who hasn't been through psychosis and come out the other end because I am currently paying for somebody to go through psychosis and that's very expensive from his salary, very expensive from tokens and I'm getting nothing out of it and he doesn't get it.
1:54:48 My feedback doesn't get through. And so like it's almost like I would encourage your students to have psychosis and then come out and go like okay what was wrong with me right because like that's what Steve just went through like a very public two year journey of this yeah it's weird on the student side there's like a lot of industry you think you're being productive you're not it's probably the student version right you're notion charting and making all these optimizations personal ass and it's like dang I went through the psychosis but nothing has changed right so go through it go through it right I haven't seen this.
1:55:22 I've seen a version of this from Boris, actually. Obviously more geared toward Claude use cases, but I remember he made a Claude artifact, which showed I think more at the team level, but I think he had four stages, not eight stages, but it kind of roughly followed the same progression of no adoption through to... Yeah, only four is good. This is too many levels. well, we'll see if Vivo can pull it up.
1:55:44 other than that, zooming out, we discussed all the things? That's the one. Oh, that's the one? Okay, here we go. It is an RFS. Okay. So it's gated, assisted, parallel. What's the last one?
1:55:55 Supervised autonomy and AI native. So it's zero index. So it's five levels, right? All right. Did you start at zero? Yeah. Wow. Actually, it's pretty good. I need to read through this in more depth. And use AI native. Yeah, yeah. I would give a shout out to dynamic workflows. is one of those things that is a sub -agent pattern that opening guys are also exploring that is clearly fitting. And so basically you're writing a harness on demand for the thing that you're trying to do.
1:56:26 And so this is maybe, like a lot of sub -agent conversation is just you're copying the parent agent, but actually should be writing way more code and then running that as a sub -agent. And that is what the Anthropic people have chosen to call dynamic workflows. Okay. But, you know. Yeah. We built a version of this as well. Yeah. It's a good pattern. For Monaco?
1:56:49 What does it do? Yeah. I mean, on the fly, sort of like harness sort of use cases for GTM. Oh, that's great. Yeah. I will note that when Boris wrote this, it's for team adoption for people that are listening. And also he says, you know, Anthropic is on stage three. We're still trying to get to stage four. Wow. He's like I am at stage four I'm the guy but you know it's not as simple as Steve's like Steve is a very good personal do this do this do this this is a much more if you're running a team and you want to AI pill your team and not just be a user of one you know, anthropic is still at stage three.
1:57:28 Don't, don't be like, yeah. Okay. I mean, you should teach this. This is, this is great. This is, this has got all this stuff that is everything that we talked about. There you go. I can't believe I haven't come across this. Yeah. Um, very, very good. okay. That was a wild tour, one semester of Stanford in one hour or two hours. thank you for doing this. any parting thoughts, any calls to action?
1:57:49 No, I, this is a, thanks for, I mean, a ton of fun. It's been nine months in the making. I'm excited. This is end of week one. A lot to do, a lot to learn. I'm excited to see what happens with especially the OSS partners and just getting students deep in the code of some complex code bases having them build some stuff. It'll be fun. Keep you guys posted. You're doing another version next year?
1:58:10 You need credits for students if OSS people want to get involved, they can reach out? Sure. Those credits matter. Do they want it? Do they actually want the credits?
1:58:20 The students? You do have credit cards. I don't think a lot of students want to spend. I'm working on an EIU credit card. Yeah, no, I think. Look, I remember there's certain number of credits for sure, but then on top of that, he's like, look, if they want to go hog wild with this, let me know, and we'll have these one -off experiments where they can just scale up to 1 ,000 plus agents.
1:58:39 This is Eno from Factory. Yeah, Eno from Factory. Yeah, yeah, yeah. No, those kind of people are valuable. So this is Wilson from Cursor, right, who is like, I'm going to write a browser, right?
1:58:48 Yeah, that's right. Those people who are ambitious and can actually execute, those are very valuable, and that's basically a recruiting call. That's right. Yeah, that's a good point. Yeah, but I do think students should understand how to be ambitious
1:59:04 because respectfully snapping a photo and turning it into an app is not that ambitious in this day and age. Writing a browser is, and there actually is a lot more projects like that. Yeah, I think that's right. Yeah, it's all about raising the floor. I mean, I think, yeah, last year it was, you know, credit to the student of last year. But, yeah, this year, obviously, we're going to expect more. That's a fail now.
1:59:24 We're going to expect more. I mean, yeah. But that's the whole point, right? leveling up, right? Last year it was like, yeah, hey, have you heard of prompting?
1:59:30 Like, here's what it is. I spent some time talking about, like, you know, K -shot prompting as one of the lessons. And, like, I don't even think you guys even think about that now. I think there's a version of thinking you stick out in your cohort of classes. And then there's also a version of you're also in early engineering roles of a bunch of people in industry, people still trying to get jobs.
1:59:51 And it's not, not to call out one demo, but you can do a lot to stick out. I'll give students a freebie of a week 11. Make benchmarks, make benchmarks and evals. This will get you a good one. It's going to get you in the door of every frontier lab every top agent lab. Good engineering blog posts with some rigor on anything interesting. There's a lot of benchmarks of low -hanging fruit there, but the engineering blog posts do really well, right?
2:00:19 You went deep on one thing. Yeah, and probably also, there's a lot of money to be made in that as well. So not just in the doors, but if you make that into businesses, you can really, really cash in on that. Yeah, people don't know, like C -Bench was like undergrad project. Yes, yes, yes, that's right. From Princeton, right?
2:00:37 Yes, yeah, yeah, yeah. Great. Awesome. That's it. Thank you so much. Thank you guys.
来源：Latent Space · latent.space
来源：RadarAI · aihot.news