ClusterMAX 3.0：GPU 云评级体系回归，覆盖 323 家 neocloud

本文为付费订阅者提供额外内容。升级即可获取完整访问权限。

自我们上次发布 ClusterMAX 重大版本以来的 8 个月里，垂涎欲滴的投资者们几乎已经找不到口袋来塞支票了。GPU 供应已降至零。与此同时，我们一直在努力工作，让各个集群经受严苛考验。

几周前，我们以一些限制级轶事预告了这份报告，这些轶事来自我们探查 neocloud 安全实践的经历，并引来了你可能听说过的一家 neocloud 客户发布的公开警告。

今天，我们终于全面深入地呈现我们的测试，ClusterMAX 3.0 比以往任何时候都更加彻底。这包括计算、网络、存储、编排、UI、监控、支持，以及你所能想到的、在 GPU 云上可以检查的几乎一切。我们将解释谁在达成交易，谁的工程师一直在努力工作，以及你应该谈判要求哪些集群附赠一瓶阿司匹林。

闲话少叙，这就是 ClusterMAX 3.0 的领奖台：

结果（执行摘要）

播客摘要讨论现已在 YouTube 上线！

• ClusterMAX 3.0 首次亮相，对 neocloud 行业进行全面回顾，涵盖 77 家提供商。

• 我们将市场覆盖范围扩大到 323 家提供商，高于 ClusterMAX 2.0 的 209 家、ClusterMAX 1.0 的 169 家，以及最初的 AI Neocloud Playbook and Anatomy 文章中的 124 家。

• 作为这项研究的一部分，我们现已采访了 200 多位 neocloud 最终用户。

• 我们更新了 涵盖 10 个类别的逐项标准清单，并更新了我们对 Slurm、Kubernetes、独立机器、监控仪表板和健康检查的期望的直接描述。所有这些内容都已在我们网站上发布。我们鼓励提供商在开发其产品时使用这些清单。我们仍将这些清单视为我们采访最终用户经验的汇总，使其能够代表最终用户对其云提供商的期望功能。

• Nebius 与 CoreWeave 一同进入 Platinum 层级。虽然 CoreWeave 仍然为其他厂商树立技术标杆，但 Nebius 现已确立为一家持续以高于其他厂商的溢价定价的提供商。Nebius 强有力的商业决策使其能够以卖方价格服务整个一类 neolab。

• Google Cloud 与 Oracle 一同进入 Gold 层级。Azure 降至 Silver，Fluidstack 降至 Unavailable，Crusoe 降至 Bronze。Lambda、Firmus 和 TensorWave 保持在 Silver，而 GMI 从 Bronze 升至 Silver。

• 许多公司从 Silver（或 Gold）降至 Bronze 或更低。本轮我们提高了门槛，全球只有 19 家 neocloud 获得 Medallion 评级。

• 我们在 Bronze 和 Underperforming 之间设立了一个层级：Participation Ribbon 层级。15 家提供商加入这一评级，这更准确地描述了我们的看法：他们只做了勉强过关的最低限度。

本文其余部分对关键趋势进行了分析：融资、Blackwell 和 Grace-Blackwell 部署、向 Vera Rubin 的迁移、横向扩展网络、可靠性、安全性（当然），以及 Agentic Coding。我们附上了对每个供应商的单独评论的附录，这再次将本文篇幅拉长到超过 30,000 字。希望您喜欢。

鉴于 SemiAnalysis 必须发布尽可能严谨的……一切……测试，我们正在为 ClusterMAX 及相邻项目招聘 MTS。如果你是 neocloud 狂热分子，在这里提交申请，让我们开始工作吧。

我们还在研究、咨询和技术人员方面进行招聘。我们在纽约和旧金山办公室有多个职位空缺。远程工作也可以接受。

• ClusterMAX MTS（全职）：考虑具有 Slurm、Kubernetes 和 GPU 各种经验水平的人选。

• Tokenomics MTS（全职）：考虑具有模型评估、测试框架、推理端点和 RL 基础设施各种经验水平的人选。

• 研究分析师 – AI 基础设施与经济学（全职或实习）：考虑具有覆盖 neocloud、neolab 和前沿实验室代币经济学财务各种经验水平的人选。

• 技术顾问（全职）：领导从技术战略到技术尽职调查的项目。考虑具有咨询和技术背景各种经验水平的人选。

范围

给后排的朋友们：我们是在对托管集群进行排名。这排除了业内许多流行的产品。我们不是在测试谁能建造最好的 powered shell 或运行最整洁的数据中心——至少不是直接测试。我们不是在手工制作镜像，也不是为了 tokens as a service 而访问 API 端点。以下是我们理想化的 ClusterMAX 读者：你从斯坦福博士退学，去追求在旧金山获得社会地位的梦想，从老可靠的那句“Cladue make me pithc deck w technial languge for agetnci ai make no mistakes”开始。你有 7 位数甚至多达 10 位数 的资金可以押注在算力上，但你对 Ubuntu 版本或如何处理 NAT 没有任何意见，而且你从未大规模管理过集群。理想情况下，所有这些基础设施都会淡入背景。你只想专注于你独特的性能优化、模型架构、训练策略、数据混合、应用，或者任何你寻求优势的地方。为了竞争，你需要最前沿的硬件，并且你希望花费 0 工程时间试图弄清楚为什么节点 19 中的 GPU 7 一直在拖延你的任务，或者为什么你集群的四分之一完全消失了。你需要一个托管集群。

这里的经济论证很简单：neocloud 可以在其庞大的集群上摊销搭建所有这些管理基础设施的成本。它付钱给一个工程师团队来构建无懈可击的健康检查、保持其机器镜像最新、制作方便的监控系统，并处理我们稍后将详细涵盖的所有相关杂务。在理想世界中，实验室为更好的产品支付溢价；它可以在不受硬件拖累的情况下运行实验；neocloud 收回投资；每个人都过得更好。

然而，总有一个时刻，实验室会愿意将这种成本内部化。在它们的规模上，前沿实验室不愿意把如此多的技术栈托付给一家 neocloud。例如，OpenAI 发表了极具洞见的博客，讲述其扩展 Kubernetes 的困难；如果它们是从一家不让它们进入 K8s 控制平面的 neocloud 租用资源，那么博客中描述的创新根本不可能实现。正如Anthropic 当前的一则招聘启事所说：

我们运营的规模已经让默认配置不再奏效。我们拥有调度器并对其加以扩展，以便将拓扑敏感的 ML 工作负载一次性放置到数千个加速器上。我们扩展控制平面本身——apiserver、etcd、控制器——使其在对象数量和节点数量增长数个数量级时仍保持响应。我们还构建每个工作负载都依赖的核心集群服务，比如服务发现，使它们能在同样的压力下支撑住。

可以说，像 OpenAI 和 Anthropic 这样速度和规模的实验室需要同时协同设计其整个技术栈，而一个现成的 Slinky 实现——即便是一个好的实现——也无法胜任。

前沿实验室在管理其基础设施时表现出的明显用心，恰恰说明了 ClusterMAX 为何重要。据我们最近查看，OpenAI 和 Anthropic 职位描述中含“Kubernetes”的开放职位的薪资区间总和为$27,769,274-$42,687,654，更不用说它们已经为庞大团队支付的薪资了。如果你不指望一场万亿美元级的流动性事件，而你又需要你的基础设施足够好，能给你一搏的机会，你就需要一家 neocloud 来替你承担繁重的工作。

至少在纸面上，较小的 neocloud 客户是大卫对抗歌利亚。OpenAI 和 Anthropic 在各自的逻辑斯蒂指数曲线上都有着惊人的增长率，而我们的数据中心行业模型目前预测它们到 2027 年底将占所有实验室算力的 56.3%。

正如我们稍后将更详细描述的，由于融资因素，这里还存在一种马太效应：盈利的前沿实验室比规模较小的实验室更容易锁定产能，后者则被困在巨额预付款、更差的价格以及更难提前数年规划之中。

这是否意味着托管集群已死？cLUsTerMaX iS wAshED？事实上，尽管 Anthropic 和 OpenAI 每年占据的份额越来越大，托管集群作为一门生意仍在指数级增长。我们已帮助无数实验室找到算力，还有无数实验室正在寻觅、愿意付费，并等待更多产能上线。Anthropic 和 OpenAI 是最大的客户，但还有一长串仅仅是数亿美元级别的交易，其总价值极为庞大。

一个可能让托管集群在前沿实验室裸金属面前取得优势的情景，涉及开源领域的进展。提供开源模型服务已经比许多人意识到的更有利可图；我们最近建模发现，“每兆瓦每年通过销售开源 token 可以赚取超过 1 亿美元”，即便按今天的租赁价格计算，也留有充足的利润空间。可以想象开源模型会提升其市场份额，无论是因为边际采用者对成本更敏感，还是算法进步缩小了与闭源模型的差距。无论动态如何，这都会提升托管集群的相对地位。几乎任何供给侧的分裂都会让实验室略微不那么愿意自行处理基础设施，而略微更愿意依赖 neoclouds。

这些集群之上的各层正在迅速成熟，但它们超出了 ClusterMAX 的范围。有几个细分领域值得考虑。有推理端点，可以是公开的或私有的，按 token 计费。有后训练服务，为特定用例定制开源模型。还有沙盒服务，为智能体提供基础设施，尤其是在 RL rollout 期间，并简化和优化容器与 CPU 管理。这些都可以松散地归入“GPU 服务”类别，它们之间的边界可能很模糊。例如，一个云可能在售出托管集群后，用闲置容量启动一个推理端点，或者它可能提供沙盒基础设施，同时在销售 RLaaS 时也在内部消费它。我们将在不久的将来发布更多相关内容，但这超出了本报告的范围。

测试方法

对于 ClusterMAX 3.0，我们向每个提供商提出了以下要求：

• 32 块 GPU（4 个 8 路 HGX 节点，或 8 个 4 路 NVL72 节点）

• 高带宽网络（我们要求 800G RoCE 或 XDR InfiniBand，但许多提供商尚未具备）

• 10TB+ 高性能文件存储（支持 NFS/POSIX 挂载以及通过 csi 驱动程序的 RWX StorageClass）

• 10TB+ 兼容 S3 的对象存储

• 监控仪表板（通常基于 Grafana）

• 5 天测试 Slurm，5 天测试 K8s（可以并行进行，在 SonK 集群上，或顺序进行，如果提供商想收回节点并重新配置）

• Blackwell，而非 Hopper（NVIDIA 的 B200、B300、GB200、GB300），或 AMD 的 MI355X 被视为可接受。H100 现在已经超过 4 年了！

我们通过以下 3 个阶段测试集群。当然，我们始终通过联系所有这些提供商的客户并考虑他们的反馈来补充测试。

阶段 1：审计

首先，审计涵盖是/否问题。集群设置是否正确？软件是否安装？是否是最新的？实用程序是否按预期工作？运行大约需要 15 分钟。可在 GitHub 上获取，或通过 {uv} pip install clustermax。

审计会在我们将配置置于负载下之前检查配置。它涵盖硬件清单、软件和固件版本、GPU 访问、容器、调度器配置、网络、存储、健康监控和安全。我们会检查哪些组件适用于我们正在测试的环境，并针对所有检查显示通过、警告、失败和跳过。我们已免费发布这一部分，并将长期维护。其余部分我们暂时保留在内部。

第 2 阶段：性能

性能阶段为集群的各个组件以及整个集群的性能特征设定通过/失败阈值。这些通过微基准测试和真实场景基准测试来完成。

具体来说，我们测试：

• GPU 计算

• 网络

• 存储

• 生命周期

• 训练

• 推理

下面我们会更详细地解释这一点，但请记住，我们测试的内容会随时间演变。

GPU 计算

我们首先检查 GPU 上的真实场景 GEMM 性能。GEMM 是现代 AI 工作负载中最关键的操作，我们之前已经解释过很多次，而 Grouped GEMM 对现代 MoE 模型尤为重要。

我们在 cuBLASLt（或 hipBLASLt）和 DeepGEMM 上测试 Grouped GEMM，涵盖多种精度类型：BF16、FP16、TF32、FP32、FP8 E4M3、MXFP8 和 NVFP4，具体取决于集群中芯片支持的类型。我们使用一组常见形状，对应 Kimi K2.5、K3 和 DeepSeek V3、V4 Pro 模型中不同批量大小下的 gate_up 和 down 投影。

然后我们测试 GEMM、GEMV 带宽和 MAMF（Maximum Achievable Matmul FLOPS，来自 Stas Bekman），后者会在每块 GPU 上按精度使用 cuBLASLt（或 rocBLAS）运行一段时间的扫描。我们会报告所有 GEMM 测试的 FLOPs。

我们还通过 GEMV 测试和 `nvbandwidth` 来测试内存。GEMV 测量的是窄向量下的矩阵-向量乘法，报告 GB/s，作为低批量解码流式行为的代理指标。NVIDIA 的 `nvbandwidth` 工具内置于 DCGM 健康检查中，用于测量 h2d（CPU 到 GPU）、d2d（GPU 到 GPU）和 d2h（GPU 到 CPU）带宽。最后，我们还有一个自定义 PyTorch 脚本，通过实际张量拷贝测量相同路径，记录固定内存 h2d、d2h 和本地 HBM 拷贝的带宽。

最后，我们为老派玩家测试 HPL-MxP。HPL-MxP 使用低精度分解求解大型稠密线性系统，这意味着我们同时触及计算、内存和通信。该测试在单个工作负载中覆盖计算、HBM、跨横向扩展网络的 MPI 广播、NVLink 和数值正确性。我们会根据芯片类型报告一系列数字格式下的 FLOPs。

尽管 GEMM 是现代 AI 工作负载的核心，但这些微基准测试很少能区分不同供应商——供应商很难把这项功能搞砸。重要的是在长时间烤机测试中检查原始 GEMM 性能（详见后文），因为如果散热不当，芯片的时钟频率可能会下降以避免过热。在 GEMM 突发测试中，你不太可能发现什么值得注意的问题。我们在每个供应商上都运行这些测试，因为它很快，而且如果失败，集群会立即不可用，但我们更担心其他测试。

生命周期

为了了解集群在整个生命周期中的易用性，我们使用了一套专门设计的测试。我们首先通过多种方法检查互联网速度，包括下载/上传速度，主要来自 Cloudflare。然后我们测试通过 pip 和 uv 安装 PyTorch 和 vLLM 等常见软件包所需的时间，以及从 Docker Hub、ghcr.io 和 nvcr.io 下载容器，并从 Hugging Face 和 ModelScope 拉取模型。

然后我们在每个可用的存储层级上使用全新的软件包缓存重新进行 pip 和 uv install 测试：home、共享文件系统、本地 NVMe 暂存区以及任何额外的挂载点（分别称为 `/home`、`/data`、`/scratch` 等）。然后我们从每个位置运行 `import torch`，看看需要多长时间。在大多数供应商上，这需要 1-2 秒，但在某些存在 LOSF 性能问题的文件系统上，不知何故可能需要 10-20 秒。

最后，我们使用之前下载的模型，运行 vllm serve 命令，测试直到能够访问 /v1/models 所需的时间。同样，虽然某些供应商可以在 10-40 秒内从共享存储将小模型加载到 GPU 内存中，但有些供应商需要超过 2 分钟。

网络

我们对节点间和节点内传输的通信性能进行了广泛的基准测试。这包括 MPI 集合通信，通过 MPI 启动运行 NCCL 或 RCCL 测试，测量从 8 B 到 16 GB 各种消息大小下 all-to-all、all-reduce、all-gather 及其他集合操作的集合带宽和延迟。相同的工作负载也通过 PyTorch 中的 torch.distributed 运行。此外，我们使用 RDMA perftest，特别是 ib_write_bw 和 ib_read_bw，测量每个网络通道上的点对点带宽，以及主机内存回退（通过 PCIe），以识别任何问题。

我们以 2 的倍数增加世界规模，直到使用整个集群，并跟踪性能如何扩展。我们还在禁用 NVLink 的情况下运行测试，特别是在 GB200 或 GB300 NVL72 集群上，以隔离横向扩展网络的性能。在这里进行核算时要小心，因为世界规模、算法以及相对于横向扩展边界的作业放置会显著影响结果。

存储

我们运行 fio、ior、Elbencho、一个用 Torch 和 Torch DCP 编写的自定义检查点保存/加载基准测试，以及一个由 Skild AI 与我们分享的自定义 datagen 基准测试。datagen 可能是最有趣的，因为它近似模拟了他们在机器人数据管道中的真实工作负载：生成视频（MP4）和 parquet 传感器数据集时的聚合写入吞吐量。

不过，发现最多问题的测试还是老派的 fio。我们以 1MiB 顺序块和 4 KiB 缓冲 / 64 KiB 直接随机块，从 c1 一直到 c32 扫描缓冲/直接 I/O 的顺序/随机读/写，或者，如果我们有超过 4 个节点，就用集群能承受的尽可能多的客户端。如果存储以任何方式配置错误，这个 fio 扫描中几乎总会有某个设置显示出糟糕的性能。我们跟踪吞吐量、IOPS 和延迟，当然，如果测试因客户端过多而失败，我们也会发现元数据中的不一致。

当然，这在很大程度上取决于我们被分配到的卷有多大，以及 SLO 是如何定义的。对这些结果要持保留态度——上面的图表实际上只表明 Azure 给了我们一个 PB 级的挂载。这种保留态度也说明了为什么拥有大量集群的数据很重要。例如，缓冲测试的延迟更高，正如你可能设想的那样，而且如果测试运行时间不够长，还可能产生非常嘈杂的结果。但延迟到底高多少，又有多嘈杂？要了解什么是“足够好”和什么需要向提供商标记之间的区别，与一个庞大的同类集合进行比较至关重要。

对于对象存储，我们针对 S3 存储桶再次运行这套测试。我们还运行一个自定义基准测试，测量顺序读/写时的大对象吞吐量、小对象 PUT 和 GET 速率，以及读延迟（p50、p90、p99）。在这项测试中，我们保持一切一致。不过，一如既往，如果我们注意到某个集群在某种工作负载上表现吃力，我们就会多停留一会儿，运行更多测试以更精确地隔离问题。

训练

进入真实世界，我们开始看到网络或存储上的问题如何在真实测试中显现出来。我们通过 torchtitan 运行两个作业：在 C4 数据集上使用 FSDP 进行 llama 3.1 8B 预训练，以及通过 torchtitan 使用专家并行进行 GPT-OSS MoE 训练。前者受 FLOPs 限制，后者受集合通信限制（除非你在 NVL72 上）。因此，当网络出现问题时，后者会一目了然。

推理

我们在单节点和多节点场景下运行 InferenceX AgentX 基准测试，使用不同的模型来找出受计算限制、受内存限制和受通信限制的状态，并开启性能分析跟踪，将 InferenceX 结果作为基准性能结果进行比较。与我们的训练测试一样，这凸显了微基准测试中涵盖的各种集群组件的重要性。如果一个集群无法通过 NCCL 测试，它肯定无法产生很多 token，而我们的 InferenceX 测试显示了有效吞吐量是如何损失的。

阶段 3：可靠性

仅仅因为我们在基准测试中看到稳定的性能，并不意味着提供商能够提供稳定的 有效吞吐量，当然也不意味着他们会提供良好的支持体验。因此，我们通过多种方式测试可靠性、监控和自动修复。

老化测试

我们首先对 GPU 和网络进行 8 小时老化测试，在张量核心上运行大型 GEMM，同时进行全对全通信，并用监控脚本跟踪温度、功耗、时钟频率、FLOPs、网络连通性、延迟、带宽，当然还会跟踪内核环形缓冲区以发现任何错误。你会惊讶于仅通过这个简单测试就能引发多少硬件问题。我们再怎么强调同时给 GPU 和网络施加压力的重要性都不为过。老化测试需要这两个组件同时经历热膨胀和收缩，以近似这些系统在真实工作负载的真实压力下的实际行为。许多提供商仍然使用分别对 GPU 和网络进行老化测试的脚本。

网络结构

接下来，我们通过循环执行大消息尺寸的全对全、全收集和归约散射，来测试横向扩展和纵向扩展网络结构的持续性能。我们报告平均、最小和最大带宽，以及集合通信错误。在前端网络上，我们还测试四个节点之间每对定向节点间的管理网络丢包、RTT 和 TCP 带宽。这项额外测试并不经常能发现老化测试未发现的错误，但我们喜欢这些数据，而且它揭示了某些以太网在负载下的巨大性能差异。

存储

继续，我们使用 fio（如前所述）在每个客户端数量下运行 7.5 分钟的混合顺序读写，然后运行 7.5 分钟的混合随机读写，以测试文件系统耐久性。我们在整个分配中增加客户端数量，并报告随时间变化的带宽、IOPS、尾部延迟和性能下降。有趣的是，我们发现一些提供商在负载下的带宽相比初始基准测试下降了近 40%。

当有对象存储可用时，我们还会运行 15 分钟的混合 PUT、GET、LIST 和 DELETE 工作负载，测量吞吐量下降、p99 GET 首字节时间漂移、限流和错误。我们在这些测试中看到了约 10% 的差异。

编排

在 Kubernetes 上，我们通过反复创建/删除 GPU Pod 并检查 API 中的 GPU 分配来测试“搅动”。我们报告调度和启动延迟，并在 kubectl delete po 卡住时报告失败。我们还增加仅 CPU 的沙箱 Pod 批次，测量有多少达到 Ready 状态、需要多长时间，以及其余为何保持 Pending。

最后，我们通过反复启动一个挂载卷、写入并 fsync 文件、然后被删除的 Pod 来测试 PVC 生命周期。我们在周期之间等待拆除完成，并使用预热周期从测量运行中排除初始镜像拉取。

破坏性测试

最后，我们来到有趣的部分：故意破坏东西。

首先，我们重启集群中的所有节点。你可能希望这不是破坏性测试，但它是。在 Kubernetes 上，我们封锁并驱逐节点，发出重启命令，并要求启动 ID 发生变化，然后等待它重新加入集群。当我们可以在一个新 Pod 中于该节点上运行 nvidia-smi 时，计时停止。在 Slurm 上，我们只需获取分配或 SSH 并运行 sudo reboot。Slurm-on-Kubernetes 略有不同，因此我们尝试从 Kubernetes 层进行操作以正确测试它。我们的脚本全程记录所有时间。

对于注入故障，我们首先使用 DCGM 注入方法。如果它不能正常工作，我们会接着向内核日志写入一条合成的 NVIDIA XID 或 SXID 消息，并观察提供商现有的健康检查和调度器如何响应。我们不干预他们的健康代理和驱逐自动化，因此检测错误并采取行动取决于他们的工具。大多数健康检查设置为从内核环形缓冲区读取，但有些不是，因此我们与提供商沟通，以确保我们的触发方式有效，并且我们有权拉取它。作为最后一种可靠方法，我们重置 GPU 的上游 PCIe 桥，这会产生真正的 XID 79。

在所有这些情况下，我们计时检测故障需要多长时间、节点保持 `DRAIN` 状态多长时间，以及节点返回集群后运行基本命令需要多长时间。在 AMD 系统上，我们注入 UMC RAS 错误并以相同方式跟踪一切。

这是我们最关键的测试。我们听到大量客户抱怨可靠性的提供商通常没有健康检查、监控仪表板和自动修复。可靠性是许多全球最大客户的首要标准，正如我们在文章“How Much Do GPU Clusters Really Cost？”中详细讨论的那样。

值得注意的是，我们的实际测试只是整体评级的一部分。有许多事情我们无法实际测试，需要使用其他研究方法详细了解。即，大规模性能、长期可靠性、支持体验、定价、GPU 可用性和交付时间表。

即将推出：代理工作负载和 RL

我们还有两个测试套件，将在后续工作中详细说明。

CPU 计算

CPU 是现代代理工作负载的关键性能考量，我们之前已经讨论过。我们开发了一个全面的基准测试，通过常见的 Linux 工具使用和即将推出的名为 SandboX 的项目中的更多微基准测试，将这些代理工作负载的 CPU 性能推向极限。

SandboX 运行一系列代表代码执行的 CPU 密集型工作负载，包括环形洗牌生产者-消费者队列、Redpanda（测量本地 Kafka 兼容代理及其吞吐量）和动态负载基准测试。

我们还对 Kubernetes 集群的沙箱生命周期进行基准测试，以了解在 GPU 集群中利用剩余 CPU/内存资源为 RL 共置沙箱可能如何工作。我们报告冷启动延迟、调度吞吐量、每节点密度、容量和基线。

强化学习

一次 RL 运行涉及许多环节：生成器负责展开轨迹，环境在沙箱中执行动作，训练器接收 rollout、更新策略并将更新后的权重推回推理引擎。扩展 RL 意味着要平衡训练器、推理引擎、环境沙箱执行、rollout 传输、权重同步等各方面的利用率最大化，其中任何一个环节出现瓶颈都可能拖垮整个 RL 栈。

我们也在看到 Post-Training-as-a-Service（即 RLaaS）的兴起：客户自带环境，或由前向部署工程师为其构建环境，托管训练服务商则搭建基础设施来对开放权重模型进行后训练。其卖点是在保护客户知识产权的同时实现定制化，这显然现在成了 Satya 的口头禅。

在我们即将推出的 PostTrainingX 基准测试中，我们会扫描训练器到推理的 GPU 分配、批大小、rollout 并发度以及策略陈旧度限制，以研究它们对性能和训练稳定性的影响。

我们通过每推理 GPU 每秒的输入/输出 token 数、rollout 延迟、KV 缓存占用率和前缀缓存命中率来追踪生成器性能。通过权重同步延迟、传输带宽和网络遥测来追踪通信效率；通过训练吞吐量、优化器步时间、MFU、训练/rollout KL、观测到的策略陈旧度、裁剪比例、梯度范数和被拒绝的 rollout 来追踪训练器性能与稳定性。

对于环境，我们测量设置时间、工具执行时间、超时和基础设施故障。GPU 利用率、内存使用和功耗提供系统层面的背景信息，而归档的轨迹、日志和精确的环境定义则使结果可复现。

PostTrainingX 的目标是对每一家服务商进行基准测试。这包括 Applied Compute、Azure Foundry、Fireworks、Thinky、Baseten、Trajectory、Engram、AI21 的平台等托管训练平台，以及 Miles、slime、prime-rl 和 verl 等开源框架。最终结果应当为客户提供数据，以便决定是在开源框架上运行自己的后训练栈，还是将其交给托管的 RL 服务商。

我们已经开始在开源框架上运行，包括 Miles、prime-rl 和 verl，接下来将扩展到托管服务商。敬请期待……

行业趋势

融资

过去几个月里，neocloud 领域最热门的话题一直是融资。每个人都想“跟着钱走”，而这正是我们在 AI Compute, Capital, and Markets Model 中所做的，我们在最近一篇文章中拆解了 NVIDIA 的 Backstop Universe。下面我们将讨论另外几个热门话题。

债务与获取债务

如今，neoclouds 已经意识到，一份信用良好的客户合同可以帮助他们以低于自身信用评级所能支持的利率获得债务融资。这意味着，没有承诺客户（或“IG offtaker”）的提供商可能需要更昂贵的债务或类似股权的资本，才能购买 GPU，并让业务起步或实现增长。与此同时，贷款方也需要证据，证明提供商能够按时交付其集群并满足客户合同中的 SLA。债务融资既取决于租赁终止权，也取决于可用于偿债的现金以及 TCV。这促使多家保险提供商带着参数化产品进入市场，为 neoclouds 及其贷款方提供保护，以应对合同终止，通常在发生终止时提供一个月的过渡期以寻找新的 offtaker。我们非常看好这种结构，并认为这是贷款方用来降低这些交易风险的一种可靠方法。

交易对手风险与 Nvidia 的“资产负债表即服务”

在一个大型 neocloud 项目中，GPU 贷款方和数据中心贷款方可能面对同一项目内不同的交易对手。例如，在 Anthropic TPU neocloud 结构中，Broadcom 支持设备融资，而 Google 支持数据中心租金。由于供应链中的多笔贷款可能依赖同一个 AI 实验室，因此即使每笔贷款指定了不同的 neocloud 或数据中心借款人，也可能存在系统性、相关性风险。正如我们刚刚讨论的，建设延迟可能触发取消并损害现金流。

Nvidia 的兜底来了。由于 Nvidia 通过收入下限、房东担保以及计划转让给第三方的租赁来支持 GPU 需求，它们能够帮助 neoclouds 和 AI 实验室在无需超大规模企业参与的情况下获得投资级融资。Nvidia 在首次销售时赚取 GPU 利润，但现在还可以获得超过其“AICP 下限”的收入分成。因此，它们的财务支持推动了对自身硬件的需求。资产负债表就是护城河。

折旧计划

自从我们去年 11 月在那篇文章中抨击 Burry 博士以来，这个话题已经有所降温。也许是因为人们正在签署的那些 6 年合同，或者是因为那些讨厌的 4 年前 H100 就是不肯降价！谁知道呢。

无论如何，我们在建模折旧时将设备购买日期与投入使用日期分开。6 年折旧假设需要针对具体提供商的支持，而在 NVIDIA 的 AICP 计划中，6 年收入下限实际上并不确立 GPU 的使用寿命。我们的折旧敏感性测试会使建模利润率发生相当大的变化，但 IRR 仍然是贷款方的优先事项，因此我们不会在此分享这些测试。当然，财务分析并不能确立买家未来愿意为 GPU 支付的真实价格。这取决于供需动态，以及最终问题：残值。由于贷款方无法将贷款偿还计划与我们用于设备折旧的会计假设分开设定，我们陷入了一个无人愿意在今天承担评估 4 年、5 年或 6 年旧 GB300 残值风险的世界。

Blackwell 和 Grace Blackwell

现在让我们回到本文其余部分的技术内容。好东西。

正如我们在ClusterMAX 2.0 中详细讨论过的，运营 8 路 H100 HGX 服务器与 GB300 NVL72 机架级系统之间的差异是巨大的。那些在 3-4 年前就开始安装 H100 的供应商，如今正面临以下挑战：

1. 基于 ARM 的 Grace CPU（而非 x86 的 Intel 或 AMD）

2. 采用 sm100/sm103 的 Blackwell GPU（需要 CUDA 13.0+）

3. 强制要求直接液冷（DLC）

4. 高压供电（每机架 130kW+，通过三相 408 或 480V 供电）

5. 横向扩展网络，使用 800G CX-8 网卡和 51.2T 或 102.4T（800GbE RoCE Spectrum-X）或 115.2T 交换机（800Gb XDR InfiniBand）

6. 纵向扩展网络，使用 NVLink 交换机和机架级背板，72 路

7. 用于前端（或融合前端/存储）网络的 BF-3 或 BF-4 DPU

8. 多平面网络，使用 shuffle 板/shuffle 线缆

所有这些变化都会影响配置软件、监控系统、技术人员培训、设施级电力、冷却和机械系统、OEM 支持合同和关系等。

仅仅因为供应商在 Hopper 代际取得了成功，并不意味着他们会在 Blackwell 代际取得成功。

在我们的排名中，我们高度重视任何向我们交付 Grace Blackwell 的供应商，坦率地说，在配置、监控和可靠性/热备件方面遇到困难时，我们给了他们更多的宽容。

这意味着，如果客户租用仅通过 RoCE 或 InfiniBand 连接的标准 HGX 节点，我们期望看到自动修复和热备件可用：端到端在一小时内完成更换，检测时间约 2 分钟。然而，对于 NVL72 系统，在不同的纵向扩展架构上热插拔托盘是没有意义的。

因此，在我们的测试中，只要健康检查发现不健康的节点（仍然在 2 分钟或更短时间内），并阻止工作负载调度到该节点上，它就算完成了任务。如上所述，我们为许多实验室和云服务商提供 GB300 NVL72 SLA 方面的建议，我们看到的最常见模式可以称为“NVL64+”。这意味着每个节点都有 SLA，但如果一个机架中的 18 个节点中有 3 个或更多同时发生故障，则整个机架被视为宕机。这是由于简单的 2 的幂次方：许多作业可以被 64 整除，并在 64 个 GPU 上运行，而填满 72 个 GPU 的最大配置是 world_size=8 * num_jobs=9。

转向 Vera Rubin

有趣的是，从 Grace Blackwell 转向 Vera Rubin 所需的变化比从 Hopper 转向 Blackwell 要少得多。供应商仍然管理基于 Arm 的 CPU、液冷机架级架构、72 路 NVLink 纵向扩展网络、多平面横向扩展网络以及一些 Bluefield DPU。当 GB200 出现时，所有这些东西都是新的！

主要变化在于横向扩展网络从 800G CX-8 网卡转向 1.6T CX-9 网卡，以及每机架功率从约 130kW 增加到约 200kW，包括800V 直流选项。

早期需求比 Blackwell 过渡期更为强劲——部分原因是将 GB 软件移植到 VR 比从 Hopper 移植到 GB 更容易——而且我们听到许多供应商表示，他们有望比预期更早交付 VR。启动过程出奇地顺利，物理设计似乎也成熟得多。

横向扩展网络

横向扩展网络是供应商仍然拥有的最后一个重大设计决策。Nvidia 和 OEM/ODM 锁定了 NVL72 机架的大部分。在此之外，供应商选择拓扑、布线，以及 Kubernetes 或 Slurm 如何将网络结构交给作业。我们在每一层都发现了问题。

从物理层开始，800G CX-8 NIC 是多平面的：每个 NIC 将其带宽拆分到独立的交换机网络结构上，称为平面。每个 GPU 的总带宽保持在 800G。更多平面意味着每个交换机拥有更多有效端口，每个 NIC 拥有更多上行链路路径，这需要进行 shuffle。每个 NIC 通道必须落在正确的平面上，因此 shuffle 盒将光纤映射放入配线模块，而 shuffle 线缆则将其构建到组件中。两者都可行，但在可靠性（需要维护设备的频率）和可维护性（维修或更换设备的难易程度）方面存在权衡。

Oracle 是这种物理设计的先驱，当你拿到 GB300 节点时就能看出来。我们拿到的每个 4x GPU 托盘都有 16 个物理 200G 端口，这意味着每个 GPU 有 4 个平面。Linux 暴露了 4 个可用的 800G RDMA VF（rdma_vf_rail0 到 rdma_vf_rail3），每个 GPU 一个，各自位于自己的 VRF 中。聚合 PF 失败，没有可用的 GID，因此我们不得不依赖他们的 SPCX NCCL 插件，每个连接使用 16 个队列对，并在各平面之间进行自适应路由。

为什么要费这个劲？答案是规模。Oracle 在此物理设计上构建了 Acceleron。他们的 MRC 拓扑论文展示了每个 NIC 如何拆分为 8x100G，从而将 51.2T 交换机变成 512 个端口，并在全带宽下支持 131,072 个 GPU，仅需 2 层，最多 3 个交换机跳数。OpenAI 于 2026 年 5 月通过 OCP 发布了 MRC，Oracle 在 Stargate Abilene 运行它。软件层允许 MRC 将一个连接分散到多条路径和多个平面上，乱序放置数据，通过选择性确认恢复丢失，并使用 SRv6 源路由绕过故障或拥塞链路。这个逻辑层在任何规模下都可能变得复杂。这正是作业出现问题的地方。

要分配 RDMA 设备并正确使用它，Kubernetes 必须将正确的设备和接口交给每个 pod。换句话说，配置和使用 NVIDIA NetworkOperator 有很多种方式。搞错了，对每个人来说都是头疼的事。

我们的脚本识别出以下 NetworkOperator 部署模式（资源名称是我们检查的集群中的示例）：

• rdma_shared_device：rdma/rdma_shared_device_* 配合资源支持的 NAD，或主机网络回退

• sriov_host_device：nvidia.com/hostdev 或 nvidia.com/rdma_host_dev 配合 HostDeviceNetwork NAD，用于直接设备访问和 GPUDirect RDMA

• sriov_legacy：通过 NAD 绑定的 nvidia.com/<resource> 和 SriovNetwork NAD 使用 VF，并在需要时使用 RDMA CNI

• sriov_ib：InfiniBand VF，例如 nvidia.com/mlnxics 配合 SriovIBNetwork NAD。分区网络结构还需要正确的 PKey 和 UFM 配置。

• ovs_offload：nvidia.com/switchdev 配合 OVSNetwork NAD，用于硬件卸载的 Open vSwitch。

• rdma_ib / rdma_roce：rdma/ib、rdma/roce 或 nvidia.com/rdma_*，每个 rail 一个 NAD，或使用主机网络回退。

• 特定提供商：AWS EFA（vpc.amazonaws.com/efa 和 libfabric）、DOKS（rdma/fabricN 配合 roce-net-fabricN@fabricN）以及 GKE DRANET（DRA claim 模板和 pod resourceClaims）。

设备设置及其上的 NCCL 插件会影响作业性能。在 Google 的 GB300 上，内置 NCCL 库选择了一个不可路由的链路本地 GID，我们在测试时它挂起了。设置 NCCL_IB_GID_INDEX=3 解决了这个问题，Google 的 gIB 插件在 16 节点、16 GiB 全对全测试中达到 99.5 GB/s，而其他提供商为 95.3 GB/s，即满性能。

话虽如此，多节点 NVLink 是另一个问题。NVIDIA DRA 驱动中的 ComputeDomain CRD 为每个作业提供自己的 IMEX 守护进程和通道 claim，独立于 RDMA。Google、GMI 和 Firmus Kubernetes 使用了 ComputeDomain，但 RDMA 路径不同：Google 用 DRANET，GMI 用共享设备加主机网络，Firmus 用每 rail VF。与此同时，Oracle、Azure Slurm、GMI Slurm、Firmus Slurm 和 Nebius Soperator 从主机提供 IMEX，没有租户 claim。Azure AKS 运行多节点 NVLink 时没有租户可见的 ComputeDomain，尽管有 GPU DRA ResourceSlices，且通道来源和租户隔离边界没有文档说明。

每个 ComputeDomain 必须保持在一个 NVLink 域内，因为跨域需要 RDMA。换句话说，这是在分层 scale-up NVLink + scale-out RDMA 网络上的拓扑感知网络。当我们错误地启动作业、没有 ComputeDomain 感知时，当放置跨机架时，它们只是在 clique 设置中挂起。我们将把关于 EFA 不支持 DeepEP 和 MoonEP 的抱怨留到后面附录中的 AWS 提供商评测。

可靠性、健康检查、自动修复和 SLA

大约在我们开始为 GPU 集群到底要花多少钱？ 做研究的时候，我们开始向测试的每个集群注入故障。提供商之间的差异很大。处理故障需要两件事：

1. 识别它发生了

2. 正确修复它

要识别故障，我们推荐监控仪表盘。好的仪表盘会显示故障组件、受影响的作业、当前调度器状态以及每项检查上次运行的时间。过期的绿色结果不是健康的证据。当然，监控仪表盘依赖某些遥测，所以你必须正确（且安全地）设置 DCGM。

要修复故障，我们期望自动修复：重启并用热备节点替换。自动修复不会覆盖所有情况。有些故障需要人工干预、RMA 以及提供商必须直接与客户沟通的其他工作。我们在 关于这个主题的文章 中涵盖了以 goodput 形式衡量的确切成本。

NVIDIA 最近开源了 NVSentinel，一个面向 Kubernetes 的故障检测与修复系统。它读取 DCGM、syslog 和云维护事件，然后可以封锁并驱逐节点、重置 GPU 或重启。默认安装仅处于监控模式。软件是简单的部分。重要的是从每个 XID 到提供商所采取行动的映射。被隔离的 XID 94 需要重启应用。未被隔离的 XID 95 需要 GPU 恢复。显然，如果你天真地为每个 XID 都重启每个节点，你会杀掉健康的工作负载；而如果在严重 XID 之后仍让 GPU 可调度，你就有工作负载崩溃的风险。

这也是为什么许多客户只想要裸金属。他们只是想让提供商关闭支持工单。关闭支持工单比听起来更难。一个给定的工单可能来自 100 多个 XID 中的任何一个，每个都有不同的含义和解决方案。SOP 涵盖从更换硬盘或电源、清洁线缆、重新插拔内存、GPU 或 NIC，一直到对单个组件或整个服务器托盘进行 RMA 的所有内容。

自动修复有很多不同的选择，正确的选择取决于系统。此外，修复对于 GB 和 HGX 来说是一个不同的问题。在 HGX 上，你换入一个带有 8 个 GPU 的热备节点。在 NVL72 上，你无法再热插 8 个 GPU 到 NVLink 域中。一个故障托盘会导致 4 个 GPU 下线，选项是让机架降级运行或更换整个机架。好消息是 GB300 往往比 GB200 故障更少。无论哪种方式，从错误到数据中心中行动的流程才是关键。

一些提供商告诉我们他们会等待一段时间再行动，以便给客户时间把他们的东西从节点上取下来。那是自我安慰。我们正在模拟硬故障。节点已死，没有什么可以从它上面取下来了。（一个普遍的真理是，如果提供商说他们选择不做某事是因为想“给客户提供可选性”，那就是自我安慰，提供商认识到这一点只是时间问题。）

这就是拥有并运营数据中心与租用 colo 的区别所在。运营自己设施的提供商控制着现场的技术人员、备件和 SOP。在 colo 中的提供商则可能依赖房东的远程人手及其日程安排。两者都会体现在容量恢复的速度上，而这正是 SLA 必须衡量的。

在 SLA 方面，我们为 3 个层级的提供商制定了一套标准化的 SLA

在这些文档中，我们提供以下内容：

• 在两种场景（HGX 8 路系统，即 H100 或 B300，以及 MGX 4 路系统，即 GB200 或 VR NVL72）下对“节点”、“机架”、“集群”和“站点”的技术定义

• 所有系统“停机时间”的技术定义

• 三个层级的关联 SLA（“Bronze”、“Silver”和“Gold”，以及相应的阈值和与停机时间相关的赔偿罚则）

• 衡量停机时间和提供赔偿的双向承诺

• 建议从停机时间衡量中排除的项目，例如软件升级、安全补丁和物理维护

• 建议的每月审查节奏，以评估提供商对照 SLA 的表现

• 当停机时间低于建议阈值时，买方的合同终止权

• 验收测试的定义，包括针对 GPU 计算、网络、存储和软件所需执行的各类测试说明，以验证集群可正常运行并可通过验收，并建议联系 SemiAnalysis 进行 ClusterMAX 技术评估

• 买方因未达成约定验收日期而享有的合同终止权

• “不可抗力”条款的定义

• ……以及更多内容

我们鼓励算力买方（Neolabs）、云服务提供商（Neoclouds）、债务/融资合作伙伴以及保险提供商以这些条款作为其合同的基础，并在验收测试和每月 SLA 绩效审查期间纳入 SemiAnalysis 技术评估。我们使用我们的 cmax 工具和一套全面的专有流程来执行此技术评估。

如需了解更多信息，请通过 clustermax@semianalysis.com 联系我们。

安全

自从我们发布“大多数 Neocloud 在安全方面很糟糕”以来，外界的反响令我们备受鼓舞。我们的朋友们一直在他们的集群上运行 我们的 CLI，以找出哪些陈旧过时的软件包需要替换，而且总体上，人们越来越有动力以应有的严肃态度来对待这个问题。

尽管如此，我们认为这个问题值得更多审视。当关于 AI 是否会杀死所有人的讨论达到白热化时，却几乎没有人分析它究竟可能通过哪些具体途径真正摆脱控制并造成危害。OpenAI 和 Anthropic 正花费巨额资金进行功能获得性研究，其部分基础设施已被曝出存在严重疑点，然而，GPU 集群——这些模型所依托的物理主机——往往并不具备你所期望的那种免疫系统，这一点却基本未受到评论。

无论你对 AI 能力持何种看法，良好安全实践的必要性都依然成立。即使这些集群并未因 AI 工具而变得更加脆弱，事实仍然是：我们正在为并不像应有的那样安全的基础设施花费数十亿美元。有大量令人不快的场景可以想象，而这些场景仅仅涉及一个会用 Vim 且熟悉操作系统的人。通读 Hugging Face 的报告，数一数有多少平平无奇的网络安全错误让火势蔓延开来。

同样值得强调的是，在某种程度上，这是一种结构性风险。当整个行业在应对严厉监管和未经审视的社会反弹所带来的危险时，LLM 工厂被攻破是它最不该招致登上头条的事情。无论是从网络安全角度——每个被攻陷的节点都会增加恶意行为者可探索的邻近节点数量——还是从普通的社会政治角度——无论好坏，批评者都会用一把大刷子给整个 neocloud 行业抹黑——这都是事实。

做个好邻居，别被黑。

智能体编程

Agentic 编码已经在给托管 GPU 集群带来利润压力，因为使用 AI 工具的客户更愿意选择裸金属或轻量托管的集群。我们在测试中一直能看到这方面的证据。如果我们需要添加 Linux 用户，但集群上没有现成脚本，Codex /goal make useradd script and add my whole team 就省去了这些跑腿活。尤其是在文档糟糕的集群上，agent 帮我们梳理了所有可能的配置，让我们得以解困。相应的 Slack 对话往往大致是这样的：

“嘿各位，XYZ 到底该怎么用？”

“算了，Codex 告诉我 ABC 了。”

“对，没错。我们会把这条加进文档。”

在成功标准清晰、上下文充分的环境中，agent 对我们的帮助最大；否则，它们处理 GPU 系统的方式就很幼稚。它们拥有相关知识，但这些知识相对于那些看似合理却错误的做法并没有被赋予足够高的权重。比如 Claude 就习惯用循环而不是作业数组，从而把 Slurm 控制器压垮。我们的一个 agent 认定 NVLink 网络配置起来太麻烦，于是改在 InfiniBand 上启动作业并宣布大功告成；另一个犯了类似的错误，在 NVMe 而不是 NFS 上跑存储测试；还有一个在试图把 GPU 作业调度到 CPU 节点上之后，把性能不佳归咎于供应商。我们通常让 agent 通宵运行，要么用 `/goal fix this part of the cluster so tests can run`，要么用 `/goal try to break this part of the cluster with a realistic workload`。总体而言，这让我们产出成倍增加，但有时验证结果比自己动手做还要花更长时间。如果你对如何搭建集群有清晰的思路，agent 会让你更强大，你甚至可能愿意跟一个更差的供应商合作。如果你根本不知道自己在干什么，你的 AGI 顾问会兴高采烈地叫你把自己的脚打掉。在这个半可读的行业里，agent 放大的是技能差距，而不是抹平它。

一个很有启发性的例子是 CoreWeave 的启动流程，它本就属于业内最严谨之列。如今，除了严格的传统流程外，他们还让 LLM 对自家机群跑统计，寻找能更早标记故障、改进流程的模式。CoreWeave 还有一个 NodeBot，在自动告警之上提供轻量推理，总结情况并建议下一步，从而简化人工介入的诊断。例如，团队描述过一起事件：一个 热界面材料泵出告警本应把该节点移出集群进行调查，但与此同时一个 PMU halt 告警却争着要重启该节点，而不是先做分诊。尽管存在这种冲突，NodeBot 还是识别出了正确的操作。CoreWeave 拥有大规模运行 GPU 的经验，掌握无数训练数据中不存在的经验法则，并且有专家不断闭合反馈回路来改进对这些模型的使用；即便 OpenAI 用 AI 生成的 kernel 造出了一款爆款芯片，我们也不认为 CoreWeave 会在短期内受到那些用 vibe coding 搞数据中心自动化的新贵 neocloud 的威胁。

尽管——说实话——业内每个人至少在某些工作流程中都在使用 AI，但我们看到的唯一一个 AI 优先的产品是 AWS 提供的一个不错的 MCP 服务器。除此之外，所有文档名义上都是为人类编写的。提供商为智能体服务所能做的最好的事情，就是提供详尽的文档——理想情况下，比人类可能通读过的要详尽得多——并在集群上留下黄金配方，作为爬山法的起点。除了那些尚未 AI 优先的遗留流程之外，还有许多系统因为不是为智能体构建的而严格来说更差。例如，虽然让 AI 管理你的作业可能有利于集群利用率，但它并不尊重为辅助研究而建立的旧式系统。CoreWeave 和 Nebius 有许多 Slurm 层优化——真正的作业核算系统，带有 RBAC 和对用户及组强制执行的优先级、精心管理的分区、作业数组、日志和指标——当一切都以智能体方式运行时，这些都被浪费了。CoreWeave 还告诉我们，他们见过头节点内存耗尽，因为远程控制的 Claude 草率地编排集群。观察提供商如何改进其 AI 智能体支持能力将会很有趣。

展望未来

如本文通篇所述，我们历来一直在测试集群。但市场显然已经朝着两个相反的方向发展。

First, the biggest labs are buying bare metal at massive scale, measured in the 100’s of MWs, and have the talent in house to manage their own training clusters, orchestration software, monitoring and reliability. In essence, their neocloud providers (when they use them, and don’t do self build) are datacenter technicians who close tickets. This is a complicated business, and not to be taken for granted: there are 172 unique, documented ways an NVIDIA GPU can fail (XIDs). And many of these failure modes can overlap, resulting in an incredible surface area of different failure scenarios that require a combination of simple troubleshooting logic, human experience, and increasingly, agentic research by AI to recover from these failures.

Meanwhile, many startups are raising 10s or 100s of millions of $$ and spending effectively all of that money on compute. But since so much of this research is driven by RL, via continual learning on real production traces for long horizon agents, the needs for both offline, throughput-optimized inference, and online, latency-sensitive inference is very strong. At the same time, more and more customers are looking for the simplest way to purchase vanilla open source tokens for their workflows.

This leads to our next article, coming soon, where we will describe the Anatomy of an Inference Endpoint, moving towards the inclusion of inference endpoint testing in future versions of ClusterMAX. We believe that all neoclouds need to have an endpoints business going forward.

We will also be evaluating RL infrastructure beyond inference: sandboxes, hosted training, evals, and more. We continue to see lots of new providers enter (and exit) the market every day. It is an exciting time to be in AI, and neoclouds are at the centre of it.

If you got to this point in the article and are still eager to learn more about our individual experience on every provider, you should consider coming to work at SemiAnalysis!

We are actively hiring across Research, Consulting, and Technical Staff. We have multiple openings in our New York and San Francisco offices. Remote work is also acceptable.

ClusterMAX Engineer: all levels of experience with Slurm, Kubernetes, and GPUs considered (full time)

Tokenomics Engineer: all levels of experience with model evals, harnesses, inference endpoints, and RL infrastructure considered (full time)

Research Analyst - AI Infrastructure and Economics: all levels of experience covering the financials of neoclouds, neolabs, and frontier labs tokenomics considered (full time, or internship)

Technical Consultant: lead engagements from technical strategy to technical due diligence. all levels of experience from consulting and technical backgrounds considered (full time)

Thanks for reading, and we look forward to seeing you for future versions of ClusterMAX.

Appendix: Provider Reviews

Platinum

CoreWeave

CoreWeave remains the industry’s sharpest, sturdiest, most proactive neocloud.

Our CoreWeave clusters are exemplary in all crucial categories. The health checks work as intended, the reliability is excellent, and nearly all tests reach expected values out of the box. Along with Nebius, CoreWeave serves as a measuring stick for others in the industry—not only in the abstract sense of “excellent performer,” but also in the literal sense that we compare their clusters to lower-rated ones as a means of precisely describing others’ shortcomings. Whereas other neoclouds have to be nagged to refresh their moldy GPU drivers, CoreWeave has not only taken care of the basics, but stood up a significant independent stack for marginal performance and reliability improvements.

The ClusterMAX 2.0 report went into great detail on CoreWeave’s design decisions, including its bare metal provisioning, Slurm fork, use of DPUs, and health checks. Since then, one smart feature that the nerds at CoreWeave have added is called “GPU straggler detection,” which is integrated into their best-in-class dashboard. Rather than leaving their customers to dig through logs or do binary search over their fleet to isolate a bad rank, CoreWeave applies handrolled algorithms to NCCL telemetry and recommends a remediation strategy based on its findings. The CoreWeave team tells us that they use this on calls with their customers just about every day. There are all kinds of soft failures that don’t write an XID or leave any other obvious signature, but waste GPU hours as the job waits on a straggler.

Better than spotting bad GPUs is not scheduling them in the first place; CoreWeave also has systems to make sure every node clears a high bar before it’s trusted with a customer’s workloads. When a node is idle in CKS, CoreWeave runs burn-ins for 20-30 minutes of every hour, validating that scale-up and scale-out fabrics are healthy, GEMM numbers are up to spec, D2H and H2D bandwidth are high, and so on. These tests are pre-empted by customer workloads; invisible to the operator, they help make sure that it’s not production workloads that catch faults. CoreWeave is certainly not the only provider with active health checks, but these are unusually robust.

This diligence extends to CoreWeave’s bring-up process. Provisioning around 10,000 GPUs per week, CoreWeave believes that they have more data than God about what proves a Blackwell rack healthy. In CoreWeave’s view, Nvidia diags are good at catching many failures, but their uniquely deep dataset has allowed them to roll out a custom suite of supplementary tests, including for NVLink. The heart of this process is the Fleet LifeCycle Controller (FLCC), which “automates node provisioning, testing, and monitoring” and owns the path from power-on to handover. Importantly, as others operating fleets at scale can corroborate, there’s a long tail of obscure failure modes not well covered by Nvidia documentation. CoreWeave keeps statistics on these failures to better predict future faults and bakes its response patterns into FLCC. This is one reason that experience with operations at scale is a crucial criterion in our ratings. There is a huge difference between a newbie and a provider like CoreWeave that has rigorous runbooks, automation, and burn-ins. As described in the “Agentic Coding” section above, CoreWeave uses LLMs on top of this system to glean insights that would not be spotted by a human engineer. CoreWeave also works with manufacturers before delivery. Involvement varies by OEM or ODM and by production stage, but in general, the goal is to work backward through the supply chain to catch issues as early as possible: never catch in prod what you could have spotted in bring-up, and never catch in L11 diags what you could have spotted on the factory floor.

Their experience with Blackwell NVL72 bring-up and validation, along with their close relationships with Nvidia and Dell, have allowed CoreWeave to be the first provider to announce a VR200 NVL72 system passing L11 diags. CoreWeave has described VR as “the generation where we bring our own IP.” As the physical demands of compute continue increasing, in CoreWeave’s view, “how the building operates is now part of how the computer operates—they are not two individual things anymore.” Thus, CoreWeave introduced its custom Racky, Valvey, and RLCC in a recent blog post. Particularly interesting is Valvey, a programmable per-rack liquid cooling valve assembly. This gives CoreWeave fine-grained control over each rack’s cooling loop and also, in case of a leak or other emergency, the ability to trigger a shutdown. Despite beginning to adopt a hyperscaler mentality in building its own hardware, CoreWeave frames its approach as anti-hyper: whereas traditional infra emphasizes redundancy and is built to prevent failures, CoreWeave builds to fail gracefully and recover fast, limiting failure domains and allowing nimbler response to the inevitable failures.

Amidst the current trillion-dollar buildout, however, CoreWeave is responding to the massive increase in customer demand and dealing with increasing cost of capital. Of course, there is plenty of good news to mention. Its relationships with Meta, Microsoft, OpenAI, and NVIDIA—its largest customers—appear strong, and it recently struck a multi-year deal with Anthropic. It has announced splashy deals with HRT and Jane Street this summer. It is also aggressively expanding its capacity in APAC, indicating that it “expect[s] international markets to become a major driver of growth.” Yet it sits with $35B in debt and as the cost of capital increases, it takes longer to get projects off the ground. As a result, CoreWeave has become focused on long-term bare metal contracts, which are easy to fund with an investment-grade counterparty, rather than managed clusters, which have higher margins but which the market is less keen to fund. Moreover, our last writeup mentioned an impressive lineup of recent acquisitions, but since then its Core Scientific merger fell through and nothing else has been announced.

In terms of feedback, some of which is a bit nitpicky:

1. We wish that enabling local GPUDirect Storage could be an 1 click button in the UX console

2. Onboarding new users and where they put the the ssh pub key input in the UX console is extremely confusing to many users that tried

CoreWeave has begun to diversify from bare metal, first by offering self-service SUNK clusters, which gives them the option to sell into the on-demand market when they have capacity for it. It is also worth noting that not all “bare metal” contracts are the same. In all of CoreWeave’s bare metal contracts, they still manage the scale-out and frontend network, do burn-in, handle repair/replace, monitoring, and other general day-two operations. By comparison, other providers claim to be a neocloud, but are locked out of their own data hall. We will dig into more of these details in an upcoming article where we compare the different bare metal offerings from different providers.

CoreWeave has also launched a managed inference platform, which recently disclosed $100M ARR. We are actively testing this for an upcoming article on serverless inference endpoints, though were disappointed to find no support for Kimi K3 and GLM 5.3, and the only way to consume the public endpoints by the token is via the Weights & Biases portal, or via OpenRouter. For now, let’s just say that CoreWeave has some work left to do on their inference endpoints.

Nebius

Nebius was rated the 2nd-best neocloud in the world in ClusterMAX 2.0, yet it remained in Gold. This time, Nebius is unquestionably an industry leader with strong offerings in every category and the ability to command a significant price premium. Nebius moves into Platinum.

Nebius’s relationship with Nvidia remains strong, including a $2B investment in March, as Nvidia disclosed a 9.3% overall stake in July. Nebius was the first provider to deploy HGX B300s back in December of 2025, and has been among the first neoclouds in VR NVL72 bring-up, expecting to begin deployment late this year or early next.

Nebius’s buildout continues across a number of sites in the US and Europe as they recently raised their 2027 year-end contracted power target to 5 GW. In terms of offtakers, Nebius has signed large deals with hyperscalers Meta and Microsoft and smaller ones with Reflection AI and Palantir, making it one of a few neoclouds with multi-hundred-MW hyperscaler agreements that still regularly competes for much smaller startup contracts. They are the most active of all neoclouds in the short-term cluster market, serving many happy customers. The days of Nebius spooking prospects with their Russian accents are behind them. This is mirrored by its capacity procurement: unlike many of its peers, Nebius has opportunistically snatched up sites in the 5-20 MW range through a broad network of datacenter and software partners.

In terms of the clusters themselves, well, a good cluster is generally boring. On Nebius, burn-ins run without errors; Slurm is topology-aware; the packages we need are on the cluster and generally recent; WAN is good; the orchestration layers are free of footguns. One point of distinction technically between Nebius and everyone else is that Nebius builds and open-sources its preferred SonK flavor, Soperator. (CoreWeave, of course, builds SUNK in house, but it is not open-source.) This round, a Gcore cluster we tested used Soperator, and as did Voltage Park and FPT clusters from previous rounds. Nebius writes that “the real magic of Soperator lies in how we use ‘jail’ Persistent Volume”: Nebius provides a VirtioFS volume at `jail`, then bind-mounts it at `/`, so you don’t need to worry about pointing your cache to the right directory, or think much about taking your files with you when you `salloc` a node. You’ll see in our writeups of other providers a handful of gnarly issues with managing the storage system on SonK: Slurm and Kubernetes have different philosophies of persistence, and it’s not always straightforward to provide the illusion of bare metal when in fact you’re several layers up, standing on an ephemeral Kubernetes pod. Nebius’s approach makes this stress-free for the operator.

In particular, Nebius provided us a handful of storage tiers on every worker: `/jail` VirtioFS, `/home` NFS, `/mnt/data/` VirtioFS, `/mnt/local-nvme` ext4, and `/mnt/memory` tmpfs, as well as first-class S3. For I/O-heavy jobs, `/mnt/data` is recommended, whereas `/home` suffices for shared code and other data with lower-concurrency access. When we first tested `/mnt/data`, we saw very slow 4 KiB sequential allocating writes, while concurrent directory creation from 128+ ranks sometimes returned `EEXIST` due to inconsistent metadata visibility, killing our jobs. With our feedback on the pathological workloads, Nebius retuned this filesystem, and it proved sturdy and fast. We pushed it around and verified that the errors were resolved; per-client, it outperformed the median, and we could further verify that it scaled to handle demanding workloads across two racks.

Nebius has put a great deal of work into its health checks, and in our testing, they performed well. Nebius automatically returned a node to service after a synthetic error injection on our GB300 rack; the process is not yet optimized for speed, taking 8h40m from start to finish, but detection is immediate, and the process ultimately works. Moreover, the other GB300 racks we tested did not automatically reboot and return failed nodes, so the fact that this process is self-driving counts in Nebius’s favor. (As we describe in the “Blackwell and Grace Blackwell” section, it’s common not to autoremediate GB300 nodes at all, since there is no real way to hot-swap a node into an NVL72 rack, and troubleshooting is generally way more complicated than with HGX servers.) Nebius also provided us with an excellent suite of dashboards shedding light on, among other things, cluster health.

This all leads to Nebius watching their revenue per MW climb compared to CoreWeave’s (and their market cap along with it). Since they have so much more capacity to sell at the current prices, we expect this trend to continue. Anecdotally, we have seen Nebius get very aggressive during recent negotiations, including a case where they offered a tranche of capacity as a straight-up auction. In general, demanding high prices and sizable prepayments that reach 100% on 1 year commits seems like quite a nice business, especially when those prepayments can cover the entire capex of the servers at the current TCV. Infinite Project IRR anyone?

At their largest bare metal sites, Nebius continues to face delays in construction and permitting, including both Béthune and Vineland, which we have covered extensively for clients of our industry-leading Datacenter Model. But as more chips of all types come online, Nebius’s track record puts them in a strong position to continue to grow.

Overall, we are consistently impressed with Nebius. While we do believe they still trail CoreWeave on many technical aspects and relationships with both NVIDIA and the frontier labs, solid business decisions have established them as the default neocloud to serve the neolabs.

Gold

Google Cloud

Amidst the strange retreat of Google Brain and DeepMind from the frontier, GCP has been dealmaking like few others. Notable partners include Palo Alto Networks for $10B and Thinking Machines Lab for a “multibillion-dollar deal,” along with expansions of its lucrative relationship with AI safety org Anthropic. With acquisitions of cybersecurity startup Wiz for $32B and energy startup Intersect for $4.75B in cash, Google has been aggressive in expanding its technical capabilities. It also exposes itself to new revenue streams by ramping TPU sales instead of only renting them through GCP, a development we are very keen to track. Its own TPUaaS is still growing, though, thanks to a $5B deal with Blackstone that is targeting 500 MW in capacity. Google has gigawatts in the pipeline, and we look forward to continuing to track its continued competition with the other hyperscalers and labs for access to power and permitting.

With a unique position cutting across so many layers of the stack, Google may have more total engineering talent than any company in the world, yet it has a notably imperfect history in supporting innovation. In ClusterMAX 1.0, we noted a myriad of issues with its platform, yet we predicted GCP would reach Gold or Platinum promptly. It has taken until ClusterMAX 3.0 for this to come true, but Google is finally and comfortably among the best managed cluster providers.

The GCP GPU experience is less refined than that of our Platinum providers, CoreWeave and Nebius. The console feels like the DMV compared to the streamlined UIs of the younger, more focused neoclouds. Access required a Google Cloud CLI that was occasionally uncooperative, forcing us sometimes to switch to the browser-based connection in Google’s console, which was also unreliable. There was a small misconfiguration on our GB200 GKE cluster, where default StorageClass refused to attach to the `a4x-highgpu-4g` nodes, forcing us to try again with a non-default class to get block storage. Nitpicks aside, GKE is very well set up, and we enjoy using it. Google’s managed Slurm is officially generally available and quite solid, with good defaults and health checks configured. A lot of the testing that we did for ClusterMAX 2.0 on their managed Slurm offering is now valid, since the bureaucrats in GCP Product Management are allowing real customers to use the offering, not sticking it with labels like “Beta” or “Pre Release” or “not yet GA.” (By the way, many of the neoclouds we’ll discuss after Google should consider using some of these labels more often—the balance is somewhere in the middle here.)

The biggest point of distinction between our GCP cluster and the other clusters we tested was networking. Our first NCCL test hung because automatic GID selection chose the wrong address, but pinning `NCCL_IB_GID_INDEX=3` completed the run. Even then, our NCCL tests were unsatisfactory: you would like to see a roughly logistic curve as performance increases monotonically in message size, but instead we had the following jagged shape:

This could conceivably have been an issue with NCCL itself rather than a problem with our GCP configuration. NCCL’s heuristics don’t always pick the right protocol and algorithm for the message size and topology, and some hand-tuning is always expected. However, we have gigabytes of networking data gathered at this point, and we knew to expect better perf. The issue turned out to be simply that we did not have the gIB plugin enabled. This is a set of NCCL plugins Google that ships for improved perf on Google’s RoCE network. As of 26.07, gIB is baked into the NGC PyTorch image, so Google’s customers get the custom stack by default. With gIB installed, we got a better chart with 4.4% higher peak throughput for 16-node all-to-all. (Note that these tests are both with the multi-node NVLink fabric shut off to isolate performance of the scale-out network, which is RoCE in this case.)

While not perfect, this looks much better, and the topline number is solid. At this point, it was clear that the fabric was healthy, the configuration was good enough, and an engineer looking to squeeze more juice out of the system would have a solid baseline to start from.

This is amazing work by Nvidia and Google to set up auto activation for Google’s ConnectX-7/8 NCCL plugin. Previously, users would need to mess around with the correct library load paths and env vars to get it set up correctly and optimize performance on ConnectX NICs on GCP Nvidia GPU machines. We gave feedback a couple of quarters ago, and now, it is fully automated!

GKE’s health checks worked well. As with other providers who have experience with NVL72 systems, Google’s health checks intervened but left it up to the operator to bring the sick node back to the fleet. In particular, after our XID was injected, the health check marked `GPUUnhealthy=True` and `cloud.google.com/health-check-status=warning`, but left the node schedulable. This configuration was easy to work with, it’s what GCP’s customers prefer as a default, and it can be configured to the user’s liking to respond to errors of various levels of severity.

To continue improving, GCP could take a look at all the quality-of-life changes that Nebius and CoreWeave have made in the recent past. It should ship a better console that’s easier to navigate, simplify IAM and RBAC, and ensure cluster access is painless, either through standard SSH or `kubectl` or by making its existing CLI bulletproof. Especially now that gIB is available in the default Nvidia images, we hope that GCP’s operators have no trouble getting good performance out of their GCP scale-out. They could also continue to pursue neolabs in the mid-market, improve the hands-on support experience, and improve their relationship with Nvidia. GCP consistently loses so much business to smaller, less capable neoclouds, because they either lack the GPU capacity or refuse to serve the market at a certain price point.

We look forward to testing GCP’s VR NVL72 systems when they are available and seeing GCP’s continued improvements on usability and performance.

Oracle

As we mentioned in ClusterMAX 2.0, Oracle is in a unique position among hyperscalers in that its growth must come from large contracts. Unlike AWS, Google, and Azure, Oracle does not have any agreements to sell frontier tokens, and unlike Meta and Google, it does not have a non-frontier lab it can hang its hat on. Instead, it continues to cut huge deals involving OpenAI, Meta, and Nvidia, reporting “the delivery of 850MW additional datacenter capacity” from June through August, an eye-watering number. We project its relative growth in 2027 to be the largest of the hyperscalers, though from the smallest base.

We tested two Oracle clusters, which we will describe in order. The first was a GB300 NVL72 cluster on Slurm, set on a multi-planar network. This was vanilla Slurm, but due to customer demand, Oracle has Slurm-on-OKE on its roadmap. Once our allocation began, we tried to find our way in through the console, but it proved, like all hyperscaler consoles, a forbidding place.

Importantly, Oracle’s console did not support RBAC—nothing on the console is connected to SSH users—so all our SSH keys had to be managed on the machine. At one point, attempting to append to `authorized_keys`, our intern forgot a `-a` flag and removed all our previous keys. Classic user error, obviously, but this goes to show why we prefer to have a hardened path on a console. We’d rather not trust our interns—especially that one, no offense—to type the magic words on the command line when a mistake can lock us out of the cluster. Thankfully, Oracle’s engineers caught the mistake immediately and fixed it for us.

We hope they fix this and add enterprise grade RBAC.

As expected, Oracle’s cluster performed well from day 1. They provided us with scripts for InfiniBand and NCCL tests, which was helpful because vanilla scripts do not properly use all 4 rails. Burn-in was flawless and Oracle’s managed Lustre saw strong throughput. Oracle’s health checks worked as intended, draining the node on our synthetic error but leaving up to the user to decide whether or not to terminate and replace the GB300 node. Oracle’s managed Prometheus and Grafana were good, though not perfect. For example, they are not integrated with the end user experience for job accounting. However, on the positive side, health metrics are now integrated into Cluster Manager in the console, making it easier to get a view of the crucial statuses at a glance.

Next, we tested OKE with MI355X. This was the only provider other than TensorWave who let us test-drive AMD GPUs. The ride was rough.

As we complained about last time, at first, we needed to SSH into an operator node to steer Kubernetes. This time, the team quickly solved our Oracle CLI issues and we were promptly able to download our credentials locally. By far the greater problem on this cluster, though, was with the NICs. Peak performance was near line rate, so there was no issue with the hardware that we could detect. Rather, the issue was with the ionic drivers. Our per-rail `ib_write`/`ib_read` sweep across all eight NICs hit 391 Gb/s on every rail, then wedged the node the moment it finished: when the pods were deleted, the perftest processes never completed their `rdma_cm` teardown, the forced `kill` timed out the NIC’s destroy-CQ command, and the RDMA admin queue on `ionic_0` and `ionic_4` went dead. Worse, both ports still showed `ACTIVE` at 400 Gb/s and the node stayed `Ready`. The passive checks did not check out the RDMA control path, and no active check had run in eight days. We found the issue by grepping `dmesg` after RCCL kept failing. Oracle diagnosed it quickly on a weekend and rebooted: their own `ibwrite` health check had failed on this cluster in the same way days before. After a call where we explained the pathological workloads, they spoke with AMD and pushed different drivers and firmware to the cluster, which solved the problem.

Separately, we had a GPU intermittently dropping off the PCIe bus, neither caught by the health check nor properly surfaced in the dashboard. Oracle promptly swapped out the node and responded to our feedback on the dashboard.

This was a mixed episode, but Oracle’s responsiveness was good. It’s especially undesirable that no health checks alerted, even though these drivers are known to be flaky. Still, Oracle’s close relationship with AMD is clearly an asset. They were able to speak to the relevant teams, diagnose the issue, and put a patch in place quickly. This type of support is not guaranteed at organizations of Oracle’s scale. Interestingly, Oracle is also the only provider ranked Silver or above, and one of two ranked Bronze or above, with zero installed libraries with applicable CVEs. This is more of an expectation than a benefit for us, but it was a clear demonstration that their automation works for deploying and upgrading clusters.

If we were a neolab, we would come away from this testing experience with confidence in working with Oracle. Yet this is a moot point given Oracle’s increasing commitment to bare metal deployments for frontier labs such as OpenAI in Project Stargate. Success in that business depends more on getting government approval for gas pipeline construction than it does on RCCL tests.

Still, Oracle has made progress since last we tested them, and the experience is very good overall. We look forward to continuing to keep tabs on them going forward.

Silver

Lambda

Lambda was most recently in the news for participating in a $35B Anthropic deal for 350 MW in Nueces County, alongside Hut8 and Nvidia. Before that, they were rumored to be preparing for a 2027 IPO, and they separately raised $1B, $926M, and $1B in debt for their buildouts. Amidst all their growth, after judging them Silver in ClusterMAX 2.0, we were interested in evaluating Lambda’s clusters this time around.

Onboarding began with a nice deck and PDF explaining what to expect on the cluster, and access was sorted out promptly.

We criticized Lambda in ClusterMAX 2.0 for their reliability; this round of testing, it was one of Lambda’s points of emphasis. Their passive health checks cover the relevant conditions and ultimately autoremediated three errors we simulated during testing. Two synthetic XIDs were autoremediated in less than 15 minutes, and there were connected dashboards to monitor node health. Our test that triggered a genuine XID 79 by resetting the PCIe secondary bus reset brought the node into `NotReady` with no `GpuXid`, no cordon, and no monitoring visibility on the Kubernetes layer, but it ultimately rejoined the fleet after 2 hours. Our node-reboot tests on the Kubernetes layer also completed without incident.

The orchestration layer has improved significantly, though it still has a handful of issues. Slurm configuration left CPUs per task unset, so Slurm’s default allocation gave one logical core unless the launch requested more. Passwordless sudo is not enabled on Slurm, as we are using managed slurm rather than unmanaged Slurm. Lambda is the only provider we’ve encountered that makes this distinction, but clearly some of their customers want root access to their own cluster, while others do not (?). We enjoy having root on our own cluster.

Also missing was SSH to the compute nodes without an allocation. This creates a headache during debugging if someone else is running a job and you can’t attach to their allocation, or a job is stuck. Another reason why we enjoy “yolo mode” on our clusters.

More importantly, Slurm had software out of date, including CUDA Toolkit 12.0 at the default location /usr/bin/nvcc (which is so old it’s incompatible with the B300 SM100 architecture), and unacceptably old drivers. On the bright side, Slurm did have topology settings configured, and the NCCL recipes that they gave us worked well. We had no authentic errors during testing, and all our tests, including burn-ins and storage, showed good performance.

Our preoccupation with Lambda’s 1-Click Clusters seems quaint in today’s world of capacity reservations backlogged for months. Indeed, as mentioned in the opening, Lambda is increasingly driving its bottom line via bare-metal builds, apart from managed clusters altogether. Regardless, Lambda’s orchestration continues to improve, and customers are happy, which lands them in the upper ranks of ClusterMAX 3.0.

Microsoft

Using Azure clusters can be very difficult due to the Microsoft bureaucracy. Their engineers do the best they can, but this is not an organization set up to be flexible in accommodating its customers. Microsoft customers do not come first in matters of taste.

During testing, we were not allowed to have our own Azure accounts, so we had to use the cluster under the user of one of Azure’s engineers, which prevented us from poking around the console and experiencing the authentic onboarding experience. Our GPUs were set to be in Virginia, but thanks to some obscure internal decree, we were not allowed to have a control plane there. Instead, our control plane ran in San Antonio, Texas. Access required setting up a VPN, which worked after a bit (see: literally 3 weeks) of back and forth. Pasting was disabled on the bastion, so to run our tests, we had to manually type in our GitHub and HuggingFace tokens. Like, literally type them in one character at a time. We got them right first try though, nbd 😮‍💨.

Once everything finally ran, we found a handful of packages out of date, and bad default NCCL configs kept our first run from going through. On AKS, the `managed-csi-premium-v2` `StorageClass` could not attach, and we got `EAGAIN` trying to run `fio` at anything more than 4 clients on Azure Files Premium. No parallel system attached to our Slurm cluster meant that one of our tests, which loaded DeepSeek-V4 from NFS, timed out.

When we went to run burn-in, we had a genuine failure on 10 nodes of our Slurm rack, killing our job. Microsoft caught the errors, traced them back to an unhealthy host, and opened up a Guest Health Report for repair. An `scontrol` resume command brought them back, and after that, both our Kubernetes and Slurm racks completed their burn-in without error.

CycleCloud drained the node immediately after we tested Azure’s health checks by injecting a synthetic XID, but it did not resume or replace it. Because we had the whole GB300 rack, this was satisfactory, as described in the “Blackwell and Grace Blackwell” section above.

Due to the account issue, we could not see our Kubernetes dash by default. Instead, Azure’s engineering team cooked up a custom dashboard setup for us, and it was extremely detailed—a solid triumph over the bureaucratic straitjacket. We want to shoutout our friend Xu Xue for this one, as we are quite certain that he went way above and beyond the call of duty to build this out for us, but wouldn’t admit it. His curated dashboard is open sourced here, a great resource for anyone building Grafana dashboards to monitor Nvidia GPU clusters from scratch.

Satya once said “we want to build out Azure as being fantastic for the long tail of the workloads,” and that Microsoft is “not in the business of just doing five contracts with five customers being their bare-metal service.” Yet it is exceedingly rare to come across a VC-backed lab that uses Azure for training and inference. “Five customers for bare-metal service” is overstating it. Really, it’s two: OpenAI and MAI. Soon to be three, with Anthropic diversifying.

Azure’s most interesting asset is that it can serve OpenAI’s models and keep 100% of the revenue. They still have full access to all of OpenAI’s IP. Yet Azure can separately boast of large bare-metal deals with OpenAI, as well as an increasingly lucrative relationship with Anthropic. This is an incredibly strong position to be in. If only for the pause.

As Azure mostly competes on a different plane with enormous deployments and hundreds of billions of capex, its ability to manage clusters only makes it more flexible. Like Google and Oracle, Azure gets its cashflow from offtakers who need no help setting up a `ComputeDomain`. But based on the quality of testing, these three could still cut smaller, higher-margin deals with labs who want a higher-touch experience.

This does not seem likely to happen any time soon, though. Support and the console is a headache, even if Azure is the only hyperscaler where you can possibly get the Nvidia Reference Architecture on your scale-out networking. As much as anything, we like to verify that these hyperscalers are light on their feet.

Firmus

One of Nvidia’s favorite up-and-comers, and our favorite in Asia-Pacific, Firmus announced in August a $2B funding round at $10.5B post-money for expansion throughout Australia, Malaysia, and Indonesia, following a $505M round at $5.5B just 4 months before. Firmus also raised a $10B debt facility in March for its Project Southgate buildout in Melbourne and Tasmania. Just last week, it disclosed 900 MW in total contracted capacity and announced OpenAI as an anchor tenant in its Malaysian expansions. All in all, Firmus has disclosed 18,400 GB300 GPUs in the deployment process today and 36,800 later this year in Tasmania, with multiple GWs and tens of thousands of VR in the pipeline.

We had a rough takeoff but a fairly soft landing while testing one of Firmus’s dev clusters. Accessing our chips required using a Microsoft Entra account, Microsoft SSO, then downloading and configuring a custom pre-release version of the vCluster CLI. We’re suckers for new tech, but where cluster access is concerned, we’d much rather just SSH. After a few rounds of back and forth, with the help of Firmus’s team, we were on.

…on the cluster, but not quite off and running. The default setups on both Kubernetes and Slinky left a lot to be desired. Slurm login had no `sudo`, no `vim` or `nano`, HPC-X installed but not on `PATH`, and no NVCC. More importantly, instead of receiving 4 Slurm nodes on one rack and 4 Kubernetes on another, we had 2 of each on each. This was not necessarily a problem, but Kubernetes had `nvidia.com/gpu.clique` configured with nothing consuming it, while Slurm used `topology/flat`. Thus, the nodes visible to each orchestrator spanned multiple racks, and there was nothing to avoid a multinode job needlessly crossing the rack boundary. We had to infer node nomenclature to make sure our tests were well placed.

On top of this, the Kubernetes control plane’s `etcd` visibility flickered several times during testing. which killed our jobs and caused both the Slurm and Kubernetes layers to restart. This was ultimately attributed to an issue upgrading switch firmware and did not recur for the last 10 days. We also lost a NIC to a wedge during reboot, which was not caught by Firmus’s health checks as they were not active on the dev cluster we tested. Firmus was the only GB300 cluster we had that exposed GPU HBM as NUMA nodes, an interesting configuration that we mark as a failure because it lets you accidentally swamp device HBM if you overflow on the host. WAN was inconsistent but often quite poor, maintaining 0.25 Gb/s to NGC. The last of our problems was that one of the racks saw significantly degraded performance on its NVLink fabric, which was explained by a power setting that Firmus had been experimenting with on their dev cluster; fixing this brought our numbers into the healthy range.

Thus, by the time testing ended, we had satisfactory though underfurnished Slinky and Kubernetes layers, we knew how to access them, and we could get strong performance across our test suite. We understand that managed clusters are not Firmus’s priority right now, but we believe there is plenty of low-hanging fruit here for their technical team. As Firmus works to bring enormous capacity online, we look forward to seeing them continue to make life easier for users of their GPU software stack.

TensorWave

TensorWave is an AMD-only neocloud whose MI355X offering we tested via K8s and Slinky. In the last writeup, we emphasized the difficulty of TensorWave’s onboarding process, which required back and forth just to get on the cluster and had a large number speedbumps even still. The improvement this time round is significant. Onboarding was smooth, the `kubeconfig` sent directly to us worked perfectly, the console allowed us easily to add team members with SSH access, and the TensorWave support team was attentive throughout our testing.

Being an AMD-only cloud, TensorWave is fighting an uphill battle dealing with AMD’s software stack, which, despite significant recent progress, still lies well behind Nvidia’s. We saw this on our TensorWave cluster on RCCL microbenchmarks, with all-gather and all-to-all hanging at 32 KiB even using TensorWave’s binaries and recipes. Other collectives showed uneven scaling and poor throughput at certain message sizes. [Editorial comment: RCCL? More like Rick L!] In general, it can be said that the AMD networking stack lacks the community support that Nvidia tooling offers. The same goes, of course, on the hardware side. More important than microbenchmarks, we see in practice that the GB300 NVL72 systems have the most demand across the industry from labs: their performance is so superior that, even accounting for their higher prices, they often offer the best perf per dollar as well. AMD’s rackscale answer, the MI455X Helios, is still ramping up production, and is certainly not yet on offer by TensorWave. Thus, TensorWave’s place in the rankings depends not only on its internal improvements, but on AMD’s ongoing work to close the gap relative to Nvidia.

For what it advertised, our TensorWave cluster was good across the board. North-south bandwidth was strong. Storage performance was exceptional and burn-ins completed without a problem. When we pushed around the Kubernetes layer, its reliability was good, and the Slinky layer was easy to use.

The health check immediately recognized the error that we simulated, bringing the node into `drng`.

The dashboard oddly marked the node as “Allocated,” rather than being broken out into its own category. However, shortly afterward, we were presented with a nice juicy button to click to approve replacing our “sick” node with a fresh one.

The replacement completed in approximately 43m. We would prefer that the autoremediation occur by default—we’d rather not even have to click a nice juicy button—but otherwise consider TensorWave’s monitoring and autoremediation exemplary.

Overall, TensorWave is the leading AMD-exclusive neocloud, providing a comparable (and sometimes better) experience on AMD GPUs than hyperscalers like Oracle and Microsoft. And they are not stopping. TensorWave has entered into an agreement with Fermi for 222 MW in the Texas panhandle with expansion rights up to 650 MW, contingent upon Fermi securing project financing. It has a tailwind with a $350M June Series B, leaving it at $1.55B post-money, and is currently working on large deals with hyperscalers. With both OpenAI and Anthropic announcing strategic partnerships with AMD for the MI450 generation, we expect that TensorWave is set to benefit, and AMD-exclusive neoclouds may be able to break into the Gold tier. We look forward to testing TensorWave’s MI455X Helios once it is available.

GMI

GMI is a neocloud headquartered in Mountain View with roots in Taiwan. With a $500M Nvidia deal announced last November and $12B more this March, hundreds of GB300 racks in the pipeline, and plans for VR down the line, they are keeping on the frontier in terms of hardware. Recently, they have also been growing their fleet by making deals with smaller neoclouds and reselling the compute to their established customer base, which offers an attractive cashflow profile because it does not require up-front capital outlay. After judging them the “top Bronze neocloud” in ClusterMAX 2.0, we were interested to test the progress of their managed clusters.

Onboarding was a bit bumpy, including an empty Kubeconfig downloaded from the console and an SSH key that we uploaded but was never synced to the cluster. After some back and forth with the team, we got in. The problem, it turned out, was that the console was view-only for our testing purposes, so we could not verify its functioning. Last round, we did not get access to a self-service console at all, so we count this as progress.

As last time, the compute was up to spec, burn-in was clean, and the InfiniBand fabric was sturdy. One notable improvement since ClusterMAX 2.0 was in GMI’s storage. GMI provided a performant RWX NFS default StorageClass, which was also available on the Slurm layer at `/home`. We had difficulty trying GMI’s S3, which turned out to be due to a typo in their doc. But once the Brothers Karamazov Karasev hopped on a call and fixed it, we verified that the object storage perf was also good. The biggest blocker in terms of cluster configuration was that GMI expected the IMEX domain to be configured on the Kubernetes cluster by a ComputeDomain CRD. This is perfectly viable—many other highly rated providers do the same thing—but GMI’s Kueue would not tolerate the claim, so our jobs sat in `Suspended`. To work around, we had to submit our jobs in a namespace with no `LocalQueue` named `default`. We recommend that GMI document that MNNVL jobs need to skip the queue, or, much better, that GMI set up its Kueue to tolerate DRA resource claims.

GMI’s dashboards, which did not exist last time we tested, were useful, including hardware information from DCGM and NVLink and InfiniBand monitoring.

We could not get the full GMI health-check experience due to a lack of spare capacity, but what we could test performed well. Nodes were quickly cordoned at the Kubernetes layer, then rebooted and auto-uncordoned. The Slurm layer also brought a node into `DRAIN` soon after a synthetic error was injected. GMI had alerts set up in Slack, so we could see their system detecting the error and thread to discuss next steps with support, which was convenient.

Our GMI experience was solid in every crucial way, and we look forward to continuing to test them as their capacity ramps quickly.

Bronze

Amazon Web Services (AWS)

Our last report summarized the AWS experience as a “headache.” This time around, we had to break out some Amazon Basic Care Ibuprofen Tablets, Fever Reducer and Pain Relief from Body Aches, Headache, Arthritis Pain and More, Brown 200 Count to get through our testing.

The difficulty with AWS begins before you get on the cluster—indeed, before you even try to provision the cluster: it begins when you contemplate AWS’s zoo of managed offerings. To get this all straight, we had to make a Venn diagram:

(Caveat lector: anyone not interested in the minutiae of AWS’s GPU offerings should skip this ¶.) SageMaker is AWS’s fully managed platform meant for minimal maintenance. If you don’t want to handle any infrastructure, it offers serverless model customization and data science environments that abstract away the underlying compute. At the next level up is the SageMaker HyperPod line, which supports both Slurm and EKS. The HyperPod line offers many conveniences like health checks, autoremediation, dashboards, and workload-level features like checkpointless training out of the box for users who want to preserve their `ssh` or `kubectl` access. If you want infrastructure abstracted away but want to handle the ML stack yourself, you can leave the SageMaker umbrella and choose EKS Auto Mode, which takes care of node provisioning, autoscaling, OS patching, and node repair. EKS Auto Mode is a wrapper on top of “managed EC2 instances,” but these are not actual EC2 instances in the sense that they do not show up in the EC2 console and do not provide the same level of customizability as ordinary EC2 instances. In EKS Auto Mode, users do not have access to the control plane, and they must allow AWS to recycle their workers every 21 days for security. Finally, for even greater customizability, there is EKS + Karpenter, which allows the user to manage autoscaling themselves with the open-source tool Karpenter.

Despite Amazon’s reputation as “customer-obessed,” it is clear that this novel-length GPU menu is an example of the adage “you ship your org chart” rather than a reflection of genuine demand segmentation. And the bureaucratic frustrations hardly end once you’ve ordered your cluster. This is a criticism of the organization rather than of any team in particular—AWS has shades of the 3rd-Century Roman Empire. To improve its standing in the next round of ClusterMAX testing, we recommend AWS choose another empire and/or time period for reference.

Nevertheless, there is a unifying trait of all these offerings: they fail to provision the first time you try. Thus, we made another clarifying Venn diagram:

This finally brings us to the AWS operator experience. One SageMaker cluster we provisioned in the console failed due to insufficient IAM permissions. Some of the EKS clusters used Terraform to simplify provisioning, but that path is not battle-tested and requires being in the right subdirectory and on the right commit of the right branch of the right (in-flight) repo and knowing exactly which knobs to turn without making an invalid request. One cluster came up without Lustre and would not let us attach any, despite extended back-and-forth with AWS engineers. For the purposes of testing, we provisioned a new cluster in a new region to gather data on filesystem perf, but if we were a real-life lab, this would have been crippling, at least temporarily. The AWS team we worked with was consistently helpful, but it is clear that they don’t have the liberty to go solve problems or even insight into how the system fits together, especially compared to the flat, nimble neoclouds we’re judging them against. It should not require six engineers on multiple calls to provision a Slurm cluster in the first place. (Edit: AWS would like to clarify that it was five of their engineers and one of our engineers to make six.)

Once the goods were delivered, there were no blockers that kept us from getting reasonable perf out of the gate. The drivers, firmware, and most utilities were up to date. This is the bare minimum from a reputable provider like AWS, however, and there was no shortage of rough edges elsewhere.

A basic point of distinction between the AWS clusters we tested and their peer B200 clusters is that AWS uses Elastic Fabric Adapter (EFA), its proprietary scale-out fabric. In our testing, peak EFA throughput was strong. On a four-node 32-GPU all-to-all NCCL test over EFA on B200 we measured 59.44 GB/s busbw at 16 GiB message size, which works out to about 92% of scale-out line rate given the proportion of traffic that crosses the node boundary. On other networking tests with other traffic patterns and launching mechanisms, as long as message size was large enough, throughput was among the best B200 clusters. However, EFA consistently demonstrated worse latency than well tuned InfiniBand comps. For example, when the message size was reduced to 64 KiB, the same four-node 32-GPU all-to-all NCCL test took 85.26µs over EFA and under 50µs on InfiniBand peers; a delta of 30-45% between EFA and InfiniBand was consistent across small-message tests, which are latency-dominated. Although this may seem like a nitpick, scale-out latency is a critical metric for modern expert-parallel workloads, which frequently send small token vectors across the node boundary. This 64 KiB NCCL test, for example, is similar to the expert routing step of models like Kimi K3 and DeepSeek V4. Our findings are corroborated by an excellent Perplexity technical blog post about optimizing EFA, which reported similar results: EFA demonstrates satisfactory peak throughput but pays a latency penalty of 20µs or so on “the message sizes exchanged during MoE dispatch and combine.” (Note that this Perplexity post compares EFA to ConnectX-7 NICs whereas we tested ConnectX-8, but the nameplate throughput of the two generations is identical.) It’s not a good sign when Perplexity’s cracked engineers are writing painstakingly detailed blog posts about how to “enable” basic functionality on your custom network stack.

EFA still requires extra setup for expert-parallel inference in upstream vLLM and SGLang. AWS reports successful vLLM deployments, but users must install EFA userspace libraries and build the relevant communication components with EFA support. Depending on the workload, these include DeepEP, NIXL, or Mooncake Store. AWS’s DeepEP fork supports EFA through NCCL GIN. DeepEP V2 uses NCCL GIN, while its NVSHMEM-based V1 path is now documented as legacy. Upstream integration and default container packaging still require work. NCCL EP is another route under development; the linked vLLM integration remains a draft PR. In general, EFA is not a priority for frontier communications libraries, and users must wait weeks or months for what support they do get.

AWS is the only cloud distinguished in this way. Every other provider we tested uses InfiniBand or RoCE. Of course, AWS’s customers with GW deals have the resources to make sure their EFA is well tuned, but we still think it a shame that the largest provider’s scale-out stack is uniquely troublesome.

In a different way, storage was another basic cluster service that often proved uncooperative. The B200 cluster we provisioned in `ap-south-1a` failed every request for storage with an ambiguous `Insufficient capacity` error, and the `Failed` filesystems took 5h46m55s to delete.

To test Lustre performance required another cluster in another region, which meant another protracted bringup—another evening of terminal-watching trying to discern whether Terraform was making progress or not.

Claiming storage on AWS via Terraform is like raising a toddler: you can only watch and hope that once it’s done destroying it will start creating. And once the storage is created, it is not necessarily straightforward to access. On the Kubernetes layer, one cluster provided no StorageClass by default, and its only provided StorageClass left our PVC indefinitely in `Pending` because it had the wrong driver. This required us to define our own StorageClass with the correct provisioner.

Powered on and wired up, storage perf was mixed. The provided FSx stack struggled significantly with ordinary buffered I/O, reaching 9.2s in p99 sequential read latency on fio testing, compared to 312ms on the same metric with buffering disabled. Greater latency on the buffered setting is expected, but this is a pathological delta.

AWS’s health checks’ performance was, again, mixed. Our EKS Auto Mode cluster caught the XID we injected, set `AcceleratedHardwareReady=False`, and gave us a new and healthy node only 24m after the original injection. This is good. Meanwhile, our SageMaker HyperPod Slurm cluster immediately detected the XID, brought the node into `DRAIN`, and completed most of the reprovisioning process before tripping over its own feet when it required a deep health check before it could rejoin the fleet. This deep health check was submitted as a Slurm job and couldn’t be scheduled until, well, the node was added back to the fleet. Thus, an operator had to manually push through a `scontrol ... State=RESUME` command, at which point the fleet returned to full health. With our feedback, AWS promptly fixed this circular dependency, and a follow-up test on a single H100 node succeeded without intervention. Better.

Our EKS Karpenter cluster’s health checks were even more self-defeating. The `dcgm-server` DaemonSet did not tolerate the `nvidia.com/gpu:NoSchedule` taint, so it did not attach to the worker node, causing the monitoring agent to report `error connecting to nv-hostengine` and flip `AcceleratedHardwareReady=False` with `Reason=DCGMError`. Karpenter then marked the `NodeClaim` for termination, the replacement node was marked unhealthy for the same reason, and the system spun in circles until we manually modified the DaemonSet to tolerate the GPU taint. The health check had another allergic reaction when it noticed the IMEX daemon did not schedule. IMEX configures the NVLink fabric across nodes, but we had ordinary B200 nodes speaking via EFA, so there was nothing for IMEX to do: this daemon’s failure to land should have been harmless. Instead, the monitoring log emitted the schizophrenic message

ignoring IMEX health code on non-NVLink multi-node system, code=122

sending condition to exporter,

Reason=NvidiaFabricError,

Severity=Fatal

This would again have recycled a healthy node if we did not manually pin NodeRepair=false to keep overzealous garbage collection from blocking our work. (Edit: AWS informs us that this was not overzealous garbage collection; rather, it was a genuine hardware error with an unclear error message. This is worse. In any case, an `nvidia-smi -q` of ours reported “Fabric Health Summary: Healthy” after this supposed failure.) Finally, as our testing came to a close, Karpenter tore down our B200 cluster 39 minutes before the EndDate, force-evicting a 4-node MPIJob in progress. We have gone into such detail here because these bugs are almost unbelievably elementary for the world’s largest cloud. Perhaps the single painfree part of the AWS lifecycle was Karpenter scale-up and scale-down, which completed in minutes with no errors. However, GPU capacity must be secured in large blocks and far in advance, so snappy autoscaling with Karpenter is a meaningless feature.

Despite the many problems with their product, AWS’s specialists and engineers were intelligent and responsive in our interactions with them. They were eager to show what they had improved, sought out feedback on documentation, circled back to rectify errors in testing, and even built a nice, forward-looking MCP server whose skills we plundered for our own codebase. AWS’s issues lie in its organizational structure. The number of basic issues that have slipped through the cracks is astonishing in an organization with AWS’s resources.

In February, AWS announced the expansion of its deal with OpenAI by $100B; in 2Q26, it posted 37% growth in net sales YoY; its Trainium business has hit $25B in ARR with triple-digit YoY growth; its Anthropic investment is appreciating to the tune of tens of billions per year; Bedrock continues to make bed-rocking margins on bed-rocking income.

Managed clusters are not AWS’s priority.

Gcore

Gcore is a Luxembourg-ish neocloud building out in ambiguous region called « Europe, » where it relies on other providers for capacity. They have made no flashy announcements recently, though they are adding Blackwell to their Hopper fleet. In ClusterMAX 2.0, we gave Gcore a Silver rating after testing their Kubernetes and SonK offering via Soperator. This time round, we only tried the SonK again.

Our insight into Gcore’s capabilities was limited by their lack of capacity: most providers granted us at least 4 B300 nodes for at least a week, but Gcore was so constrained that they could only part ways with 3 H200s. Nevertheless, there was plenty to keep us on our toes on this cluster.

Our testing on the Soperator layer required several rounds of back-and-forth with the Gcore team to fix some rough edges, but the team consistently proved helpful. When the majority of our first sweep failed due to a misconfiguration that prevented non-root users from running their workloads, the team responded promptly, confirming `kernel.apparmor_restrict_unprivileged_userns=0` would get us unblocked. Gcore’s team was again responsive in correcting another poor default when each Slurm worker reported `RealMemory=2048 MiB`, apparently having erroneously inherited CPU configurations. This kept us from even running `nvidia-smi`, and we were glad to see it quickly repaired.

Debugging was made slightly more challenging on the Soperator layer because inter-node SSH was disabled by default, but the team pointed us to a `soperator-createuser` convenience script to make life easier. The team also troubleshot an error with their dashboard as we tested.

On the Kubernetes layer, there was no default StorageClass, so users had to select NFS explicitly. The storage on the Slurm layer was properly configured, with the easy-to-use Soperator default of NFS at `/`, providing the illusion of a single shared volume wired to all the nodes. This storage performed well across workloads. The N/S network was fine, and the E/W network ultimately performed up to spec, but the RDMA devices had non-standard names and no `topology.conf`, so configuration took extra steps.

The XID that we injected was quickly caught and displayed in Grafana; interestingly, the node went `Down` on the Slurm layer, but on the Kubernetes layer nothing changed, even though it was injected via K8s. We could not test autoremediation because Gcore had no spare capacity.

We were impressed by Gcore’s attentiveness and their Soperator offering that, after some massaging, did the basics right. We look forward to testing Gcore’s B300/GB300 management, including its handling of XDR ConnectX-8 NICs, and seeing the improvements of their Soperator and Kubernetes products.

GMO

GMO’s managed Slurm cluster delivered strong networking and storage performance on two B300 nodes. Its main gaps were restricted profiling, limited enterprise controls, and a lack of autoremediation.

Slurm arrived with a configured head node, partitions, and topology.conf. Drivers and CUDA/NCCL/HPC-X modules were current and consistent across nodes, and Pyxis was available. Onboarding was straightforward, though the Japanese-only console required some manual ctrl+c, ctrl+v translation for us to understand.

Grafana access took us five days of troubleshooting. We initially missed the macOS login-keychain instruction, then entered the certificate password where the system password was required. GMO investigated and repeatedly followed up. The final mistake was ours, but clearer certificate instructions would have saved time. Or, no unnecessary custom certificates to access the website. Japanese security theater strikes again.

The sixteen 400 Gb/s rails per node delivered strong NCCL results, though two nodes provide less scaling evidence than our usual four-node minimum. MPI launches needed explicit NCCL_IB_HCA settings, which GMO said it already supplied in nccl.conf, but which didn’t work for us initially. Burn-in showed no network errors, retries, or ECC errors, with normal temperatures and power.

Shared home/data/work directories worked out of the box. FIO sequential reads and writes, random reads, and torch.save were solid, built on a well-tuned DDN Lustre filesystem. However, our contrived tests such as `import torch` and vLLM Serve from shared storage performed poorly; GMO offered to investigate but we chalked it up to classic Lustre metadata performance tuning issues and moved on. Like many other Lustre implementations, the GMO storage offering lacked snapshots, automated backups, and disaster recovery options.

The cluster also lacked `sudo`, Docker on the workers, and required the use of snodes, a convenience script that they built to show metadata about the cluster, since they prevent users from running any scontrol commands on their own cluster. Security theater once again.

As a final step, SSH to worker nodes on the cluster was also blocked. And inside Slurm jobs, `RmProfilingAdminOnly=1` blocked Nsight Compute’s GPU counters, and perf was not usable for our profiling needs. We believe that managed clusters should allow users to profile their own workloads if they choose to go for the yolo-mode option. We wanted it, but GMO could not provide.

Finally, Grafana exposed useful DCGM metrics but lacked XID detection, estimated TFLOPS, SM Active, and SM Occupied views that we enjoy for performance and reliability monitoring. The Slurm job summary view was also pretty basic. Our synthetic XID 79 log injection produced no observed alert or node drain. GMO said it runs dcgmi health in the Slurm prolog, confirmed XID 79 was not whitelisted, and agreed to investigate for future. We finished our testing by determining that GMO lacked any autoremediation system in place, or at least we couldn’t verify it.

Overall, GMO’s lacking Kubernetes and enterprise features such as RBAC, SSO, and customer-accessible audit logs put it in the camp of niche, local player serving the Japanese market effectively. Our support experience with their team was helpful, but followed standard Japanese business hours. Going forward, we expect GMO to continue to be a leader in Japan, but lack the ambition required to expand their business beyond their local geography.

Verda

Helsinki-based Verda (formerly DataCrunch) now has $450M in total funding as of September 2026. They maintain a close relationship with Nvidia and continue to aggressively pursue financing. On the technical side, they have implemented a number of features since we tested them in November, and we are optimistic about their engineering team and roadmap.

Last round of testing, Verda’s Slurm was officially in beta, and they did not offer Kubernetes. This time, they offer a managed Kubernetes and have moved their Slurm offering to Slurm-on-Kubernetes to simplify deployment. We tested their cluster on both the Slurm and K8s layers; it had its fair share of rough edges, but represents significant progress and a strong starting point for continued development. We also found that security was solid.

The purchase and onboarding process was smooth, and the cluster provisioned without hiccups in 30m. The Grafana dashboard worked out of the box, and we SSHed onto the cluster without incident.

Once we got on the cluster, though, there were some bumps in the road. Our Slurm testing began with an interesting footgun: when we SSHed in, we assumed we were on the Slinky layer, but we actually had landed on the outer VM with wrappers that ran `salloc`, `sinfo`, and similar commands via `kubectl exec` into the `LoginSet`. The behavior was unusual—all Slurm commands escalated us to `root`, for example, and our Python environment disappeared—and it took an evening of debugging before we puzzled out that our Slurm access was illusory and a different SSH command would be required to access the Slinky layer properly.

Verda’s engineers were responsive, quickly hopping on a support call in which they reasonably explained the rationale behind these convenience functions and pointed us to the supported path. Although this particular feature caused us a headache, it’s bullish when engineers have gone out of their way to try to make their cluster easier to use.

The Slinky layer proved serviceable but imperfect. Verda ran a NCCL-test prolog that exceeded Slurm’s 10-second `MessageTimeout`, causing a worrisome though harmless `Prolog hung` alert every allocation. More significantly, Verda did not support Pyxis or Enroot, so we had to stand up our environment via a massive `.sif` file. Once our workaround was finished, we completed an 8-hour burn-in without incident and measured satisfactory numbers on our microbenchmarks.

Meanwhile, the Kubernetes layer could only be steered through the bastion, offering no way to download a `kubeconfig` through the console. This is a relatively simple fix to align Verda’s offering with users’ standard workflows.

The biggest shortcomings that surfaced during our performance sweep were of the shared filesystem. Gross throughput numbers were excellent, but cross-node file locks seemed simply not to be enforced. We found 100/100 exclusive-lock violations for both `flock()` and `fcntl()` across nodes, while same-node controls refused every attempt. This strongly suggests the issue is with `virtiofs` failing to publicize locking behavior. We also saw corrupted data when downloading with multiple clients from Hugging Face and `ENOENT` failures on Elbencho with 32 clients. This is more low-hanging fruit, and crucial given the fundamental importance of a reliable shared filesystem when running Slinky.

Lastly, Verda’s health checks programs are not yet fully configured. They are literal “health checks” in the sense that they evaluate whether a node is healthy, but there is nothing to consume their output—they exit `1`—“yup, this one’s dead”—and move on. A user must therefore bring their own reliability suite if they don’t want bad nodes silently killing their workflows. There is of course also no autoremediation provided by Verda.

Verda consistently communicated promptly and clearly and seemed to have engineers with the liberty and motivation to improve their product. On September 10, after our testing completed, they pushed a number of changes to Instant Clusters that appear meaningful. We look forward to seeing them continue to sand the edges of their Slinky and Kubernetes offerings, tune their storage system, and implement robust health checks and autoremediation.

Moonlite

Moonlite, a neocloud headquartered in Chicago, is a new inclusion as of ClusterMAX 3.0. With an ex-Crusoe founding team, Moonlite currently operates a Hopper fleet and has recently handed over its first B300s and GB300s, which it will be continuing to ramp up in the coming months. According to LinkedIn, Moonlite has the fewest employees of any company we worked with this round and oriented around engineering.

We tested Moonlite’s Kubernetes and Slinky. The handover process was smooth, with a nice onboarding document defining validated recipes and a Kubeconfig that gave us everything we needed. By default, everyone shared the same root user on Slinky; when we asked for non-root users, Moonlite made a `moonlite-adduser` convenience script in half an hour and baked it into the login-pod image same-day. This shows Moonlite’s responsiveness to feedback, but also their greenness. We loved their response. We would rather have a battle-tested system, ideally through the console, to handle basics like this.

On the hardware side, everything we tested worked well out of the box. NCCL tests hit ConnectX-7 specs, NFS was good, compute benchmarks were solid, and burn-in completed without a problem. However, it bears repeating that we were testing H100s, which by now are quite familiar to the industry and are therefore easier to get right than the GB300 NVL72s we had with other providers. Nonetheless, we were happy with our Hopper perf.

One component of the system that we could not test was the NVMe, which existed but could not be accessed. Nothing exposes it: no local-path `StorageClass`, `hostPath` unavailable, and Slurm workers are Slinky pods with only VAST `/home`. On the Kubernetes layer, there was no default `StorageClass` at all, and unqualified PVCs hung `Pending`.

We tested Moonlite’s health and monitoring system via a PCIe secondary bus reset, and Moonlite alerted us in 11 minutes. With our permission, they cordoned the node, ran diags, and returned it to our fleet after 57 minutes total, including a few minutes where they waited for our instruction on how to proceed. In practice, with actual customers, they define policies ahead of time so that their team can follow a runbook depending on the particular error. Moonlite’s typical SLA is to respond within 15 minutes as their team receives alerts in Slack and PagerDuty. We hear from their customers that so far that this is exactly what they do.

In our view, any autoremediation process that brings us back to a full fleet within an hour is good enough. But while manual processes are fine, at scale, we want to see this automated. We would also prefer greater visibility on the cluster in the event of an error. The only change visible to the user was that the node, while remaining `Ready` on the Kubernetes layer, advertised 7 GPUs instead of 8 in the time between the reset and Moonlite’s manual intervention.

The Slinky layer lacked any container runtime on the workers but otherwise got the job done, and had most of what we ask for on the login pod. Moonlite’s Grafana was fine, but it lacked detail at the scheduling layer, and does not display the most crucial information, which is error status.

What Moonlite attempted to do, it did well. We look forward to testing them again as their products mature and they get some GB300 calluses on their hands.

Together

In ClusterMAX 2.0, we wrote the following about Together:

Together is a strong provider with a robust cluster offering for both Slurm and kubernetes, but it is held back from the gold category due to reliability issues.

This round, Together’s issues—generally but not exclusively having to do with reliability—are cause for downgrading it to Bronze.

Two nodes of ours failed during a compute test; one came back, but the other was sent to RMA without notice, so we were left wondering where it went. Later in testing, Together’s Slurm `HealthCheckProgram` tried to run `dcgmi diag -r 1` under a 45s timeout on nodes that were already scheduled; the diag did not land, the timeout was misread as a failure, and the nodes were drained mid-run. Together quickly fixed this with our feedback. Separately, one of our nodes could not access VAST because its storage NICs were in `operstate=down`, which Together explained was due to a race condition in NIC initialization in provisioning their new lightweight VM stack. There was nothing to catch this error, and manual intervention was required to reprovision a healthy node and bring our test fleet back up to 4. While we like the sound of “reduced cluster provisioning time,” we obviously don’t like to see it affecting reliability.

Together’s health checks did work correctly in some instances, showing progress since ClusterMAX 2.0. When we triggered a PCIe secondary-bus reset on the Slurm cluster, the node was drained in 67s, and an automatic node replacement completed in 35m7s without manual intervention. A synthetic error also cordoned a node on the Kubernetes layer and correctly did not trigger autoremediation because we had `Auto Repair` turned off.

The Together Kubernetes and Slinky layers were both functional, though with a handful of challenges. Interestingly, Together advises to avoid using the Kubernetes layer of the Slinky cluster: to convert Slinky nodes to K8s, we were instructed to tear them down and reprovision as vanilla Kubernetes. The Slinky offering should therefore be evaluated as plain Slurm. Summarizing its pain points as briefly as possible: OCI image extraction failed because of default settings on `/tmp`, so we had to redirect extraction to `/scratch`; when this happened, Slurm exited `0` anyway; NFS rejected any filename containing `:`; and non-login worker shells have neither `nvcc` nor `mpirun` on `PATH`, so ordinary batch scripts behave differently from interactive setups and lack the tooling that they need. Together also offered S3, but it was slow and unreliable. None of these is fatal, but adding them up, along with other more minor errors, meant a few dev days before we were able to run our stuff unimpeded.

A public IP check confirmed that this flaky B300 cluster had org `AS53735 IREN`. This N+0 Prince George datacenter is one of two datacenters in the industry notorious for unreliability, along with Crusoe’s cursed Reykjavik site. We strongly suspect that one or both of these datacenters was not inaugurated with a Land Acknowledgement. We know of multiple exceptionally unhappy customers of Together struggling with link flaps, power issues, cluster access being shutoff, random upgrades that aren’t approved by the users going wrong and not being rolled back, and weekend-long outages that eventually get blamed on a single ISP (yes, a single ISP at this site—no redundancy in Northern Canada). Maybe their other sites are better but the TogetherAI site at IREN Prince George is completely bad.

After playing the blame game for some time, it is clear that reliability issues cannot be blamed entirely on underlying providers. Unforced errors from customer support engineers have created a lack of trust between customer and provider.

Of course, managed clusters are not Together’s bread and butter, nor are they its primary source of revenue. When, in July, they announced an $800M Series C, the post made passing reference (at best) to their managed clusters business, emphasizing, on the contrary, that “Together AI is a research-driven company.” The offtakers of their 250MW in Saudi are as yet undisclosed, but we are aware that Together would prefer to get paid per token, not per GPU-hour. Despite this, we hope that Together manages to Get It Together, and address its cluster reliability issues to make its Slurm and Kubernetes products something to be proud of.

Crusoe

The Crusoe section of ClusterMAX 2.0 ended on a cautious note:

Crusoe is at risk of being downgraded to ClusterMAX Silver due to many of their top individual contributor engineers quitting, leaving the culture in their cloud division beginning to resemble big tech. There are too many middle managers across the organization, especially in engineering. This has caused incredibly slow moving releases, such as their AutoClusters feature, leaving us with concerns about the future of Crusoe’s public cloud offerings. Chase needs to do a rapid course correction if he doesn’t want to lose all of his 10x engineers and eventually lose their Neocloud business with it.

In the eight months and change since we published that paragraph, Crusoe has had its fair share of wins. Its valuation has grown from around $10B to $30.9B, it has announced 900MW and 1.0GW campuses in Texas and many more elsewhere, and it remains an innovator in the wild wild west of datacenter construction. They have the largest real pipeline of any datacenter builder.

Its inference endpoints business is also extremely highly regarded, following the acquisition of some cracked Israeli engineers from Atero. They have also landed awesome deals with firms like Jane Street.

Plenty to keep their hands full and business churning, some clusters are amazing, but others from them are terrible.

Yet this round of ClusterMAX testing finds that Crusoe’s neocloud business has not course-corrected rapidly enough. The problems go back to the fundamentals: reliability and hygiene.

We emphasize health checks in our expectations and our write-ups, but genuine hardware failures are rare in testing because our ClusterMAX allocations are too small and short-lived. Crusoe bucks this trend. On an H100 cluster we had in May to test AutoClusters, we saw an impressive barrage of XIDs—more real errors on one cluster over five days than we saw in the rest of testing combined.

In a pattern that would repeat throughout testing, exceptionally poor performance in some areas was paired with mature services in others: this storm of XIDs was accompanied by convenient emails, attractively styled, politely notifying us of the errors.

In addition to being responsible for the majority of this round’s hardware failures, Crusoe bears the distinction of having had the oldest Linux kernel and the oldest GPU drivers we tested. Crusoe has since rectified these shortcomings, but at the time of testing there were several software versions that were well below our minimum expectations: the aforementioned NVIDIA drivers failed our `cmax audit security`, along with Docker, ConnectX firmware, and `runc`. These issues speak to a lax security posture and poor engineering hygiene.

Nor are these software-version criticisms merely academic. The Linux kernel that our B300 cluster ran, `5.15.0-185-generic`, was earlier than a Linux patch series called “writeback: Avoid lockups when switching inodes.” The particular problem that our kernel experienced is well explained by the description of commit `e1b849c`, which fixed it for upstream kernels:
```
There can be multiple inode switch works that are trying to switch inodes to / from the same wb. This can happen in particular if some cgroup exits which owns many (thousands) inodes and we need to switch them all. In this case several inode_switch_wbs_work_fn() instances will be just spinning on the same wb->list_lock while only one of them makes forward progress. This wastes CPU cycles and quickly leads to softlockup reports and unusable system.

```

When might a `cgroup` that owns thousands of `inode`s exit? One answer: when a Slurm job is torn down. In our case, as an ordinary workload wrapped up, every CPU on the NUMA node ended up stuck contending for the same `list_lock`, making the processor unusable. The local NFS client, which happened to be on the frozen NUMA node, couldn’t get CPU time, and a liveness probe for that client timed out and reported the node unhealthy. Kubelet tried to control the damage with a `SIGTERM` and a `StopContainer` operation, but the kernel was taking its sweet time doing the laundry—reassigning dirty `inodes`—and took over two hours before it could honor the termination requests. The final result was another `NotReady` email from Crusoe—this one, uniquely, downstream of an old CPU kernel.

By the end of our testing, this kernel was not merely old and unperformant: it was officially marked a security vulnerability by CVE-2026-64378, which exploited the buggy writeback behavior, published on July 25 with severity 7.8; a separate 7.8 for this kernel was posted on July 27. Thus, we added the Linux kernel to our collection of Crusoe software exposed to CVEs. Crusoe has informed us that they will be upgrading to a 6-major Linux kernel, a necessary patch.

When we ran our XID injection on Crusoe’s Kubernetes layer, the error was quickly detected and a `Node Replacement` workflow was triggered, but the node could not be put in `drain` due to a mismatch in Slurm naming convention that slipped through Crusoe’s CI. The health monitoring system spun its wheels, repeatedly catching the error and trying but failing to mark the node unhealthy. Crusoe’s engineering team diagnosed the error impressively quickly, rolling out a replacement within hours. However, our replacement node had faulty storage NICs, immediately bringing it into `drain`. This issue persisted after the node’s VM was manually reset. Crusoe’s engineering team has informed us that the failure of AutoClusters to remediate in this latter event was “a known gap,” and that they will be “rolling out auto-remediation actions for this in a few weeks.” This is a surprisingly lethargic response to a basic failure of reliability.

Last and briefly, it bears mentioning that this cluster’s WAN was among the slowest we tested, clocking in between 0.47 Gbps and 0.78 Gbps depending on the context. Iceland has some skinny cables getting off the island.

All these problems add up to a cluster notorious for its flakiness, as confirmed by our friends at frontier labs who complain about both the number of failures and the apparent lack of interest from Crusoe engineers to fix the issues promptly. To be clear, we at one point viewed Crusoe as the #2 neocloud in the world, and as a result the #2 place in the world to rent GPUs. We made recommendations to this effect. Oh, how the mighty have fallen.

But while Crusoe’s machine images are badly due for a refresh, and their engineers can’t get Claude Tags or Codex approved, they do have automations like this set up in their internal Slack:

Does a CVE from November of 2025 count as “historical baggage”?

In all seriousness, this is a problem of priorities, a problem of bureaucracy run amok. We are not sure what to make of job postings like the above one for a senior Linux kernel engineer. Literally, we’re just asking everyone to keep things up to date. You don’t need to hire Linus to understand why this makes sense. So while it’s cool to see Crusoe flexing its muscles and paying top dollar for a good person, it’s not a lack of people or funding or technical skill that has brought Crusoe here in the first place. Our complaints focus on the most basic issues possible, not the Slinky configuration or console or CLI that no doubt take up much more engineering time and that we enjoy using.

We enjoy working with Crusoe’s engineering team and believe they have plenty of talent to improve their standing. Frankly, in the last few months, things seem to be getting back on track. Engineers are locked in, pushing code, and excited about upcoming releases. The Atero acquisition has been worth its weight in gold, skyrocketing Crusoe to the top of many buyer’s lists when it comes to inference endpoint quality. We love to see it.

Still, in the managed clusters business, TCO comes down to goodput, and goodput comes down to reliability, and Crusoe’s clusters are exceptionally unreliable.

Crusoe comes in towards the bottom of our Bronze category for this round.

Prime Intellect

Prime Intellect closed a $130M Series A this July at a $1B valuation, disclosing $100M of “annualized revenue run rate across compute, RL and post-training, sandboxes, inference, environments, and evaluations.”

We tested Slurm and Kubernetes B300 clusters and mostly enjoyed both of Prime Intellect’s orchestration layers, although each was missing a handful of packages we would’ve liked to have. The huge Slurm control plane was nice and snappy, having machines with 2TB of RAM for some reason. Plenty of swap space to run salloc.

After a quick fix to standardize the NCCL bootstrap interface, our networking tests ran up to 13 nodes on the ConnectX-8 NICs with excellent throughput. Storage was no issue, with Weka /data volume performing very well at 4 nodes and scaling steadily beyond. PrimeIntellect’s managed Grafana was one of the nicer ones we encountered. However, we were on a cluster that was still going through burn-in, so we saw a steady stream of errors. An easy way to test the monitoring and health checks on the system!

First, a node on our Kubernetes cluster was marked NotReady, apparently due to on-site techs monkeying with adjacent servers; it returned within 5 minutes. Then, during a PyTorch networking test, a different Kubernetes node stopped posting kubelet status and the distributed job could not complete; the node came back after a reboot and the job completed. The next day, 2 Slurm nodes went down, one with NVLink issues and the other stuck after a reboot. Both returned promptly after intervention, but the latter was soon back to its shenanigans with another surprise reboot, causing a storage test to fail. An MPI test aborted in its last phase, and the corresponding Slurm job remained in COMPLETING as all workers remained unreachable in the brief period before we handed the cluster back.

All of these reliability issues can be blamed on the fact that we got on this cluster in a small window while they were still doing burn-in for a paying customer. Fair enough. But the lack of compute at Prime’s disposal that would allow them hold back hot spares points to a bigger problem of growing capacity constraints. Prime is a reseller of other’s compute, and depends on their underlying providers to guarantee customers with a high quality experience. This comes with challenges that high quality training/RL frameworks, dashboards, and maniacal Slack engagement from a cracked team of engineers can’t always cover up.

With that said, Prime’s health checks performed impressively throughout, quickly intervening and limiting the damage of each error. We thought the system’s default responses were sensible and even liked the Slack alerts, which the team promptly followed up on. Given that Prime gets its capacity from a number of providers, the quality of their software layer is important for labs to feel comfortable with what they’re getting. Of the marketplaces we’ve tested, Prime seems to be the best.

This leaves Prime Intellect with a straightforward assignment, and one that we cannot have direct insight into: improve reliability, build a base-load of compute capacity, and rise in the rankings as a true neocloud. We have no doubts about the team’s technical ability; they have done a lot of hard things correctly on this cluster, and are the only provider on this list that injects real experience training models into their approach to software and support. For better or worse, their work elsewhere is much more impressive than the bread-and-butter managed compute offering evaluated here.

We got a preview of their hosted training product during this period and are very impressed. RL infrastructure is decidedly more complex than a simple managed Slurm or Kubernetes cluster, since its built on top of the same foundation but with three components that need to stay in sync:

1. Training

2. Inference

3. Sandboxes/environments

Coming soon, we will be evaluating hosted training infrastructure providers and RLaaS companies for hire through a project provisionally called PostTrainingX. If we had to put out an initial cut at those rankings, Prime would be in the hunt for Platinum! Stay tuned.

DigitalOcean

This marks the first round of testing that we have been able to get the full DigitalOcean developer experience. Marketing itself as “the first cloud built end-to-end for the inference and agentic era,” DigitalOcean distinguishes itself from “bare-metal focused neo-clouds and inference wrappers that lack cloud platforms.” In fewer words: DigitalOcean sells, among other things, managed clusters.

DigitalOcean’s managed clusters are orchestrated by Kubernetes, which proved solid but slightly immature. Their onboarding doc asked us to install Multus ourselves, then create network attachments, install MPI Operator, and create the MPIJob manifest. There’s no reason not to set this up for the customer in advance. Once configured, our cluster performed well, hitting all the standard benchmarks for compute and XDR networking, and completing burn-in without incident. The cluster also lacked a RWX StorageClass out of the box, adding another basic setup step that the better K8s providers handle for you. Once the storage was configured, its performance was interesting: sequential writes were approximately 6x faster than sequential reads at several different client counts, the inverse of what we see on most clusters. We expect that DigitalOcean’s sequential read performance could be significantly improved with retuning.

When we went to test the health checks, our first XID injection went unnoticed: the monitoring system did not detect anything, and the node remained available for scheduling. When we alerted the DigitalOcean team, they responded promptly, found the bug in their system, and asked us to reperform the test later in the week. The second try went smoothly, and we had a fresh node 50 minutes after injection.

When a neocloud is “built end-to-end for [...] inference,” that means it doesn’t offer Slurm. The DigitalOcean team helpfully offered us a runbook to stand up Slinky, and it got the basics right, but would require significant work before it could be considered production-ready. No SSH access, nor, as mentioned previously, any RWX volume by default, which means that there is nowhere to keep a `/shared`. Anyone who plans to run Slurm on DigitalOcean should expect to stand up their orchestration themselves.

DigitalOcean will be adding 60 MW of capacity throughout 2027, and, aside from its managed K8s, it has an interesting selection of GPU services like Droplets and bare-metal Hopper and MI300X that fall outside the scope of ClusterMAX testing. But no plans for GB’s or VR’s from what we can see at this point. We look forward to the team continuing to improve its managed Kubernetes service the next time we test.

Hyperstack

Hyperstack, under parent NexGen Cloud, is a UK-based neocloud that advertises Kubernetes clusters in the US, Canada, and Norway. It recently raised $45M at a $354M valuation with plans for many AI new products for “full-cycle development.” As it announced a $34M debt facility to build out a B200 fleet, it also disclosed plans to deploy 4,500 B300s later this year and for an additional 56 MW online in 2027.

The managed Kubernetes cluster that we tested marks significant progress since ClusterMAX 2.0, and a solid foundation for its continued growth. Compute and networking tests were up to spec, and burn-in sustained good numbers with no hard errors. Our Kubernetes node-reboot recovery was quick, and we had no trouble mounting and unmounting PersistentVolumes. The N/S network was fine, and storage was good enough, although we were not able to get Weka and VAST, which Hyperstack usually offers, on our timeline.

Our synthetic XID was detected in under a second with a `NvidiaFatalXid=True` flag. The test was awkward as Hyperstack had no extra capacity, so we had to take a node out of our fleet to simulate a spare. The support team notified us that they had detected the error 12m after we injected it, and our spare rejoined the fleet after a total of 52m. Interestingly, at this point, the “sick” node was still not cordoned, which took until 1h30m to register on the Kubernetes layer. This was all driven by manual intervention by Hyperstack’s SRE team after our synthetic error triggered Grafana alerts.

Overall, this test was a success, and we appreciate Hyperstack’s commitment to helping it go smoothly. However, as the team is aware, the amount of human support required makes us worry about its replicability. Hyperstack offers 24/7 support and targets intervention in under 60m in the event of single-node failure, but it’s better not to have to count on another engineering team pushing the right buttons before your cluster gets back to full health.

The only outright error during our testing was that two pods, `cilium-envoy` and `csi-hyperstack-node`, crash-looped with `Too many open files`. This was a minor annoyance and did not affect our workflows.

We look forward to continuing to work with Hyperstack as they grow their team, gain experience managing larger clusters, and continue to automate and harden their infrastructure.

Participation Ribbon

Vultr

Vultr advertises itself as “the world’s largest privately-held cloud infrastructure company,” which they are not, and “the world’s largest privately held hyperscaler,” which they wouldn’t be even if they were. Still, they are a relevant player in neocloud land, with modern GB300s and MI355X including a claimed 50 MW AMD site in Ohio. In ClusterMAX 2.0, Vultr’s cluster was delivered with a handful of basic errors. Unfortunately, the pattern is the same almost a year later.

The images that Vultr built for our testing had many libraries with applicable CVEs, including CUDA, DCGM, Docker, runc, and ConnectX firmware. Our Slinky login pod did not mount `/shared` and lacked the Pyxis plugstack config, so we had to orchestrate from a worker pod until the Vultr support team reprovisioned the login pod mid-campaign, fixing both issues. There was another basic problem with the storage visible from the K8s layer: the default storage configuration was incompatible with our bare-metal cluster. The default StorageClass, `vultr-block-storage`, provisioned and bound without complaint, then the pod sat in `ContainerCreating` forever. However, the Lustre tier, which came attached to the worker nodes, was excellent, achieving 91.7 GB/s aggregate read across 32 clients. Grafana was configured but only partially correct, as NVLink read 0 due to unconfigured `DCGM_FI_PROF_NVLINK_*` fields.

Vultr is known to have been experiencing issues with the reliability in its scale-out networking, which has discouraged partners from beginning or expanding existing deals. We recommend that Vultr continue to improve the sturdiness of its clusters and work to create golden images with all relevant utilities and orchestration reasonably configured and up to date.

Vessl

Vessl is a new Korean neocloud who fancies itself a “multi cloud orchestrator” rather than a broker, buying bare metal long-term and providing a management layer with SLAs on top. They claim 5,000 GPUs currently in operation and have an ambitious target of 100 MW by the end of 2027. An old Vessl BizOps job posting says to expect “Speed. We MUST move fast, learn fast, and iterate constantly,” demanding “a minimum of 60 hours per week.”

We love the ambition; however, the immaturity of Vessl’s cluster, which we sampled in its Kubernetes and Slinky flavors, gives them plenty to work on. Our onboarding began with a notice to pin three settings to render the RDMA fabric usable. A heads-up is better than nothing, but we much prefer to be given a cluster that performs well out of the box.

The third bullet was in response to a particularly cumbersome configuration: by default, due to Vessl’s Kubernetes `LimitRange`, each container would hard-fail if it used more than 2GB of host memory. Although a defensive default might make sense in certain cases, this is a poor setting. Nodes rarely are called on to run more than one job at a time, so containers can expect to have all of the host DRAM to play with—hundreds of GB—and OOM-killing them for going above 2GB is stifling. We recommend that Vessl relax this limit, bringing it in line with the physical capacity of the host while also making failure more graceful.

In terms of management, Vessl provided no way to handle users on the Slinky layer except via traditional Linux tools on the machine. The SSH key we added from the console did not show up on the cluster, nor did the one we added via the Vessl CLI; we later learned that these utilities are independent of Vessl’s Blackwell clusters. We could therefore use only the private key and `kubeconfig` DMed to us on Slack.

The cluster provided a reasonable (though strange) `add-user` convenience script to make new Linux users.

It did not provide, however, basic packages like `pip`, `git`, or even `sudo`.

The cluster therefore took some work to get to the starting line. But our intern was still not done putting in reps as Linux sysadmin. Vessl ran the Slurm login node with an ephemeral OverlayFS root, so runtime changes to `/etc/passwd` and `/usr/bin` existed only in the container’s writable layer and were not preserved when Kubernetes recycled the pod. This caused our users to disappear midway through testing.

Another Slinky misconfiguration brought down the Slurm layer for part of testing. Our workloads wrote to an overlay that they did not clean up, accumulating 863 GiB in the root volume, causing Kubernetes to mark the node `DiskPressure` and restart it. This not only brought down the local worker, but the Slurm controller daemon as well, rendering the whole Slurm layer intermittently inaccessible. We infer that `slurmctld` was scheduled on to a node also running `slurmd`. Co-scheduling a `slurmd-pyxis` pod with a `slurmctld` is generally a bad practice for reasons well illustrated by this anecdote: the data plane should not be bogged down by the administrative duties of the control plane, and the control plane should not be within the blast radius of the data plane. Vessl quickly stopped the bleeding by having Enroot write to a larger volume, but to our knowledge the root cause—the co-scheduled worker and controller—remains unaddressed. It is clear that Vessl does not have experience running Slinky at scale in production.

A separate error made Kubernetes access inconsistent. The URL in our `kubeconfig`’s `server` field did not always resolve to the same IP address: Vessl had erroneously published the private `10.x` IP to the record set in addition to the public one. Sometimes our `kubectl` command resolved to the right (public) IP, and we could access, but other times it would resolve to the wrong one and our request would hang until a retry went through. We recommend that Vessl remove the `10.x` IP from the public record set.

The `NetworkPolicy` we set on the Kubernetes layer of our Vessl cluster was accepted but not enforced, as HTTP requests between all tenant pods were accepted without regard to defined rules. Vessl did not have a health check program configured and should prioritize this along with all the mundane “quality of life” features indicated above.

The hardware itself that Vessl offered performed well throughout testing, including on NCCL tests over InfiniBand. Given that it has such a large fleet advertised as coming online soon, Vessl has every motivation to harden its offering, providing superior services to its users and enabling it to command superior prices. We will be interested to observe Vessl’s progress when we next test.

Runpod

Runpod is a neocloud headquartered in Moorestown, New Jersey with a fleet including RTX PRO 6000, H100, H200, B200, and B300. This June, they raised $100M at a $1B valuation, announcing on Twitter that their revenue doubled to $240M ARR from February to June.

The cluster we were able to test was in Seattle, and was so new that it did not even have storage configured yet. Our “Instant Cluster” was just about instant, spinning up in under 2 minutes.

Runpod’s user model is unusual. In an ideal world, we manage permissions via a web console with RBAC, where we can add our team members by email, increase or decrease privileges, make groups, and drop in our SSH keys to be automatically synced to the cluster. Instead, taking advantage of its snappy provisioning, Runpod recommends minting Instant Clusters as needed. Engineers then operate their cluster with a single tenant and return them to the pool when finished. This means that budgeting, monitoring, and other such policies are handled above the cluster level, rather than via traditional tools like Linux user management or Slurm accounting. Runpod’s design choice is viable, but it is revealing we did not end up using the cluster in the suggested manner, preferring to fall back to the more familiar pattern of making Linux accounts tied to our SSH keys.

Our first burn-in failed as one of our nodes had port 1 links go down on two NICs, suspected to be due to an issue with a leaf switch. There was no health check to catch the error, but there was a Grafana dashboard that highlighted the error if you know where to look. With our feedback, the Runpod team quickly improved the dashboard to make InfiniBand state easier to spot.

On the second burn-in attempt, the networking performed well and the test completed without error.

The cluster was so new that there was no shared filesystem to test. On the software side, though, our initial audit found many things to be improved. The CUDA Toolkit, GPU driver, and NIC firmware were deprecated for security reasons, and the CUDA Toolkit was additionally a liability because 12.8 does not support SM103, the B300 SM architecture. This caused some of our benchmarks to fail on the first try. The cluster was also missing lots of expected tooling, with no container runtime and many basic HPC packages absent. Slurm also occasionally marked waiting jobs as `InvalidAccount`, which prevented jobs from running, even though `AccountingStorageEnforce=none` and `AllowAccounts=ALL`. The workaround was to spam `scontrol update JobId=<jobid> Account=me` until Slurm accepted.

We also saw a node drained when an image extraction filled our 50 GB root overlay, which was shared with `/var/spool/slurmd`, leaving `slurmd` unable to write. Our tenant lacked credentials to `scontrol update`, so our testing continued shorthanded. Separately, Linux rejected `unshare -Ur`, so we had to run with `udocker` instead. Our standard XID injection route was blocked because we lacked permission to write to `/dev/kmsg`; we had no mechanism to issue node reboots; passwordless `sudo` was not enabled by default. The theme here is clear: within the Runpod environment, users lack many permissions they need to get their work done.

We find the Runpod team unusually receptive to feedback and eager to explain their decisions. We like that the majority of their job postings are in serious engineering roles, and we hope they continue to work to strike a better balance between ease of access and developer control.

Bitdeer

Bitdeer was a new inclusion in ClusterMAX testing as of ClusterMAX 2.1. Yet another crypto miner pivoting to AI, Bitdeer already has an impressive 1.75GW in total electrical capacity according to its most recent announcement. However, this mostly feeds cryptocurrency mining rigs: our SemiAnalysis Datacenter Industry Model estimates they only have a handful of AI MW online. Still, Bitdeer is steering hard toward the higher-margin GPU business, with hundreds of MW already in the pipeline over the next couple of years as it works on sites, either converted or now, in Knoxville, Wenatchee, Fox Creek, and Rockdale, as well as in Norway and Malaysia. Hoping to grow its business vertically as well as horizontally, Bitdeer is not content with mere colo deals, with fledgling offerings from token-as-a-service to managed agents to, of course, managed clusters.

We spent a lot of time trying to figure out what exactly Bitdeer expected us to test when they gave us credits on their public console. Health checks were not set up, the monitoring dashboard did not work, many basic libraries like PyTorch and Docker were not installed, networking settings were empty, IMEX was not configured, and the console did not support ordinary conveniences like RBAC. Bitdeer’s monitoring installer downloaded a binary for the wrong CPU architecture. SSH worked eventually, but oddly required an RSA key rather than ed25519. Kubernetes queries repeatedly lost their connection. In short, the management of this cluster was strictly nominal. The most that can be said is that it had GPU drivers, NCCL, CUDA, and an OS and kernel, and that all were sufficiently modern.

Once we had the time to set things up, performance was fine. GPUs hit expected numbers on burn-in and NVLink supported the expected traffic.

However, the cluster’s WAN performance was exceptionally bad, averaging around 0.1 GB/s across several tests. The cause of this poor performance is unknown to us; needless to say, it would be an impediment to real-world usage.

Most of all, getting support was a massive pain. The entire technical team was based in Asia (Malaysia, or Singapore in our case), which meant 8-12h TAT when we were trying to troubleshoot issues. It was impossible to consider Bitdeer for anything other than this tier.

We look forward to seeing Bitdeer’s improvements in future rounds of testing as its managed cloud offering matures and it implements features like health checks and managed Kubernetes. It will be interesting to see where Bitdeer allocates its time and attention, as management has unambiguously announced that its neocloud business is not its main focus: “Executing on colocation lease agreements is THE top priority.”

Shadeform

Shadeform continues to focus on brokering and small scale marketplace reselling of individual GPU VMs with a small team. We enjoy the interface and working with the Shadeform team, but without more ambition they will stay in our lowest tier.

Radiant

Radiant was formed after Brookfield acquired Ori, a UK-based neocloud that we discussed in ClusterMAX 2.1. Unfortunately we have not been able to test a working, secure cluster. Despite almost a year of planning, along with quite a bit of marketing effort, and announcements that mention Billions and Gigawatts, Radiant is yet to deploy a Blackwell GPU.

FPT

FPT suffered from a number of security issues during our testing, which we described in ClusterMAX 2.1. We have yet to re-test and are not aware of any Blackwell capacity.

Core42

Core42 is a dark horse in this category. They have a solid team, allocation of GPUs, political backing, and an infinite money glitch in the form of Mubadala. We expect to see them put the pieces together and rocket up the rankings as they enter the US market.

Latitude

Latitude is making progress to keep their modest cloud business afloat, being recently acquired by Megaport and as a result going public. We had the chance to test some RTX Pro 6000 Blackwell servers, but as long as Latitude is missing the latest and greatest GPUs, we expect them to be stuck in this category.

IBM Cloud

We have not had the opportunity to re-try IBM Cloud since our horrendous experience in ClusterMAX 2.0.

BuzzHPC

Buzz is making waves in Canada, latching onto the Sovereign AI wave associated with the Carney vs Trump showdown, to the tune of 1.2GW and $50B in Saskatchewan, in partnership with Bell and others. This is quite the complement for their 300MW+ elsewhere in the country. BuzzHPC is one of the many public companies on the ClusterMAX list, via parent company HIVE Digital Technologies Ltd. Unfortunately, all the GPUs that we have tested from Buzz were in clusters of questionable quality, relying on questionable software vendor partner choices. We have not had the opportunity to re-test since ClusterMAX 2.0, but we hope this changes in the near future.

Neysa

Indian neocloud Neysa announced a $1.2B in financing this February, including $600M equity and $600M debt, in a round led by Blackstone that left them at $1.4B post-money. They claim “9,216 air-cooled NVIDIA B300 staged between December 2026 and March 2027,” with an equal number of AMD MI350X in the pipeline as well. Impressive growth, and clearly the leader in India in this regard.

This round, our testing unfortunately revealed a cluster that was riddled with errors. Many software packages that came pre-installed during cluster provisioning were months or even years out of date, with several serious CVEs applicable. CUDA Toolkit 12.0, for example, was released in December 2022, making it older than Neysa itself. Somehow this was installed on our cluster at /usr/bin/nvcc.

We enjoyed working with the Neysa team and encourage them to continue to improve their security posture and shore up their fundamentals as their capacity spikes and the bring on large customers from abroad. While current Neysa customers are mostly stuck on the Hopper generation, with a few B300s available, they are no GB’s (and, as a result, no direct liquid cooling) that have made it to production on the Indian subcontinent.

Vast.ai

A reseller/marketplace that keeps coming up in customer conversations for random GPUs on-demand, representing one of the better partnerships teams in the industry, finding unused capacity all over the world. Unfortunately, the UI is just terrible, so hard to use.

Hyperbolic

We haven’t had the chance to re-test Hyperbolic since the last round, but they got a basic security compliance attestation done finally! Hyperbolic moves onto the list.

STN

STN has been very confusing. One of the earliest clouds to deploy B300s, and one of the only providers with an option for “private cloud”, where customers take care of procuring the cluster themselves, and keep the chips on their balance sheet, but STN takes care of cluster operations. We hear from multiple customers how they enjoy this arrangement and prefer it over price gouging neoclouds.

We’ve always enjoyed working with the STN team, but unfortunately each of the last three times we’ve engaged with the team, we have run into some performance, reliability, or configuration issue, asked for a fix, been promised a fix, not gotten the fix, and then time has run out.

We regrettably keep STN in this tier until we can see a complete end to end testing experience as the customer would experience it.

Not Recommended - Underperforming

Based on hands-on testing, these providers can quickly rise to Bronze by fixing one or more critical issues, such as offering only older GPUs, missing basic security attestation (SOC 2, ISO 27001), misconfiguring key server features (leaving PCIe ACS enabled, or failing to enable GPUDirect RDMA), or charging for GPU hours during cluster creation or hardware downtime.

SharonAI

Sharon AI is an Australian neocloud that claims 132MW in its pipeline, with 116MW already contracted out, against a current footprint that we estimate around 4 MW. They do not have SOC 2 or ISO 27001. We were able to test 2 bare metal H200 nodes to verify basic functionality. We look forward to testing Sharon AI down the line as their offerings mature.

IREN

IREN continues to come up in our conversations with neocloud customers, and usually for the wrong reasons. They’ve got rock bottom prices on HGX B200 and B300 GPUs in Prince George and Mackenzie in Northern BC, Canada, and we’ve heard the stories. Multi-day power outages, network upgrades, storage failures, air quality controls leading to link flaps, XID’s everywhere. The #1 worst site in the industry according to users. And we are some of their users, as we’ve gotten to test IREN GPUs from multiple providers that resell their capacity. A lot of our users that aren’t having an good time that we are hearing from aren’t renting from IREN resellers but IREN themselves

With that said, the new builds in Childress and Sweetwater look much better (read: not N+0 power, cooling, and ISPs!). We’ve covered plenty about these sites in our Datacenter Model, suffice it to say that despite all the technical shortcomings in their cloud services, we expect the $2.1B investment from NVIDIA will help them cut less corners this time around.

We recommend that IREN stop pretending to offer managed clusters and inference endpoints in public marketing and investment materials, and focus on their bare metal offerings, where there is plenty of market demand for their services.

Hydra Host

Hydra Host raised a $100M Series A in June 2026, with NVIDIA among its backers. They are focusing on brokering deals, and our earlier testing is all we have to go on for managed cluster experience. We need to test a real managed cluster before considering an upgrade.

FarmGPU

We couldn’t make much progress in our testing of FarmGPU. The Slurm layer did not advertise GPU resources properly, while Kubernetes did not expose any RDMA devices for scale-out networking.

With that said, we very much appreciate FarmGPU’s open development culture, including their solid Grafana monitoring experience and detailed notes on provisioning. We find the technical team to be trustworthy and solid to work with, even if they are stretched too thin just trying to get a few small clusters deployed correctly.

WhiteFiber

WhiteFiber raised $159.4M gross at its initial IPO closing in August 2025. Our prior testing found good network performance but unusable Slurm/Kubernetes integration and weak job monitoring. More recent customer feedback suggested improvement, though the ephemeral Slurm login filesystem remained a concern. We need to verify that these operational problems are fixed.

PaleBlueDot

PaleBlueDot raised a $150M Series B led by B Capital in January 2026. We previously found its marketplace easy to use for individual VMs, but it lacked a complete managed-cluster experience. Unfortunately, they have shut down their on-demand console in favor of serving some big customers in dedicated datacenters.

Akamai

Not much has changed in the last year at Akamai/Linode, with single-node GPUs being the focus and no managed Slurm or Kubernetes available. We’ll keep watching to see if this sleeping giant wakes up anytime soon.

Hetzner

Hetzner’s low-cost hosting model still provides some cheap PCIe GPUs, specifically the RTX 4000 and 6000 Pro Blackwell Edition. We’re waiting to see if they take the plunge with anything bigger, building on all their datacenter operations experience to really serve the AI market.

Mithril

Mithril, formerly Foundry, announced $80M in funding back in 2024 to “restore the promise of public cloud to AI”. Today, it’s a marketplace wrapping 3 Nebius availability zones with H100 or H200, though only the H200’s have an option with a scale-out network.

There were some announcements about TPU support that we were quite excited about, but that partnership seems to have been scrubbed away.

OVHCloud

OVHcloud has substantial infrastructure globally, operating in tons of datacenters globally, some of which are shared with top neoclouds on this list. But our concerns remain that general purpose IaaS is not the growth engine of the AI age with companies doing crazy things to get access to the latest and greatest. Another sleeping giant.

Massed Compute

Massed Compute last announced capital raise involved up to $300M in equity and revenue-share financing from Digital Alpha in August 2025. They have used this to drive reasonable growth for a bare metal business, but that alone does not cut it on the ClusterMAX rating system, especially since their slop cannon chatbot is still getting indexed by web crawlers.

Not Recommended - Unavailable

Our “Not Recommended - Unavailable” tier is the same as ever, but to clarify, for this section we include four types of companies:

1. Those that we have previously tested who now claim to have zero spare capacity and/or refuse to test with us. This includes Fluidstack, Cirrascale, Lightning AI (recently merged with Voltage Park), Scaleway, CUDO Compute, Denvr Dataworks, and Atlas Cloud.

2. Those that are yet to launch, yet to be tested, and are generally interesting to us, though we believe some unnamed of this bunch are misleading investors (i.e., lying) by claiming in public marketing materials that they do managed clusters when they really just do bare metal. This includes SpaceXAI, Mistral, Poolside Infrastructure Company, Nscale, Highrise, Corvex, Andromeda, Volta, Firebird, Tatra, Sesterce, Yotta, Boostrun, GlobalAI, Argentum and Qumulus.

3. Those who are big and important, but who we cannot test properly due to geographic or regulatory concerns. This includes Alibaba Cloud, MegaSpeed, BytePlus, RunSun, SK Telecom, Naver Cloud and Indosat/Zankore/Lintasarta.

4. Those who are too small to be relevant. These are still covered in our market view, but not part of the rating system.

Fluidstack

Fluidstack has (currently) exited the managed clusters market, as far as we are concerned, because their focus has shifted from managed Nvidia clusters to bare-metal TPU deployments at 100K+ chip scale. We look forward to testing with them again in the future, with whatever chips they may offer at that time.

Cirrascale

Unfortunately, after receiving a legal letter to modify our previous article’s description of our experience working with Cirrascale, we have not been able to establish a productive working relationship.

Lightning (merged with Voltage Park)

Following the merger of Lightning and Voltage Park, the combined company has unfortunately run out of GPUs, and we haven’t been able to test the reconciled products. This is despite the company consistently advertising that they are the 3rd biggest neocloud in the world in terms of GPUs deployed. They are not in the top 10.

Scaleway

Recently, we hear that Scaleway has de-prioritized a number of customer engagements and decided to voluntarily break ties with NVIDIA, in favour of AMD, for seemingly no good reason. We think this is a shame.

CUDO

CUDO did not make a representative managed Slurm or Kubernetes environment available for this cycle as it focuses on other business priorities (namely, managing the build out of a bunch of bare metal datacenters). We look forward to reassessing CUDO when an environment becomes available.

Denvr Dataworks

Despite changes in leadership and random bouts of activity in the community, we have yet to make any progress testing Denvr and have not seen any meaningful growth in the business that would allow them to dig themselves out of the hole their founders put them in.

Atlas Cloud

We haven’t been able to get GPUs from Atlas recently, despite big announcements about bare metal datacenters from their crypto parent company and random appearances on endpoint provider lists with their managed inference offering.

SpaceXAI

We love Colossus and can’t wait to test the SpaceXAI neocloud offering!

Mistral

Mistral just raised a €3B Series D, and has yet to confirm the validity of a hacker from TeamPCP claiming to put its entire codebase up for sale on the dark web. Anyway, we are very excited to test their neocloud offering. The team told us the platform was still coming up and onboarding its first customers in August, so we very much look forward to testing with them in the future. It seems obvious to us that companies with experience building training clusters for their own models (see above) will be successful neoclouds if they choose to pivot.

Poolside Infrastructure Company

Following a $6B licensiquihire by Nvidia that took an incredible group and buried them in the Nemotron bureaucracy, Poolside’s legacy lives on through the Poolside Infrastructure Company, which is developing their Project Horizon campus in West Texas. We have not yet tested anything from them, but their experience and ambition makes them an immediate contender if the skeleton crew decides to go for it on the neocloud business.

Nscale

Nscale announced a $2B Series C in March, announced the acquisition of AnyScale in May, and just filed an S1 to go public on NYSE a few days ago. It is unbelievable to us that they can have such a big business, and continue to claim in public marketing materials that they offer managed clusters and inference endpoints, when this is clearly not true. Obviously, though, they’ll continue to be successful. They’ve got access to lots of land and power, some construction and datacenter operations experience, and people really want that bare metal at a nice price.

Highrise

Highrise, Hut 8’s AI Cloud business, is constantly in the news. We hope we can test some managed clusters from them soon.

Corvex

Corvex continues to run scared from ClusterMAX, though they announced a $33M private placement in August to expand their datacenter capacity. Their choice of merger partner was Movano, maker of the Evie smart ring. Quite a change in product roadmap. Privately, they claim some secure government customers in the USG. We look forward to testing.

Andromeda

Andromeda takes capacity from other providers and puts a managed cluster service on top. We have discussed testing multiple times, and enjoyed working with the technical team, but find the business side to be a little shady. Resellers have a perverse incentive to keep the customer in the dark about who the underlying provider is, out of fear that the customer can just go around them and contract with them directly. Unfortunately, this can lead to bad games of broken telephone, and lots of finger pointing in the worst case. Specifically, we have a problem with non-circumvention clauses and hope that Andromeda can build a durable business model that allows them to be honest about their solid technical team and the value it can provide.

Specifically, we are aware of three different options from Andromeda:

1. Rent compute - typical 1:1 neocloud relationship, where Andromeda aggregates capacity from underlying providers and resells it, but depends on them to deliver support

2. Manage compute - some providers have a growing list of 3 or 4 different providers and want one throat to choke, so Andromeda becomes that throat

3. Bring your own cluster - also called private cloud, this is when customers want to own their own hardware (and control their own supply chain) but still need help managing it all, effectively SRE-for-hire

We are seeing big growth in Option #3 on that list. And we look forward to testing some of Andromeda’s Blackwell capacity soon, wherever it may be.

Volta

Volta launched with $300M in venture funding and a separate $5B financing pool for customers. Its first large announced contract uses Bitdeer’s Norway site. We have not tested the cloud software or customer operations.

Firebird

Firebird is building GPU infrastructure in Armenia, with $60M in construction financing announced by Ameriabank, and expansion plans in Kazakhstan. Recently, they became the subject of a story about Nvidia chip access and the Armenia-Azerbaijan peace process; co-founder Razmig Hovaghimian responded that the project had roots going back years before those diplomatic developments. GPU diplomacy. Pressure’s on to deliver for the people of Armenia! We still need to see the managed cloud in action.

Tatra

Tatra is developing B300 and GB300 capacity in Slovakia, using the country’s nuclear and hydro power base. We have some great feedback about the technical chops of the father/son team—hard working, impressive family. For now, though, we need them to get their first tranche fully deployed so we can put them to the test.

Sesterce

We spun up an on-demand H100 through Sesterce in August. The console showed an out-of-date Ubuntu option, provisioning took its time, and eventually we got a working machine. Then we found Shadeform underneath. A broker on top of a broker. We would quite like to know how many companies need a margin before the customer gets to run a workload! With that said, Sesterce is pursuing some big bare metal Blackwell clusters with big offtakers. We want to test those.

Groq

Groq set out to challenge Nvidia, but after a licensiquihire took Jonathan Ross and most of the engineering team, the remaining company has pivoted… to renting out Nvidia GPUs! Becoming a neocloud is the move if you want another $350M to live out your dreams (as announced in August). We look forward to testing more than just endpoints one day. Jensen gets the chip team and another customer. Not bad.

Yotta

The four-node Yotta H100 Kubernetes cluster we tested had many problems. Within a week, there was genuine XID 94 on one node and a bad rail on another. Yotta’s bring-up process appears brittle, as our small cluster had important discrepancies from node to node. It also had many out-of-date packages. One of our nodes failed to come back online after we rebooted it. Yotta’s health checks did not intervene when we manually reset the secondary PCIe bus to trigger an XID 79, leaving us with one GPU unreachable until manual intervention, although the Kubernetes layer did nothing to distinguish the shorthanded node.

Yotta must successfully deploy Blackwell GPUs and improve its reliability to keep pace with the industry.

Darya

Darya is bringing GPU cloud infrastructure to Tajikistan, right on the border of Afghanistan, which certainly expands the map of places we need to test. Its H200 cluster launched in June 2025, and it subsequently signed an agreement with Yotta to develop a hydropower-fed AI datacenter in Darvoz. Interesting stuff, but after our experience navigating Yotta’s portals, we are particularly curious which parts of the operating model make the trip to Tajikistan. We have not yet had access to Darya’s cluster, though we would love to visit the site.

Boostrun

Boostrun has some solid bare metal, but bought its way past some of the Kubernetes engineering: vCluster says the company launched managed Kubernetes in under 45 days with no new platform engineering hires. They went with isolated tenant control planes, dedicated private nodes, and Netris for network provisioning. Sensible. We would rather see a small provider use working software than spend a year rediscovering Kubernetes. In customer discussions, we have still considered adding another operator for a fully managed training experience. We have tested a Boostrun cluster indirectly, through another provider reselling their capacity, but await the opportunity to engage with their support teams directly. We want to see how much of the day two operations Boostrun handles for themselves in a real engagement before moving them onto the list.

Global AI

Global AI takes the sovereign-cloud pitch quite literally: dedicated, single-tenant, air-gapped clusters sold by the data hall (or so they say). Customers decide when Global AI can access their systems, a solid offering for bare metal. However, when advertising an option for managed clusters, we need to see more. Just to assess bare metal we want to see things like how provisioning and security patches get handled, how comprehensive the monitoring stack is, how reliability and health checks work, and generally how competent the onsite team is at closing tickets quickly and correctly. We have not yet gotten to see this level of detail.

Argentum

Argentum’s website says “200,000+ GPUs available now.” Had you heard of Argentum before you got to this section of the article?

QumulusAI

QumulusAI found a different source of GPU debt: a $500M non-recourse facility through USD.AI, using GPU Warehouse Receipt Tokens as collateral for stablecoin borrowing, with financing for up to 70% of approved deployments. GPU Warehouse Receipt Tokens is the actual name. The announcement describes a facility, so we are not counting $500M of installed hardware. In our customer discussions, Qumulus has come up as a capacity supplier that may need another operator on top for managed training. We still need to inspect its own software and support, but initial customer feedback has been surprisingly solid.

Alibaba Cloud

Alibaba has considerably more software to show us than the usual neocloud startup. One example is its ACK Slurm operator: a SlurmCopilot component coordinates resource allocations between Slurm and Kubernetes so idle resources do not remain stranded in one scheduler. This is the kind of orchestration plumbing we like to poke at, and obviously this is a company that is not ready for a poking. We have an account, have been whitelisted for a few model endpoints, and have engaged directly with the sales team, but have yet to get access to any modern GPUs or this ACK software for testing.

Megaspeed

Plenty of hardware to investigate in their Malaysian and Indonesian sites, but no login yet.

BytePlus

BytePlus’s GPU service advertises switch-affinity placement on a multi-rail network, direct RDMA access to its vePFS parallel filesystem, and integration with our favourite model ever: Seedance 2.5. We have not yet completed an evaluation of their clusters, but there is clearly stuff to watch out for. Tik, tok.

Humain

Humain has given itself plenty to do. Alongside the Saudi GPU buildout, Tareq Amin announced a roughly 6 GW datacenter ambition, as well as Humain One, a real-time voice interface for any computer. We would be happy to start with a login and a functioning cluster, but we have yet to get a response from the team, leaving us wondering what secrets they’re hiding next to their Groq systems.

SK Telecom

SK Telecom runs over 1,000 B200s for Korea’s sovereign foundation-model initiative. As we covered in a recent article, the government also rented ~3,000 H100 equivalents from SK Telecom and Naver combined for the competition’s first round. SK Telecom has announced a much larger 2 GW NVIDIA DSX AI factory, expected to deploy Vera Rubin systems with SK Hynix HBM4, within SK Group’s planned 5 GW first-phase buildout. We have tested SK Telecom GPUs through Vessl, and they were solid, but nothing with SK directly. We need to see more from them on the international market to consider them a serious player, but the technical foundation is clearly there.

Naver

Naver, much like SK Telecom, is highly credible. It already operates hyperscaler-class datacenters, and its July plan with NVIDIA and Brookfield calls for expanding the GAK Sejong AI factory from an initial 55 MW buildout to 200 MW by 2028. We are interested in how much of that operational experience reaches external customers, vs the internal research teams at Naver. We have yet to test that experience.

Indosat (Zankore)

Zankore has announced the first phase of roughly 200 MW of GB300 NVL72 capacity is to be delivered in the first half of 2027. Even if they are a little behind schedule on deploying Blackwell, they are still well on their way to a long-term 1 GW ambition, primarily to serve offtakers (rumoured to be) from mainland China. And they’ve landed a $3.1B loan to go do it!

Where ClusterMAX Goes From Here

Read more