Hugging Face 增量上传：239GB 数据集仅传输 2.6MB
Hugging Face 增量上传省下 239GB 流量
microagi 向 Hugging Face 重新上传一个 239 GB 数据集，仅小幅修改，实际只有 2.6 MB 走网络，约一分钟完成。增量上传在规模化时持续见效，因为机器人录制数据高度重复；配合 Cloudflare Workers 分发到任意云，超过 1 PB 成本约 500 美元。
Thomas Wolf· @Thom_Wolf · X·2026-10-26 23:57· 47 分钟前AI 评分47
AI 导读
来自 @microagi 关于传输 PB 级机器人数据的文章里有个有趣的数据： 他们向 Hugging Face 重新上传了一个 239 GB 的数据集，只做了小幅修改。结果只有 2.6 MB 走了网络，大约一分钟就完成了（！） 而且这在规模化时持续见效，因为机器人录制数据高度重复（相同的场景、相同的设备）。然后 @Cloudflare workers 把它分发到任何云，超过 1 PB 只需约 500 美元 团队写的这篇文章很棒：https://www.microagi.ai/articles/delivery-platform
正文 · 原文
fun stat from @microagi’s write-up on shipping petabytes of robot data:
they re-uploaded a 239 GB dataset with small changes to Hugging Face. only 2.6 MB went over the network, done in about a minute (!)
and it keeps paying off at scale, because robot recordings are super repetitive (same scenes, same rigs). then @Cloudflare workers fan it all out to any cloud, >1 PB for ~$500
great write-up by the team: https://www.microagi.ai/articles/delivery-platform
As our customers scale to training robot models using millions of hours of our recordings, we needed to build a new kind of data infrastructure. Terabytes of data delivered in minutes, and the capacity to deliver over a Petabyte per day to each of our customers, to multiple customers at the same time. That’s more than 10 Gigabytes a second, delivered into any cloud, wherever our customers need it. Using our customer portal, customers select precisely the data they want and it starts arriving in minutes at their destination, including AWS, GCP, Cloudflare, Hugging Face and any cloud provider who can handle the ingest speed. We’ve written about how we worked with @huggingface and @Cloudflare to build this secure, scalable infrastructure. Link in comments.在 X 查看被引用的帖子
来源：Thomas Wolf · x.com
来源：RadarAI · aihot.news