arXiv cs.AI· Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, Jure Leskovec·· 3 天前AI 评分30
RT-J：面向上下文高效预测的关系型 Transformer 大规模预训练
RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions
AI 导读
研究团队推出 85M 参数的关系型基础模型 RT-J，仅需数百个甚至更少的上下文示例即可做出准确预测。该模型基于 639 个网页抓取数据库、6,023 个 LLM 生成预测任务构建的 Join 数据集预训练，支持最长 8,192 个数据库单元格的上下文，并采用随机游走检索器、上下文调优与集成扩展测试时计算。
正文 · 原文
Abstract:Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.
| Comments: | Accepted to NeurIPS 2026. Website: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Databases (cs.DB) |
| Cite as: | arXiv:2610.17867 [cs.LG] |
| (or arXiv:2610.17867v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17867 arXiv-issued DOI via DataCite |
Submission history
From: Rishabh Ranjan [view email]
[v1]
Tue, 13 Oct 2026 07:27:44 UTC (1,023 KB)
[v2]
Wed, 14 Oct 2026 04:27:19 UTC (1,029 KB)
来源：arXiv cs.AI · arxiv.org