arXiv cs.CL· Yingtong Dou, Shubham Jain, Junpeng Wang, Hadi Abdullah, Vineeth Rakesh Mohan, Yuzhong Chen, Liang Wang, Yiwei Cai·· 9 小时前AI 评分38
Visa Research 用 Llama 3.2 1B 微调出 VRLLM,表格分类超越树模型与生产深度学习模型
VRLLM: Turning LLMs into Powerful Tabular Classifiers
AI 导读
Visa Research 将预训练的 Llama 3.2 1B 在数十亿行标注表格数据上微调,得到 VRLLM,其表格分类表现超过多种表格分类器乃至其生产深度学习模型。论文公开了数据准备、训练配方、设计选择消融与部署验证,并在同一任务上评测前沿 LLM 的预测行为。可解释性实验显示 VRLLM 学到特征相关的预测关联,最早两层 FFN 促成线性可解码任务信息的出现。
正文 · 原文
Abstract:Large Language Models (LLMs) have shown strong performance on text, but whether pretrained LLMs are effective for large-scale tabular classification remains unclear, as many previous works demonstrate that LLMs' tabular data classification performance falls behind tree-based models or neural architectures designed for tabular data. At Visa Research, we managed to fine-tune a pretrained Llama 3.2 1B LLM on billions of rows of labeled tabular data and make it outperform a wide array of tabular classifiers and even our competitive production deep learning model. For ease of reference, we refer to the model as the Visa Research LLM (VRLLM). In this paper, we share how we achieved this exciting result, covering data preparation, training recipe, ablation studies of design choices and deployment validation. We also benchmark frontier LLMs on the same task and analyze their prediction behavior across model families and generations. Our interpretability experiments suggest that VRLLM learns feature-specific predictive associations, while component ablations indicate that the earliest two FFN layers contribute to the emergence of linearly decodable task information. Our paper provides an engineering recipe and a focused empirical case study on adapting compact LLMs to industrial tabular prediction tasks.
| Comments: | Technical Report. WIP |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.22308 [cs.LG] |
| (or arXiv:2611.22308v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22308 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingtong Dou [view email]
[v1]
Tue, 17 Nov 2026 07:06:25 UTC (883 KB)
来源:arXiv cs.CL · arxiv.org