LLM-jp-4 技术报告：8B 稠密与 32B-A3B MoE 日语主权大模型

LLM-jp-4 Technical Report

AI 导读

LLM-jp-4 是一系列日语主权大语言模型，包含 8B 稠密模型和 32B-A3B MoE 模型，两者均从零开始训练，使用约 12 万亿 tokens。团队为训练数据、外部模型和软件的选择制定了明确标准并公开开发流程，以支持独立验证与复现。评测显示其较前代 LLM-jp 模型大幅提升，与同规模开放权重模型相比具备竞争力；33B 稠密模型结果未包含在本版本中。

正文 · 原文

Authors:LLM-jp: Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana, Hirokazu Kiyomaru, Jun Suzuki, Koshi Eguchi, Kouta Nakayama, Qianying Liu, Rio Yokota, Ryo Takahashi, Sadao Kurohashi, Sosuke Hosokawa, Taishi Nakamura, Takashi Kodama, Yun-Ang Wu, Yusuke Miyao, Yusuke Oda

View PDF HTML (experimental)

Abstract:This paper presents LLM-jp-4, a family of Japanese sovereign large language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-jp-4 comprises an 8B dense model and a 32B-A3B mixture-of-experts model, both trained from scratch on approximately 12 trillion tokens. To support independent verification and reproduction, we establish explicit criteria for selecting training data, external models, and software, and make the development process publicly available. Evaluation results demonstrate substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale. By documenting the development principles, training procedures, and empirical findings of LLM-jp-4, we aim to provide a reference for future sovereign AI development.

Comments:	Results for the LLM-jp-4 33B dense model are not included in this version and will be added in a future revision
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.19331 [cs.CL]
	(or arXiv:2610.19331v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.19331 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Hirokazu Kiyomaru [view email]
[v1] Wed, 14 Oct 2026 00:16:21 UTC (986 KB)

来源：arXiv cs.CL · arxiv.org