arXiv cs.AI· Maryam Arshad, Muhammad Khuram Shahzad·· 1 天前AI 评分20
CAFIN：基于 CUDA 内核融合加速 AdaFisher 二阶优化器
CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion
AI 导读
CAFIN 是一个 GPU 原生 CUDA 内核融合框架，在不改变 AdaFisher 优化公式的前提下提升其执行效率。在 GTX 1660 Ti 和 RTX 3060 Laptop 上，FIM 构建延迟最多降低 78.4%，优化器单步延迟降低 2.1-26.9%，每步减少约 18.91 MB 中间显存分配。数值验证在 FP32 容差内保持一致，收敛精度可比。
正文 · 原文
Abstract:Second-order optimizers improve deep neural network training using curvature information, but their practical efficiency is limited by GPU execution overhead. AdaFisher reduces Fisher Information Matrix (FIM) approximation costs through diagonal Kronecker structures and exponential moving averages, yet its PyTorch implementation requires sequential CUDA kernel launches, intermediate tensor allocations, and repeated memory accesses. We introduce CAFIN, a GPU-native CUDA kernel fusion framework that improves AdaFisher execution efficiency without changing its optimization formulation. The proposed approach progressively integrates FIM preconditioner construction, exponential moving average updates, gradient preconditioning, momentum accumulation, weight decay, and parameter updates into fused CUDA kernels. Shared-memory reuse, register-resident computation, and in-place operations further reduce memory traffic and intermediate allocations. Experiments using ResNet-18 on CIFAR-10, CIFAR-100, and Tiny ImageNet across NVIDIA GTX 1660 Ti and RTX 3060 Laptop GPUs demonstrate up to 78.4% lower isolated FIM construction latency and 2.1-26.9% lower optimizer-step latency. The implementation also eliminates approximately 18.91 MB of intermediate GPU memory allocations per optimizer step. Numerical validation confirms consistency within the predefined FP32 tolerance, while convergence accuracy remains comparable across evaluated datasets and random seeds. These findings demonstrate that CUDA kernel fusion can substantially accelerate second-order optimization while preserving AdaFisher's underlying mathematical formulation and learning behavior.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14589 [cs.LG] |
| (or arXiv:2611.14589v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14589 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Muhammad Khuram Shahzad [view email]
[v1]
Tue, 10 Nov 2026 07:45:44 UTC (5,616 KB)
来源：arXiv cs.AI · arxiv.org