arXiv cs.AI· Aaron J. Gutknecht, Fernando E. Rosas, David A. Ehrlich, Abdullah Makkeh, Pedro A. M. Mediano, Michael Wibral·· 1 天前AI 评分26
Shannon observables：可扩展的信息分解新框架
Shannon observables: A scalable approach to information decomposition
AI 导读
研究者提出“Shannon observables”框架，这类量能刻画高阶信息处理的关键性质，却可直接由 Shannon 熵计算并高效扩展到大系统，从而消解多变量信息论指标的长期解释歧义。该框架已用于分析纠错码在不同信道噪声下的编码区间、深度分类器与卷积自编码器逐层及训练过程中的信息处理特征，以及人脑 MEG 记录，并生成揭示定向信息传递区域结构的皮层图谱。
正文 · 原文
Abstract:Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order structure of the information within the system with distinct properties across scales. Investigating how these systems process information remains challenging due to difficulties in defining appropriate multivariate metrics and ensuring their scalability to large systems. To address these challenges, we introduce a novel framework based on what we call "Shannon observables": quantities that capture essential properties of high-order information processing, yet can be computed directly from Shannon entropies and scale efficiently to large systems. Our theoretical results demonstrate how Shannon observables can be used to resolve long-standing ambiguities regarding the interpretation of widely used multivariate information-theoretic measures. On the practical side, we demonstrate the framework's versatility across several domains. An analytical study of error-correcting codes shows how Shannon observables trace out distinct encoding regimes as channel noise varies; analyses of deep classifiers and convolutional autoencoders reveal characteristic information-processing signatures across layers and over training; and applied to magnetoencephalography recordings from the human brain, the framework yields a cortical atlas exposing pronounced regional structure in directed information transfer. Overall, our framework resolves fundamental limitations in analysing high-order information phenomena and offers broad opportunities for theoretical developments and empirical analyses.
| Comments: | 28 pages, 8 figures, 3 tables. v2: substantially revised and retitled (formerly "Shannon invariants"); adds an analysis of error-correcting codes, an extension to directed information flow with a cortical atlas from MEG recordings, single-neuron lesion and dropout experiments, and bounds linking vulnerability to performance loss after neuron removal |
| Subjects: | Information Theory (cs.IT); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Adaptation and Self-Organizing Systems (nlin.AO); Data Analysis, Statistics and Probability (physics.data-an) |
| Cite as: | arXiv:2504.15779 [cs.IT] |
| (or arXiv:2504.15779v2 [cs.IT] for this version) | |
| https://doi.org/10.48550/arXiv.2504.15779 arXiv-issued DOI via DataCite |
Submission history
From: Aaron Julian Gutknecht [view email]
[v1]
Tue, 22 Apr 2025 10:41:38 UTC (532 KB)
[v2]
Wed, 11 Nov 2026 12:35:42 UTC (1,644 KB)
来源：arXiv cs.AI · arxiv.org