Consider the problem of understanding the behavior of a system of interest and predicting its future behavior. In many cases, we assume that the underlying mechanisms are complex or unknown, and difficult to deductively understand from first principles. In such situations, statistical learning provides a basic framework for addressing this problem by fitting a postulated model to the observed data. Because the size of the available data is finite, obtaining a model that completely describes the behavior of the system is intrinsically impossible. Hence, when a specific model is fitted using finite data, its quality and reliability need to be assessed by quantifying prediction errors and uncertainties.

Initially, this may appear to be a problem in information science and unrelated to physics. However, Somplinsky et al. (1990) noted that, if one regards the parameters of the fitted model as dynamical degrees of freedom, goodness of fit as energy, and observed data as impurities, the problem can be viewed as one of the statistical physics of disordered systems. Subsequently, theoretical studies of statistical learning have become standard research topics in statistical physics.

Despite this progress, most statistical-physics-based studies rely on approaches that derive exact solutions using simplified setups. In physics, this is analogous to solvable mean-field models, such as the infinite-range Ising model. While such approaches are useful for gaining insight into fundamental mechanisms, predicting behavior quantitatively in practical data analysis with real data is difficult because exact solutions are not always available. As a result, their connection to conventional mathematical statistics and learning theory often remains unclear.

In this study, we address this problem by combining the replica method of the statistical physics of disordered systems with a grand canonical ensemble and variational approach, focusing on analyzing learning parametric models. By introducing a grand canonical ensemble in which the number of data points, rather than the number of particles, is allowed to fluctuate, we can derive a simple formula for the prediction error, and the variational approximation can be performed in a straightforward manner. Consequently, asymptotic expressions for prediction errors, known as information criteria in mathematical statistics, were rederived within a statistical physics framework. Furthermore, for linear regression, we demonstrated that the prediction error can be quantitatively predicted, even in a nonasymptotic, high-dimensional regime, using real data.

The problem addressed in this study is still elementary, and a full assessment of the practical usefulness of the proposed method is left for future work. Nevertheless, the present results demonstrate new possibilities for statistical-mechanical approaches for analyzing statistical learning.

(Written by Takashi Takahashi on behalf of all the authors)

Share this topic

Fields