Neural networks employ a multitude of parameters, particularly weights, to map input to output, which can be learned from available data. This research investigates the distribution and relationships among weights in neural networks by utilizing univariate and multivariate $\alpha$-stable distributions. Our findings indicate that over 80% of layers in pretrained neural networks we used for analysis deviate from the assumption of a Gaussian distribution and exhibit heavy-tailed behavior. Approximately, 10% to 35% of the layer weights are asymmetric. Furthermore, we demonstrate that there exists a higher degree of interdependence among the channels and kernels of the initial network layer. These results provide a foundation for weight initialization, Bayesian neural network prior selection, and model compression, among others.

The fundamental use of neural networks is in providing a nonlinear mapping between input and output data with possibly a high number of parameters that can be learned from data directly. Consequently, studying the model's parameters, particularly the weights, is of paramount importance. The distribution and interdependencies of these weights have a direct impact on the model's generalizability, compressibility, initialization, and convergence speed. By fitting the weights of pretrained neural networks using the $\alpha$-stable distributions and conducting statistical tests, we discover widespread heavy-tailed phenomena in neural network weights, with a few layers exhibiting asymmetry. Additionally, we employ a multivariate $\alpha$-stable distribution to model the weights and explore the relationship between weights within and across layers by calculating the signed symmetric covariation coefficient. The results reveal a strong dependence among certain weights. Our findings indicate th...