Comparing Deep Feature Representations through Singular Value Decomposition

DOI: https://doi.org/10.62517/jbdc.202601403

Author(s)

Qiangqiang Li*

Affiliation(s)

Division of Information Engineering, Shizhen College of Guizhou University of Traditional Chinese Medicine, Guiyang, Guizhou, China *Corresponding Author

Abstract

Singular value decomposition (SVD) was employed to examine and compare the intermediate representations produced by a multilayer perceptron (MLP), a convolutional neural network (CNN), a residual network (ResNet), and a vision transformer. All four models were trained from random initialization on MNIST using the same optimization protocol. Activation matrices were constructed in a manner suited to each architecture and analyzed in terms of effective rank, cumulative singular-value energy, condition number, and leading singular vectors. Although the models achieved test accuracies within one percentage point of one another, their spectral characteristics differed substantially. The MLP concentrated most of its activation energy along a small number of directions, whereas the CNN and ResNet distributed their representations across broader subspaces. The ResNet also showed the most stable condition-number profile across layers. The transformer occupied an intermediate position between the MLP and the convolution-based models in terms of spectral breadth, while its effective rank exhibited the greatest sensitivity to additive noise. These findings indicate that activation spectra can expose redundancy, anisotropy, and sensitivity to perturbations that are not apparent from accuracy alone. Since activation matrices must be defined differently for different architectures and the experiments are limited to a single, relatively simple dataset, the reported measures should be viewed as descriptive diagnostic tools rather than as a general ranking of representation quality.

Keywords

Singular Value Decomposition; Feature Representation; Effective Rank; Convolutional Neural Network; Residual Network; Vision Transformer; Interpretability

References

[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Advances in Neural Information Processing Systems, vol. 25, 2012. [2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016, doi: 10.1109/CVPR.2016.90. [3] A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2021. [4] K. Han, Y. Wang, H. Chen, et al., "A survey on vision transformer," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 87-110, 2023, doi: 10.1109/TPAMI.2022.3152247. [5] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, "Transformers in vision: A survey," ACM Computing Surveys, vol. 54, no. 10s, Art. 200, 2022, doi: 10.1145/3505244. [6] Q. Zhang, Y. Xu, J. Zhang, and D. Tao, "ViTAEv2: Vision transformer advanced by exploring inductive bias for image recognition and beyond," International Journal of Computer Vision, vol. 131, pp. 1141-1162, 2023, doi: 10.1007/s11263-022-01739-w. [7] J. Chen, P. Wu, X. Zhang, R. Xu, and J. Liang, "Add-Vit: CNN-transformer hybrid architecture for small data paradigm processing," Neural Processing Letters, vol. 56, Art. 198, 2024, doi: 10.1007/s11063-024-11643-8. [8] W. Ding, M. Abdel-Basset, H. Hawash, and A. M. Ali, "Explainability of artificial intelligence methods, applications and challenges: A comprehensive survey," Information Sciences, vol. 615, pp. 238-292, 2022, doi: 10.1016/j.ins.2022.10.013. [9] R. H. Riedi, R. Balestriero, and R. G. Baraniuk, "Singular value perturbation and deep network optimization," Constructive Approximation, vol. 57, pp. 807-852, 2023, doi: 10.1007/s00365-022-09601-5. [10]Y. Xie and N. Thuerey, "Reviving autoencoder pretraining," Neural Computing and Applications, vol. 35, no. 6, pp. 4587-4619, 2023, doi: 10.1007/s00521-022-07892-0. [11]J. Kim, W. Lee, M. Eo, and W. Rhee, "Improving forward compatibility in class incremental learning by increasing representation rank and feature richness," Neural Networks, vol. 183, Art. 106969, 2025, doi: 10.1016/j.neunet.2024.106969. [12]D. Murfet, S. Wei, M. Gong, H. Li, J. Gell-Redman, and T. Quella, "Deep learning is singular, and that is good," IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 12, pp. 10473-10486, 2023, doi: 10.1109/TNNLS.2022.3167409. [13]W. Jia, M. Sun, J. Lian, and S. Hou, "Feature dimensionality reduction: A review," Complex and Intelligent Systems, vol. 8, pp. 2663-2693, 2022, doi: 10.1007/s40747-021-00637-x. [14]H. Choi, S. Jin, and K. Han, "ICEv2: Interpretability, comprehensiveness, and explainability in vision transformer," International Journal of Computer Vision, vol. 133, pp. 2487-2504, 2025, doi: 10.1007/s11263-024-02290-6.