Difficult to Be Partial to Stability or Speed in Traditional SGD: How Can SGD Be Modified to Balance Stability and Speed in Image Classification?
DOI: https://doi.org/10.62517/jbdc.202601405
Author(s)
Guo Chen
Affiliation(s)
Computer Science and technology, Beijing Jiao Tong University, Beijing, China
Abstract
In deep learning, training generative models generally requires large-scale datasets. The stochastic gradient descent (SGD) algorithm is widely adopted for optimizing such models. However, traditional SGD method have several limitations. For example, the unavoidable noise in large datasets can lead to unstable parameter updates and poor gradient convergence. Various modified SGD algorithms have been proposed in order to address these issues. This paper compares several representative variants of SGD from the perspectives of stability and convergence speed, and analyses their applicability in image classification tasks. The paper aims to introduce a comprehensive overview of the balance involved and to guide the selection of appropriate optimizers in practice.
Keywords
Deep Learning; Stochastic Gradient Descent; Gradient Convergence; Image Classification
References
[1] Fucheng, D., Wanjie, W., Ao, G., Xiaoqi, W., & Fan, W. (2025). Gradient Descent Algorithm Survey. https://doi.org/10.48550/arxiv.2511.20725
[2] Ziyin, L., Liu, K., Mori, T., & Ueda, M. (2021). Strength of Minibatch Noise in SGD. https://doi.org/10.48550/arxiv.2102.05375
[3] Ziyin, L., Li, B., Galanti, T., & Ueda, M. (2023). Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent. https://doi.org/10.48550/arxiv.2303.13093
[4] Block, A., Foster, D. J., Krishnamurthy, A., Simchowitz, M., & Zhang, C. (2023). Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression. https://doi.org/10.48550/arxiv.2310.11428
[5] Yao, Z., Yu, R., Chang, G., Li, Y., Zhang, Y., & Li, D. (2023). Signal Processing Meets SGD: From Momentum to Filter. https://doi.org/10.48550/arxiv.2311.02818
[6] Hermant, J., Renaud, M., Aujol, J.-F., Dossal, C., & Rondepierre, A. (2026). Gradient correlation is a key ingredient to accelerate SGD with momentum. https://doi.org/10.48550/arxiv.2410.07870
[7] Do, T., Hannibal, S., & Jentzen, A. (2024). Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail to converge to global minimizers in the training of deep neural networks with ReLU activation. https://doi.org/10.48550/arxiv.2410.10533
[8] Chen, Y., Wang, Y., & Sundaram, H. (2025). DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation. https://doi.org/10.48550/arxiv.2509.16173
[9] Nagwekar, A. (2025). Towards Guided Descent: Optimization Algorithms for Training Neural Networks At Scale. https://doi.org/10.48550/arxiv.2512.18373
[10] Yeo, S., Bae, M., Jeong, M., Kwon, O., & Oh, S. (2023). Crossover‐SGD: A gossip‐based communication in distributed deep learning for alleviating large mini‐batch problem and enhancing scalability. Concurrency and Computation, 35(15), Article e7508. https://doi.org/10.1002/cpe.7508
[11] Liu, E., Chu, Z., & Zhang, X. (2025). Wasserstein GAN for moving differential privacy protection. Scientific Reports, 15(1), Article 19634. https://doi.org/10.1038/s41598-025-03178-2
[12] Zhang, Z., Hu, Y., & Ye, Q. (2022). LR-SGD: Layer-based Random SGD For Distributed Deep Learning. Proceedings of the 8th International Conference on Computing and Artificial Intelligence, 6–11. https://doi.org/10.1145/3532213.3532215
[13] Quercia, A., Nader, F., Morrison, A., Scharr, H., & Assent, I. (2025). Focal Sampling: SGD biased towards early important samples for efficient image classification with augmentation selection: Focal Sampling: SGD biased towards early important samples. Knowledge and Information Systems, 67(11), 11161–11191. https://doi.org/10.1007/s10115-025-02563-7
