Fine-Tuning distilBERT for Enhanced Sentiment Classification

DOI: https://doi.org/10.62517/jbdc.202401417

Author(s)

Sarah Ling

Affiliation(s)

Markville Secondary School, Ontario, L3P 7P5, Unionville, Canada

Abstract

This research examines the fine-tuning of the DistilBERT model for sentiment classification using the IMDB dataset of 50,000 movie reviews. Sentiment analysis is vital in natural language processing (NLP), providing insights into emotions and opinions within textual data. We compare the fine-tuned DistilBERT and LLaMA 3 models, focusing on their ability to classify reviews as positive or negative. Through few-shot training on the dataset, our findings reveal that while LLaMA 3 8B excels in capturing complex sentiments, DistilBERT-base-uncased offers a more efficient solution for simpler tasks. The results underscore the effectiveness of fine-tuning. This paper contributes to optimizing sentiment analysis models and suggests future research directions, including hybrid models and advanced training techniques for improved performance across diverse contexts.

Keywords

Sentiment Classification; Fine-Tuning; Natural Language Processing; Large Language Models; Text Classification; Machine Learning; Transformer Models

References

[1] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023). [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.).Association for Computational Linguistics, Minneapolis, Minnesota,4171–4186. https://doi.org/10.18653/v1/N19-1423 [3]Diefan Lin, Yi Wen, Weishi Wang, and Yan Su. 2024. Enhanced Sentiment Intensity Regression Through LoRA Fine-Tuning on Llama 3.IEEE Access 12 (2024), 108072–108087. https://doi.org/10.1109/ACCESS.2024.3438353 [4] Bing Liu. 2020. Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge university press. [5] Haochen Liu, Sai Krishna Rallabandi, Yijing Wu, Parag Pravin Dakle, and Preethi Raghavan. 2023. Self-training Strategies for Sentiment Analysis: An Empirical Study. arXiv preprint arXiv:2309.08777 (2023). [6] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (Eds.). Association for Computational Linguistics, Portland, Oregon, USA, 142–150. https://aclanthology.org/P11-1015Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2023. Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research. IEEE Transactions on Affective Computing 14, 1 (2023), 108–132. https://doi.org/10.1109/TAFFC.2020.3038167 [7] V Sanh. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). [8] Amira Samy Talaat. 2023. Sentiment analysis classification system using hybrid BERT models. Journal of Big Data 10, 1 (2023), 110. [9] A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems(2017). [10] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385 (2024).