Design and Evaluation of new Deep Learning based approaches for classification of Scientific Literature

Abstract

In this thesis, we address the challenge of efficient and accurate text clas- sification within the domain of scientific literature, where the sheer volume of published works demands robust and scalable solutions. Traditional ma- chine learning algorithms such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) often struggle with modelling long- term dependencies, detecting local phrase features, and adapting to variable length inputs, limiting their effectiveness for this task. Recent advancements in transformer-based models have significantly improved natural language processing capabilities, yet their parameter inefficiency remains a notable drawback. Weproposeaparameter-efficienttransformer-basedmodel,sBERT(Small BERT), tailored specifically for scientific literature classification. sBERT aimstooptimizememoryusage,trainingtime,andinferencetimewhilemain- taininghighaccuracyandreducingtheenvironmentalimpactassociatedwith large-scale language models. Our comprehensive experiments, conducted on datasets from Web of Science, ArXiv, Nature, Springer, and Wiley, demon- strate that sBERT achieves a favorable balance between performance and efficiency. The findings suggest that sBERT is a viable alternative to more resource-intensive models, offering a sustainable and effective solution for text classification in scientific domains. This research contributes to the field by presenting a scalable, efficient approach to handling the growing volume of scientific publications, thereby facilitating better information retrieval and knowledge management in academic and research settings. newline