An Optimized Transformer Ensemble Model for Context-Aware Speech Act Classification

Abstract

The rapid growth of digital communication has increased the need for understanding user intent in textual data, leading to the development of speech act classification systems. Historically, traditional approaches relied on rule-based methods and basic machine learning techniques, which struggled to capture the complexity of natural language. The main problem lies in accurately classifying unstructured and context-dependent text into meaningful categories such as assertive, directive, expressive, and question. Conventional systems often depend on manual feature extraction techniques like bag-of-words and TF-IDF, which fail to capture semantic relationships and contextual dependencies. These limitations result in lower accuracy, poor generalization, and difficulty in handling imbalanced datasets. To address these challenges, there is a need for a more robust and intelligent system that can effectively process textual data and improve classification performance. This research proposes a transformer-driven approach that utilizes eXtreme Language Network (XLNet) for feature extraction, combined with multiple machine learning models including Logistic Regression (LR), Random Forest (RF), Support Vector Machine (SVM), and a Boosting Fusion Model (BFM) integrating Light Gradient Boosting Machine (LGBM) and Categorical Boosting (CB). The system incorporates Natural Language Preprocessing (NLP) preprocessing techniques and Synthetic Minority Oversampling Technique (SMOTE) based data balancing to enhance learning efficiency. The proposed system demonstrates significant improvements in accuracy and reliability, with the BFM achieving an accuracy of 96.33%, outperforming LR, RF, and SVM.