AI‑Driven Big Data Analytics for CRM Document Classification and Domain Prediction

Abstract

This system looks into how Big Data and AI can be used together to improve Customer Relationship Management (CRM) document analysis and figure out how big text datasets can be used to predict domain relationships. The main dataset is the Web of Science (WoS) database, which has 840 domain-specific documents from different release years. To begin, descriptive methods are used to clean the data by removing stop words, network-based visualization is done through word clouds, and contextual analysis is done using Stochastic Neighbor Embedding (SNE) clustering and Non-negative Matrix Factorization (NMF) topic modeling to find the main themes. Text that has been processed is turned into number vectors, which lets Apache Spark do processing that is spread out and quick. Support Vector Machine (SVM) and Decision Tree are two machine learning methods that are used for domain classification. SVM gets 70% accuracy and Decision Tree gets 94% accuracy. Principal Component Analysis (PCA) is used to narrow down the list of traits from 100 to 60 important ones in order to improve the accuracy of the predictions. When you retrain the Decision Tree with features chosen by PCA, you get a 100% classification accuracy, which is better than the first setups. We tested and implemented the whole process using Jupyter Notebook and a Flask-based web application that lets users easily upload documents and get predicted domains in real time. This shows how effective AI-driven CRM analytics can be when combined with Big Data processing.