A CLOUD-NATIVE APPROACH TO STATISTICAL ANOMALY DETECTION AND AUTOMATED DATA QUALITY VALIDATION

Abstract

The research proposes a statistical anomaly detection cloud architecture and automatic data quality validation with multi-tenant cloud resource utilization data. A model, based on utilization of XGBoost, is utilized to determine hidden overutilization of the resources with high accuracy and with appropriate support of precision-recall analysis and confusion-matrix analysis. Pre-model data quality checks are automated to ensure that there is an audit trail of reliability. SHAP-based explainability leads to better transparency and governance, which proves the efficiency of the framework at the scale of reliable cloud monitoring and data quality assurance.