Machine Learning-Based DDoS Attacks Detection and Mitigation in O-RAN Enabled 5G Networks

Abstract

The integration of O-RAN into 5G networks marks a major paradigm shift in mobile communications, introducing openness, programmability, and AI-driven control through centralized RAN Intelligent Controllers (RIC). Inspired by Software-Defined Networking (SDN), O-RAN’s core innovation is the decoupling of control logic from the data plane, enabling dynamic and intelligent network management. However, these advancements also come with significant security challenges, particularly the increasing threat of Distributed Denial of Service (DDoS) attacks. To address these vulnerabilities, this project introduces an intelligent framework for real time DDoS detection and mitigation within a next generation Open RAN (O-RAN) 5G network. At the core of our solution is a fully containerized and slice aware 5G Simulated Network, built using OpenAirInterface (OAI) and orchestrated through the FlexRIC controller. This architecture enables dynamic and programmable network management, offering precise control over user connectivity and network slicing at the RAN level. To enhance detection capabilities, we designed and evaluated a suite of machine learning models. These include Random Forest, Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory (BiLSTM), and an ensemble approach. Among them, the Random Forest model demonstrated the best performance, achieving 99.0% accuracy during training and 90% accuracy under real time traffic simulations, all while maintaining low computational overhead. This detection system was integrated into the User Plane Function (UPF), allowing real time traffic analysis and anomaly detection. When malicious activity is identified, a dedicated xApp deployed in the Near-RT RIC via FlexRIC triggers an RRC Release command, immediately disconnecting the compromised User Equipment (UE). Experimental results validate the effectiveness of our system. It achieves accurate, real time DDoS detection with a low false positive rate and consistently low latency, even under attack conditions, while maintaining high uplink throughput. This demonstrates the framework’s ability to preserve performance for legitimate users. Overall, this work presents a practical and scalable architecture that significantly strengthens the security and resilience of 5G networks against emerging cyber threats.