REFINE: Robust Evaluation Framework for IDS under Concept Drift in Dynamic Environments

Abstract

Most machine learning-based Intrusion Detection Systems (IDSs) are designed for stationary environments, where data distributions is assumed to remain constant over time. However, modern network environments are dynamic, and this can lead to significant changes in the observed environment since the training phase, causing degradation in IDS performance. Consequently, increasing attention has been given to online learning techniques designed to address such phenomenon, known as concept drift. Designing such adaptive systems is a far from trivial task, due to a multitude of factors, such as experimental biases as well as the lack of real-world labeled datasets with precise drift annotations. Moreover, the evaluation of such systems still lacks a standardized methodology, and critical aspects are often inconsistently addressed, making comparisons between approaches particularly difficult. To address these challenges, this work proposes REFINE, a Robust Evaluation Framework for IDS under concept drift in dynamic environments. REFINE combines a Concept Drift Stream Generator (CDSG), which produces realistic datasets from real network traffic with controlled drift characteristics, and a robust online evaluation pipeline that mitigates experimental biases. Results demonstrate that REFINE enables accurate, unbiased evaluation and comparison of online IDSs, providing critical insights into their adaptation and detection capabilities across various drift scenarios.