Skip to main content
SpringerLink
Log in

A real-time adaptive network intrusion detection for streaming data: a hybrid approach

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This study aimed at improving the performance of classifiers when trained to identify signatures of unknown attacks. Furthermore, this paper addresses the following objectives: (1) To establish and examine most commonly used classifiers in the implementation of IDSs (KNN and Bayes); (2) To evaluate the performance of the individual classifiers independently; and (3) To model a hybrid classifier based on the strengths of the two classifiers. This study adopted a quantitative methodology of collecting and interpreting data. The study had used the NSL-KDD and the original KDD 1999 datasets. This paper evaluated the devised mechanisms over virtualised networked environments and traffic workloads. SVM was used for detecting cycle numbers whereas coefficients and signal shifts were used for completing period detection. Also, this paper has presented rare data for detecting anomalies. Anticipated events that have not occurred and unanticipated events can be detected at various sampling frequencies based on a hybrid approach since no one has proposed a hybrid approach for detecting anomalies. This paper has ranked features from a network traffic database based on a combination of feature selection wrappers and filers and determined that 16 features showed a strong contribution to the anomaly detection task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. RapidMiner, http://rapidminer.com.

  2. MATLAB, http://www.mathworks.com.

References

  1. Dymora P, Mazurek M (2019) Anomaly detection in IoT communication network based on spectral analysis and hurst exponent. Appl Sci 9:5319

    Article  Google Scholar 

  2. Thudumu S, Branch P, Jin J, Singh JJ (2020) A comprehensive survey of anomaly detection techniques for high dimensional big data. J Big Data 7(1):1–30

    Article  Google Scholar 

  3. Huang H, Al-Azzawi H, Brani H (2014) Network traffic anomaly detection. arXiv preprint https://arxiv.org/abs/1402.0856.

  4. Kalinichenko L, Shanin I, Taraban I (2014) Methods for anomaly detection: a survey. In CEUR Workshop Proceedings (Vol. 1297, p. 2025).

  5. Chowdhury MN, Ferens K, Ferens M (2016) Network intrusion detection using machine learning. In Proceedings of the International Conference on Security and Management (SAM) (p. 30). The Steering Committee of the World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp).

  6. Satheesh N, Rathnamma MV, Rajeshkumar G, Sagar PV, Dadheech P, Dogiwal SR, Sengan S (2020) Flow-based anomaly intrusion detection using machine learning model with software defined networking for openflow network. Microprocess Microsyst 79:103285

    Article  Google Scholar 

  7. Kang B, McLaughlin K, Sezer S (2016) Towards a stateful analysis framework for smart grid network intrusion detection. In 4th International Symposium for ICS & SCADA Cyber Security Research 2016 4 (pp. 124–131).

  8. Thottan M, Ji C (2003) Anomaly detection in IP networks. IEEE Trans Signal Process 51:2191–2204

    Article  Google Scholar 

  9. Van NT, Thinh TN (2017) An anomaly-based network intrusion detection system using deep learning. In 2017 International Conference on System Science and Engineering (ICSSE) (pp. 210–214). IEEE.

  10. Atli BG, Miche Y, Kalliola A, Oliver I, Holtmanns S, Lendasse A (2018) Anomaly-based intrusion detection using extreme learning machine and aggregation of network traffic statistics in probability space. Cogn Comput 10:848–863

    Article  Google Scholar 

  11. Viegas EK, Santin AO, Cogo VV, Abreu V (2020) A Reliable Semi-Supervised Intrusion Detection Model: One Year of Network Traffic Anomalies. In ICC 2020–2020 IEEE International Conference on Communications (ICC) (pp. 1–6). IEEE.

  12. Abdulhammed R, Faezipour M, Abuzneid A, AbuMallouh A (2018) Deep and machine learning approaches for anomaly-based intrusion detection of imbalanced network traffic. IEEE Sens Lett 3:1–4

    Article  Google Scholar 

  13. Tang TA, Mhamdi L, McLernon D, Zaidi SAR, Ghogho M (2016) Deep learning approach for network intrusion detection in software defined networking. In 2016 International Conference on Wireless Networks and Mobile Communications (WINCOM) (pp. 258–263). IEEE.

  14. Zaman M, Lung CH (2018) Evaluation of machine learning techniques for network intrusion detection. In NOMS 2018–2018 IEEE/IFIP Network Operations and Management Symposium (pp. 1–5). IEEE.

  15. Lakhina A, Crovella M, Diot C (2005) Mining anomalies using traffic feature distributions. ACM SIGCOMM COMP COM 35:217–228

    Article  Google Scholar 

  16. Selvakumar B, Muneeswaran K (2019) Firefly algorithm-based feature selection for network intrusion detection. Comput Secur 81:148–155

    Article  Google Scholar 

  17. Samrin R, Vasumathi D (2017) Review on anomaly-based network intrusion detection system. In 2017 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT) (pp. 141–147). IEEE.

  18. Aldwairi T, Perera D, Novotny MA (2018) An evaluation of the performance of restricted boltzmann machines as a model for anomaly network intrusion detection. Comput Netw 144:111–119

    Article  Google Scholar 

  19. Vasan KK, Surendiran B (2016) Dimensionality reduction using principal component analysis for network intrusion detection. Perspect Sci 8:510–512

    Article  Google Scholar 

  20. Timčenko V, Gajin S (2017) Ensemble classifiers for supervised anomaly-based network intrusion detection. In 2017 13th IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) (pp. 13–19). IEEE.

  21. Condomines JP, Zhang R, Larrieu N (2019) Network intrusion detection system for UAV ad-hoc communication: From methodology design to real test validation. Ad Hoc Netw 90:101759

    Article  Google Scholar 

  22. Lavrova DS, Alekseev IV, Shtyrkina AA (2018) Security analysis based on controlling dependences of network traffic parameters by wavelet transformation. Autom Control Comput Sci 52:931–935

    Article  Google Scholar 

  23. Du Z, Ma L, Li H, Li Q, Sun G, Liu Z (2018). Network traffic anomaly detection based on wavelet analysis. In 2018 IEEE 16th International Conference on Software Engineering Research, Management and Applications (SERA) (pp. 94–101). IEEE.

  24. Puuska S, Kokkonen T, Alatalo J, Heilimo E (2018) Anomaly-based network intrusion detection using wavelets and adversarial autoencoders. In International Conference on Security for Information Technology and Communications (pp. 234–246). Springer, Cham.

  25. Teng M (2010) Anomaly detection on time series. In 2010 IEEE International Conference on Progress in Informatics and Computing (Vol. 1, pp. 603–608). IEEE.

  26. Chalapathy R, Chawla S (2019) Deep learning for anomaly detection: A survey. arXiv preprint https://arxiv.org/abs/1901.03407.

  27. Habeeb RAA, Nasaruddin F, Gani A, Hashem IAT, Ahmed E, Imran M (2019) Real-time big data processing for anomaly detection: a survey. Int J Inf Manage 45:289–307

    Article  Google Scholar 

  28. Ahmad S, Lavin A, Purdy S, Agha Z (2017) Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262:134–147. https://doi.org/10.1016/j.neucom.2017.04.070

    Article  Google Scholar 

  29. Keogh E, Lin J, Fu A (2005) Hot sax: Efficiently finding the most unusual time series subsequence. In Fifth IEEE International Conference on Data Mining (ICDM'05) (pp. 8-pp). IEEE.

  30. Malhotra P, Vig L, Shroff G, Agarwal P (2015) Long short-term memory networks for anomaly detection in time series. In Proceedings (Vol. 89, pp. 89–94). Presses universitaires de Louvain.

  31. Chandola V, Mithal, V, Kumar V (2008) Comparative evaluation of anomaly detection techniques for sequence data. In 2008 Eighth IEEE international conference on data mining (pp. 743–748). IEEE.

  32. Gama J, Žliobaitė I, Bifet A, Pechenizkiy M, Bouchachia A (2014) A survey on concept drift adaptation. ACM Comput Surv (CSUR) 46:1–37. https://doi.org/10.1145/0000000.0000000

    Article  MATH  Google Scholar 

  33. Pimentel MA, Clifton DA, Clifton L, Tarassenko L (2014) A review of novelty detection. Signal Process 99:215–249

    Article  Google Scholar 

  34. Chen PY, Yang S, McCann JA (2014) Distributed real-time anomaly detection in networked industrial sensing systems. IEEE Trans Ind Electron 62:3832–3842

    Article  Google Scholar 

  35. Spinosa EJ, de Leon F de Carvalho AP, Gama J (2007) Olindda: A cluster-based approach for detecting novelty and concept drift in data streams. In Proceedings of the 2007 ACM symposium on Applied computing (pp. 448–452).

  36. Faria ER, Gama J, Carvalho AC (2013) Novelty detection algorithm for data streams multi-class problems. In Proceedings of the 28th annual ACM symposium on applied computing (pp. 795–800).

  37. Wang T, Xu J, Zhang W, Gu Z, Zhong H (2018) Self-adaptive cloud monitoring with online anomaly detection. Future Gener Comput Syst 80:89–101. https://doi.org/10.1016/j.future.2017.09.067

    Article  Google Scholar 

  38. Han M, Zhang S, Xu M, Qiu T, Wang N (2018) Multivariate chaotic time series online prediction based on improved kernel recursive least squares algorithm. IEEE Trans Cybern 49:1160–1172. https://doi.org/10.1109/TCYB.2018.2789686

    Article  Google Scholar 

  39. Xia L, Liu M, Ning X, Chakrabarty K, Wang Y (2017) Fault-tolerant training with online fault detection for RRAM-based neural computing systems. In Proceedings of the 54th Annual Design Automation Conference 2017 (pp. 1–6). https://doi.org/10.1145/3061639.3062248

  40. Andrysiak T, Saganowski Ł, Maszewski M (2017) Time series forecasting using Holt-Winters model applied to anomaly detection in network traffic. In International Joint Conference SOCO’17-CISIS’17-ICEUTE’17 León, Spain, September 6–8 2017 Proceeding (pp. 567-576). Springer, Cham

  41. Ali AM, Angelov P (2018) Abnormal behaviour detection based on heterogeneous data and data fusion. Soft Comput 22:3187–3201. https://doi.org/10.1007/s00500-017-2989-5

    Article  Google Scholar 

  42. Bezerra CG, Costa BSJ, Guedes LA, Angelov PP (2020) An evolving approach to data streams clustering based on typicality and eccentricity data analytics. Inf 518:13–28

    MathSciNet  MATH  Google Scholar 

  43. Martins RS, Angelov P, Costa BSJ (2018) Automatic detection of computer network traffic anomalies based on eccentricity analysis. In 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) (pp. 1–8). IEEE.

  44. Lee K, Johnson EN (2017) Robust state estimation and online outlier detection using eccentricity analysis. In 2017 IEEE Conference on Control Technology and Applications (CCTA) (pp. 1350–1355). IEEE.

  45. Ding N, Ma H, Gao H, Ma Y, Tan G (2019) Real-time anomaly detection based on long short-term memory and Gaussian mixture model. Comput Electr Eng 79:106458. https://doi.org/10.1016/j.compeleceng.2019.106458

    Article  Google Scholar 

  46. Fahim M, Sillitti A (2019) Anomaly detection, analysis and prediction techniques in IoT environment: a systematic literature review. IEEE Access 7:81664–81681. https://doi.org/10.1109/access.2019.2921912

    Article  Google Scholar 

  47. Huang J, Chai Z, Zhu H (2019). Detecting anomalies in data center physical infrastructures using statistical approaches. J Phys: Conf Ser (Vol. 1176, No. 2, p. 022056). IOP Publishing. https://doi.org/10.1088/1742-6596/1176/2/022056

  48. Pang G, Shen C, Cao L, Hengel AVD (2021) Deep learning for anomaly detection: a review. ACM Comput Surv (CSUR) 54(2):1–38

    Article  Google Scholar 

  49. Zhu G, Li Z, Wu N (2018) Model-based fault identification of discrete event systems using partially observed Petri nets. Automatica 96:201–212. https://doi.org/10.1016/j.automatica.2018.06.039

    Article  MathSciNet  MATH  Google Scholar 

  50. Iglesias F, Zseby T (2015) Analysis of network traffic features for anomaly detection. Mach Learn 101:59–84

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The author is very thankful to all the associated personnel in any reference that contributed in/for the purpose of this research.

Funding

This research is not funded by any resource.

Author information

Authors and Affiliations

  1. Department of Computer Science, Prince Sattam Bin Abdul Aziz University, Al-Kharj, Saudi Arabia

    Mozamel M. Saeed

Authors
  1. Mozamel M. Saeed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mozamel M. Saeed.

Ethics declarations

Conflict of interest

The author further declares that the study holds no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Attribute No

Attribute Name

Description

1

Duration

Length of time duration of the connection

2

Protocol_type

Protocol used in the connection

3

Service

Destination network service used

4

Flag

Status of the connection (Normal or Error)

5

Src_bytes

Number of data bytes transferred from source to destination in single connection

6

Dst_bytes

Number of data bytes transferred from destination to source in single connection

7

Land

If source and destination IP addresses and port numbers are equal then, this variable takes value 1 else 0

8

Wrong_fragment

Total number of wrong fragments in this connection

9

Urgent

Number of urgent packets in this connection. Urgent packets are packets with the urgent bit activated

10

Hot

Number of ‘hot’ indicators in the content such as: entering a system directory, creating programs and executing programs

11

Num_failed_logins

Count of failed login attempts

12

Logged_in

Login Status: 1 if successfully logged in; 0 otherwise

13

Num_compromised

Number of ‘compromised’ conditions

14

Root_shell

1 if root shell is obtained; 0 otherwise

15

Su_attempted

1 if ‘su root’ command attempted or used; 0 otherwise

16

Num_root

Number of ‘root’ accesses or number of operations performed as a root in the connection

17

Num_fule_creations

Number of operations on access control files

18

Num_shells

Number of file creation operations in the connection

19

Num_access_files

Number of operations on access control files

20

Num_outbound_cmds

Number of outbound commands in an ftp session

21

ls_hot_login

1 if the login belongs to the ‘hot’ list; else 0

22

ls_guest_login

1 if the login is a ‘guest’ login; 0 otherwise

23

Count

Number of connections to the same destination host as the current connection in the past two seconds

24

Srv_count

Number of connections to the same service (port number) as the current connection in the past two seconds

25

Serror_rate

The percentage of connections that have activated the flag among the connections aggregated in count

26

Sev_serror_rate

The percentage of connections that have activated the flag among the connections aggregated in srv count

27

Rerror_rate

The percentage of connections that have activated the flag REJ, among the connections aggregated in count

28

Srv_rerror_rate

The percentage of connections that were to the same service, among the connections aggregated in srv_count

29

Same_srv_rate

The percentage of connections that were to the same service, among the connections aggregated in count

30

Diff_srv_rate

The percentage of connections that were to different services, among the connections aggregated in count

31

Srv_diff_host_rate

The percentage of connections that were to different destination machines among the connections aggregated in srv_ count

32

Dst_host_count

Number of connections having the same destination host IP address

33

Dst_host_srv_count

Number of connections having the same port number

34

Dst_host_same_srv_rate

The percentage of connections that were to the same service, among the connections aggregated in dst_host_count

35

Dst_host_diff_srv_rate

The percentage of connections that were to different services, among the connections aggregated in dst_host_count

36

Dst_host_same_src_port_rate

The percentage of connections that were to the same source port, among the connections aggregated in dst_host_srv_count

37

Dst_host_srv_diff_host_rate

The percentage of connections that were to different destination machines, among the connections aggregated in dst_host_srv_count

38

Dst_host_serror_rate

The percentage of connections that have activated the flag among the connections aggregated in dst_host_count

39

Dst_host_srv_serror_rate

The percent of connections that have activated the flag

40

Dst_host_rerror_rate

The percentage of connections that have activated the flag REJ, among the connections aggregated in dst_host_count

41

Dst_host_srv_rerror_rate

The percentage of connections that have activated the flag REJ, among the connections aggregated in dst_host_srv_count

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saeed, M.M. A real-time adaptive network intrusion detection for streaming data: a hybrid approach. Neural Comput & Applic 34, 6227–6240 (2022). https://doi.org/10.1007/s00521-021-06786-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06786-x

Keywords

Search

Navigation