A Dynamic Processing Algorithm for Variable Data in Intranet Security Monitoring

Abstract

Nowadays corporate Intranet Security Monitoring generally relies on SIEM products or SOC platforms. The data comes from a large number of system logs, application running logs and business data, which are generated by network device, security protection device and application systems, etc., is finally stored as normalized data after word segmentation, field parsing and data type mapping. The Intranet Security Monitoring are extremely sensitive to data quality because of the efficiency and accuracy requirements, but the continuous business changes and system upgrades in the intranet environment make both of the data structure and content variable. The existing automated log parsing algorithms are mainly aimed at system logs with a fixed structure, cannot handle variable data with multiple types and structures, besides, the parsing work only completes word segmentation and field parsing. As for data type identification and mapping, there should be several security experts to wait to write static templates, in case the data is changed. In response to the above problems, an ontology model of data knowledge for Intranet Security Monitoring is constructed, and using the computing power of cloud computing, a dynamic processing algorithm for variable data (DPAVD) based on structural information entropy is proposed, in which the correlation between data fields is used as the core factor, can reduce the interference caused by the difference in character expression and meet the requirements of high-quality data for Intranet Security Monitoring.