Evaluation in the absence of absolute ground truth: toward reliable evaluation methodology for scan detectors

Abstract

Although network reconnaissance through scanning has been well explored in the literature, new scan detection proposals with various detection features and capabilities continue to appear. To our knowledge, however, there is little discussion of reliable methodologies to evaluate network scanning detectors. In this paper, we show that establishing ground truth labels of scanning activity on non-synthetic network traces is a more difficult problem relative to labeling conventional intrusions. The main problem stems from lack of absolute ground truth (AGT). We identify the specific types of errors this admits. For real-world network traffic, typically many events can be equally interpreted as legitimate or intrusions, and therefore, establishing AGT is infeasible since it depends on unknowable intent. We explore how an estimated ground truth based on discrete classification criteria can be misleading since typical detection accuracy measures are strongly dependent on the chosen criteria. We also present a methodology for evaluating and comparing scan detection algorithms. The methodology classifies remote addresses based on continuous scores designed to provide a more accurate reference for evaluation. The challenge of conducting a reliable evaluation in the absence of AGT applies to other areas in network intrusion detection, and corresponding requirements and guidelines apply.

Appendices

Appendices

A: Obtaining network traffic for evaluation

This appendix provides background related to Sect. 2. Ideally, non-synthetic data sets labeled to identify the target classes of intrusions or anomalies might be used to evaluate the detector results. However, for scan detectors, not only are there no publicly available non-synthetic labeled data sets, but there are also no synthetic labeled data sets. In fact, the problem is not simply that currently no such labeled data sets are available, but rather that for scan detection, it is not clear that such labeling is reliable, given the impossibility of determining the intent of a connection attempt (see Sect. 3.1). The following subsections summarize known mechanisms used in the literature to obtain network traffic for evaluation.

1.1 A.1 Simulated data sets

In simulation approaches, network traffic is generated by custom network simulation software to model real network behavior of a particular configuration. Network simulation helps in verifying the correctness of a new or existing network intrusion detector and in predicting its performance, especially in complex network settings. Using simulation, both the network traffic and broader scanning campaigns are simulated. In addition to avoiding the legal and privacy issues regarding using real-world traffic, simulation enables testing of a scan detector using various network configurations while controlling the characteristics of the scans.

On the other hand, simulation has known shortcomings. First, given that network traffic is diverse and variability is expected in both network-level (including background traffic) and application-level protocols [34], the simulated traffic may not realistically represent real-world traffic. It is important to test detectors on traces from operational networks that resemble those where the detectors will be deployed. Simulation must include Internet traffic that is hard to simulate or model due to heterogeneity, scale, and rapid change in the Internet’s dynamics [6]. Second, the generated intrusions or reconnaissance activity (i.e., scan events) might only represent a subset of existing patterns in the wild. The inadequate understanding of many anomalies and reconnaissance activity make it hard to characterize them precisely [28].

Simulating network scanning activity is even more challenging because: (i) Based on the remote host’s intent, an inbound connection attempt can be considered either as a scanning activity or benign; and (ii) it is not necessary that network traffic originated from a scanner contains or is associated with malicious payloads.

1.2 A.2 Emulated data sets

In network emulation, a subset of the studied traffic is the real-world data. For example, starting with a real-world data set, scanning events could be simulated to the target IP address range and then injected in the data set. Emulation approaches seem better than simulation because network traces can be collected from an environment representative of where the detector will be deployed. However, in emulation, the starting data set could include scanning data as well which will affect the evaluation of the detector; filtering out such scanning traffic requires an accurate scan detector in the first place and therefore full knowledge of ground truth. Also, the feedback effects of the interaction between the simulated scanning events and the real-world network traffic are not captured. In addition, similar to simulation, the generated scanning traffic might not reflect the variety of scanning patterns in the wild. Both simulation and emulation approaches are known to be a good starting point to test the detector ability to identify certain types of intrusions but not sufficient in its own to evaluate the detector accuracy and capabilities [27].

1.3 A.3 Real-world traffic data sets

Given the drawbacks of simulation and emulation, and the lack of publicly available non-synthetic accurately labeled data sets, researchers usually evaluate detectors using their own network traffic. Such gathered data sets provide real-world data of both network traffic and intrusions (e.g., scanning events). While anonymization of these data sets (e.g., see [36]) helps substantially in removing private and other sensitive information, information leakage is still possible [5], and thus, for ethical, privacy, and legal reasons, such data sets often stay private.

Such collected data sets are subject to the following issues: (i) The network environment of the data set may exhibit artifacts that until identified and explained, significantly affect detector performance; (ii) the nature and size of the network might not represent a realistic target deployment environment; (iii) with the lack of control over the characteristics of the scan events, it may be hard to determine the conditions under which a detector will perform well; and (iv) obtaining a reliable ground truth of the existing intrusions that the detector is designed to detect is challenging and might be infeasible for some types of intrusions.

B: Example Heuristic for establishing a continuous GTR

In establishing a GTR of scanners for a given network data set, time and computational resource constraints are less than those for a real-time detector. As discussed in Sect. 3.4, remote hosts’ traffic over the entire data set capture period can be examined to establish a GTR of scanners. While the detection features used to obtain a GTR should differ from those used in the evaluated scan detection algorithm, to avoid a circular argument, we argue that monitoring network traffic over a relatively long period of time and/or over a large IP address space breaks such circularity. That is, we argue that time duration is a distinct dimension in detection. Nevertheless, it remains desirable that the GTR includes detection features that are not used in the evaluated online detector. The requirements of using a GTR for evaluating a detector or comparing two or more detectors are discussed in Sect. 2. In this section, we present one detection heuristic as an illustration example of how to build a continuous GTR of remote scanners that assigns each remote host a score according to its observed network traffic, as described in Sect. 4.3.

Several heuristics were proposed in the literature for detecting known scanning behaviors rarely observed in benign traffic. Heuristics are often developed for real-time scan detectors and thus oriented toward fast detection with as few false positives as possible. The following are commonly used detection heuristics: (i) contacting non-existing local hosts or network services [9, 33]; (ii) making unsuccessful connection attempts since, unlike legitimate network traffic, most connection attempts that are part of reconnaissance activities are expected to fail, given that active network services are unknown prior to scanning [8, 12, 14]; (iii) exchanging low volume of data with local hosts [8, 40]; (iv) contacting many local hosts or ports in a small time window [3, 9, 29, 30, 33]; and (v) contacting rarely accessed local hosts or ports [16, 18, 35].

Given that some scanning events might look like rare normal traffic if analyzed in isolation, repeated occurrences of what might individually be considered abnormal and the absence of normal traffic provide more confidence of malicious intent. Among the commonly used scan detection heuristics discussed above, to obtain a GTR of scanners, we employ two heuristics of abnormal traffic that seem hard to evade by scanners: (i) failed connection attempts initiated by the remote; and (ii) lack of data exchange, particularly outbound traffic to the remote. In addition, we employ two heuristics as a sign for normal traffic: (iii) successful connections initiated by the remote and (iv) connection attempts initiated by local hosts to the remote (whether successful or unsuccessful). While we combine these four heuristics in one main heuristic that captures many known scanning patterns, it is expandable by accommodating the addition of further detection heuristics.

For any connection attempt {remote IP address, local IP address, destination port} in a data set, including both inbound and outbound traffic, only the first event involving this pair is taken into account. Let $n_i$ be the number of local hosts with port $i$ open; the probability of a scanner to make a successful connection to the port $i$ is $1/n_i$ (assuming random scanning). Therefore, a successful connection to port $i$ is assigned a weight of $1- (1/n_i)$, whereas a failed connection attempt to port $i$ is always assigned a weight of $1$. We combine the connection state heuristics (i.e., (i), (iii), and (iv)) in one ratio GTR1 that is calculated for each remote $R$ as follows (see Table 4 for notation):

$$\begin{aligned} {\text{ GTR1}}_{R}=\frac{F_R}{F_R + \sum _{j=1}^{S_R} W_{R}^{j} + O_R } \end{aligned}$$

(3)

The closer the ${\text{ GTR}}1_{R}$ score value is to one, the higher the probability that $R$ is a scanner. Similarly, the closer the score value of $R$ is to zero, the higher the probability that $R$ is benign. That is, failed inbound connection attempts increase ${\text{ GTR}}1_{R}$ toward being a scanner, while successful inbound and outbound connection attempts (whether successful or unsuccessful) decrease ${\text{ GTR}}1_{R}$ toward being benign.

Table 4 Notation for Eqs. (3) and (4)

Unlike previous approaches, for the data exchange feature [i.e., (ii)], we use the count of successful inbound connections that contain at least $k$ outbound packets with only ACK flag set (we call this count $X_{R}$). For a given remote, this is an indication of traffic sent by local hosts to the remote. Thus, the higher this count, the more data packets are sent to the remote. The higher the rate of inbound connections with low data exchange to all successful inbound connections initiated by the remote, the more evidence of malicious intent. Since the majority of successful TCP connections have many outbound packets with only the ACK flag set, a small threshold, say $k < 5$, indicates a low data exchange. Note that for failed inbound connection attempts, there will be no outbound packets with only the ACK flag set. We argue that this feature is particularly valuable to identify fortuitous scanners that have probed some active services since the data exchange with these services is expected to be minimal. The data exchange feature (${\text{ GTR}}_2$) is calculated as follows:

$$\begin{aligned} {\text{ GTR}}2_{R} = \frac{X_{R}}{S_R} \end{aligned}$$

(4)

The closer the ${\text{ GTR}}2_{R}$ score is to one, the higher the probability that $R$ is a scanner since this indicates a low data exchange in most connection attempts initiated by $R$. Similarly, the closer the ${\text{ GTR}}2_{R}$ to zero, the higher probability that $R$ is benign. The total score of $R$ is derived by taking the maximum value of the scores obtained from the detection features used in the evaluation:

$$\begin{aligned} {\text{ GTR}}_R = {\text{ max}}({\text{ GTR}}1_{R}, {\text{ GTR}}2_{R}) \end{aligned}$$

(5)

Note that given that there are several scan detection heuristics that capture different known scanning patterns, the absence of one pattern is not an evidence that $R$ is not a scanner. Therefore, ${\text{ GTR}}_{R}$ should reflect the scan detection heuristic with the highest value. To evaluate a TOE D, for each $R$ in an evaluation data set, ${\text{ GTR}}_{R}$ is compared with $D_{R}$ as explained in Sect. 4.3. While we argue that computing ${\text{ GTR}}1_{R}$ and ${\text{ GTR}}2_{R}$ over a relatively long period of time and/or over a large IP address space is sufficient to identify the majority of scanners, further detection features can be added to Eq. (5) in a similar way, according to the TOE(s) in question.