An empirical test of the accuracy of an attack graph analysis tool

The purpose of this paper is to test the practical utility of attack graph analysis. Attack graphs have been proposed as a viable solution to many problems in computer network security management. After individual vulnerabilities are identified with a vulnerability scanner, an attack graph can relate the individual vulnerabilities to the possibility of an attack and subsequently analyze and predict which privileges attackers could obtain through multi-step attacks (in which multiple vulnerabilities are exploited in sequence).

The attack graph tool, MulVAL, was fed information from the vulnerability scanner Nexpose and network topology information from 8 fictitious organizations containing 199 machines. Two teams of attackers attempted to infiltrate these networks over the course of two days and reported which machines they compromised and which attack paths they attempted to use. Their reports are compared to the predictions of the attack graph analysis.

The prediction accuracy of the attack graph analysis was poor. Attackers were more than three times likely to compromise a host predicted as impossible to compromise compared to a host that was predicted as possible to compromise. Furthermore, 29 per cent of the hosts predicted as impossible to compromise were compromised during the two days. The inaccuracy of the vulnerability scanner and MulVAL’s interpretation of vulnerability information are primary reasons for the poor prediction accuracy.

Although considerable research contributions have been made to the development of attack graphs, and several analysis methods have been proposed using attack graphs, the extant literature does not describe any tests of their accuracy under realistic conditions.