Abstract
In a cybersecurity operations center (CSOC), under normal operating conditions in a day, sufficient numbers of analysts are available to analyze the amount of alert workload generated by intrusion detection systems (IDSs). For the purpose of this paper, this means that the cybersecurity analysts can fully investigate each and every alert that is generated by the IDSs in a reasonable amount of time. However, there are a number of disruptive factors that can adversely impact the normal operating conditions such as (1) higher alert generation rates from a few IDSs, (2) new alert patterns that decreases the throughput of the alert analysis process, and (3) analyst absenteeism. The impact of all the above factors is that the alerts wait for a long duration before being analyzed, which impacts the readiness of the CSOC. It is imperative that the readiness of the CSOC be quantified, which in this paper is defined as the level of operational effectiveness (LOE) of a CSOC. LOE can be quantified and monitored by knowing the exact deviation of the CSOC conditions from normal and how long it takes for the condition to return to normal. In this paper, we quantify LOE by defining a new metric called total time for alert investigation (TTA), which is the sum of the waiting time in the queue and the analyst investigation time of an alert after its arrival in the CSOC database. A dynamic TTA monitoring framework is developed in which a nominal average TTA per hour (avgTTA/hr) is established as the baseline for normal operating condition using individual TTA of alerts that were investigated in that hour. At the baseline value of avgTTA/hr, LOE is considered to be ideal. Also, an upper-bound (threshold) value for avgTTA/hr is established, below which the LOE is considered to be optimal. Several case studies illustrate the impact of the above disruptive factors on the dynamic behavior of avgTTA/hr, which provide useful insights about the current LOE of the system. Also, the effect of actions taken to return the CSOC to its normal operating condition is studied by varying both the amount and the time of action, which in turn impacts the dynamic behavior of avgTTA/hr. Results indicate that by using the insights learnt from measuring, monitoring, and controlling the dynamic behavior of avgTTA/hr, a manager can quantify and color-code the LOE of the CSOC. Furthermore, the above insights allow for a deeper understanding of acceptable downtime for the IDS, acceptable levels for absenteeism, and the recovery time and effort needed to return the CSOC to its ideal LOE.
Similar content being viewed by others
References
CIO, DON Cyber Crime Handbook, Dept. of Navy, Washington, DC (2008)
Zimmerman, C.: The Strategies of a World-Class Cybersecurity Operations Center. The MITRE Corporation, McLean (2014)
Anderson, J.P.: Computer security threat monitoring and surveillance. Tech. rep., James P. Anderson Co., Fort Washington, PA (1980)
Denning, D.E.: An intrusion-detection model. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 118–131. Oakland, CA (1986)
Denning, D.E.: An intrusion-detection model. IEEE Trans. Softw. Eng. 13(2), 222 (1987)
Northcutt, S., Novak, J.: Network Intrusion Detection, 3rd edn. New Riders Publishing, Thousand Oaks (2002)
Di Pietro, R., Mancini, L.V. (eds.): Intrusion Detection Systems, Advances in Information Security, vol. 38. Springer, Berlin (2008)
Subrahmanian, V.S., Ovelgonne, M., Dumitras, T., Prakash, B.A.: The Global Cyber-Vulnerability Report. Springer, Switzerland (2015)
Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 305–316 (2010)
Barbara, D., Jajodia, S. (eds.): Application of Data Mining in Computer Security, Advances in Information Security, vol. 6. Springer, Berlin (2002)
Helm, J.E., AhmadBeygi, S., Van Oyen, M.P.: Design and analysis of hospital admission control for operational effectiveness. Prod. Oper. Manag. 20(3), 359 (2011)
Chen, Z., King, W., Pearcey, R., Kerba, M., Mackillop, W.J.: The relationship between waiting time for radiotherapy and clinical outcomes: a systematic review of the literature. Radiother. Oncol. 87(1), 3 (2008)
Guerriero, F., Guido, R.: Operational research in the management of the operating theatre: a survey. Health Care Manag. Sci. 14(1), 89 (2011)
Vansteenwegen, P., Van Oudheusden, D.: Decreasing the passenger waiting time for an intercity rail network. Transp. Res. Part B: Methodol. 41(4), 478 (2007)
Kelly, C.: A framework for improving operational effectiveness and cost efficiency in emergency planning and response. Disaster Prev. Manag. Int. J. 4(3), 25 (1995)
Robbins, T.R., Medeiros, D.J., Dum, P.: Evaluating arrival rate uncertainty in call centers. In: Proceedings of the 2006 Winter Simulation Conference, pp. 2180–2187. IEEE (2006)
Jack, E.P., Bedics, T.A., McCary, C.E.: Operational challenges in the call center industry: a case study and resource-based framework. Manag. Serv. Qual. Int. J. 16(5), 477 (2006)
Tijms, H.: New and old results for the M/D/c queue. AEU Int. J. Electron. Commun. 60(2), 125 (2006)
Marianov, V., Serra, D.: Location models for airline hubs behaving as M/D/c queues. Comput. Oper. Res. 30(7), 983 (2003)
Julisch, K., Dacier, M.: Mining intrusion detection alarms for actionable knowledge. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 366–375 (2002)
Goodall, J.R., Lutters, W.G., Komlodi, A.: I know my network: collaboration and expertise in intrusion detection. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 342–345 (2004)
Ganesan, R., Jajodia, S., Cam, H.: Optimal scheduling of cybersecurity analyst for minimizing risk. ACM Trans. Intell. Syst. Technol. 8(4), 52:1–52:33 (2017). doi:10.1145/2914795
Ganesan, R., Jajodia, S., Shah, A., Cam, H.: Dynamic scheduling of cybersecurity analysts for minimizing risk using reinforcement learning. ACM Trans. Intell. Syst. Technol. 8(1), 4:1 (2016). doi:10.1145/2882969
Acknowledgements
The authors would like to thank Dr. Cliff Wang of the Army Research Office for the many discussions which served as the inspiration for this research.
Author information
Authors and Affiliations
Corresponding author
Additional information
Shah, Ganesan, and Jajodia were partially supported by the Army Research Office under grants W911NF-13-1-0421 and W911NF-15-1-0576 and by the Office of Naval Research under grant N00014-15-1-2007.
Rights and permissions
About this article
Cite this article
Shah, A., Ganesan, R., Jajodia, S. et al. A methodology to measure and monitor level of operational effectiveness of a CSOC. Int. J. Inf. Secur. 17, 121–134 (2018). https://doi.org/10.1007/s10207-017-0365-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10207-017-0365-1