Skip to main content
SpringerLink
Log in

Online and Scalable Adaptive Cyber Defense

  • Chapter
  • First Online:
Adversarial and Uncertain Reasoning for Adaptive Cyber Defense

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 11830))

Abstract

This chapter introduces cyber security researchers to key concepts in the data streaming and sketching literature that are relevant to Adaptive Cyber Defense (ACD) and Moving Target Defense (MTD). We begin by observing the challenges met in the big data realm. Particular attention is paid to the need for compact representations of large datasets, as well as designing algorithms that are robust to changes in the underlying dataset. We present a summary of the key research and tools developed in the data stream and sketching literature, with a focus on practical applications. Finally, we present several concrete extensions to problems related to ACD applications throughout this book, with a focus on improving scalability.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ahn, K.J., Guha, S., McGregor, A.: Analyzing graph structure via linear measurements. In: Proceedings of the Twenty-Third Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 459–467. SIAM (2012)

    Google Scholar 

  2. Ahn, K.J., Guha, S., McGregor, A.: Graph sketches: sparsification, spanners, and subgraphs. In: Proceedings of the 31st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, pp. 5–14. ACM (2012)

    Google Scholar 

  3. Alahakoon, T., Tripathi, R., Kourtellis, N., Simha, R., Iamnitchi, A.: K-path centrality: a new centrality measure in social networks. In: Proceedings of the 4th Workshop on Social Network Systems, p. 1. ACM (2011)

    Google Scholar 

  4. Alon, N., Matias, Y., Szegedy, M.: The space complexity of approximating the frequency moments. J. Comput. Syst. Sci. 58, 137–147 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  5. Andoni, A., Krauthgamer, R., Onak, K.: Streaming algorithms via precision sampling. In: 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science (FOCS), pp. 363–372. IEEE (2011)

    Google Scholar 

  6. Bader, D.A., Kintali, S., Madduri, K., Mihail, M.: Approximating betweenness centrality. In: Bonato, A., Chung, F.R.K. (eds.) WAW 2007. LNCS, vol. 4863, pp. 124–137. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-77004-6_10

    Chapter  Google Scholar 

  7. Bar-Yossef, Z., Jayram, T.S., Kumar, R., Sivakumar, D., Trevisan, L.: Counting distinct elements in a data stream. In: Rolim, J.D.P., Vadhan, S. (eds.) RANDOM 2002. LNCS, vol. 2483, pp. 1–10. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45726-7_1

    Chapter  MATH  Google Scholar 

  8. Barabási, A.L., Albert, R.: Emergence of scaling in random networks. Science 286(5439), 509–512 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  9. Bergamini, E., Meyerhenke, H., Staudt, C.L.: Approximating betweenness centrality in large evolving networks. In: 2015 Proceedings of the Seventeenth Workshop on Algorithm Engineering and Experiments (ALENEX), pp. 133–146. SIAM (2014)

    Google Scholar 

  10. Boldi, P., Rosa, M., Vigna, S.: HyperANF: approximating the neighbourhood function of very large graphs on a budget. In: Proceedings of the 20th International Conference on World Wide Web, pp. 625–634. ACM (2011)

    Google Scholar 

  11. Boldi, P., Vigna, S.: Axioms for centrality. Internet Math. 10(3–4), 222–262 (2014)

    Article  MathSciNet  Google Scholar 

  12. Brandes, U.: A faster algorithm for betweenness centrality. J. Math. Sociol. 25(2), 163–177 (2001)

    Article  MATH  Google Scholar 

  13. Brandes, U., Pich, C.: Centrality estimation in large networks. Int. J. Bifurc. Chaos 17(07), 2303–2318 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  14. Cárdenas, A.A., Manadhata, P.K., Rajan, S.P.: Big data analytics for security. IEEE Secur. Priv. 11(6), 74–76 (2013)

    Article  Google Scholar 

  15. Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams. In: Widmayer, P., Eidenbenz, S., Triguero, F., Morales, R., Conejo, R., Hennessy, M. (eds.) ICALP 2002. LNCS, vol. 2380, pp. 693–703. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45465-9_59

    Chapter  Google Scholar 

  16. Clarkson, K.L., Woodruff, D.P.: Numerical linear algebra in the streaming model. In: Proceedings of the Forty-First Annual ACM Symposium on Theory of Computing, pp. 205–214. ACM (2009)

    Google Scholar 

  17. Clarkson, K.L., Woodruff, D.P.: Low-rank approximation and regression in input sparsity time. J. ACM (JACM) 63(6), 54 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  18. Cohen, R., Katzir, L., Yehezkel, A.: A minimal variance estimator for the cardinality of big data set intersection. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 95–103. ACM (2017)

    Google Scholar 

  19. Cormode, G., Datar, M., Indyk, P., Muthukrishnan, S.: Comparing data streams using hamming norms (how to zero in). IEEE Trans. Knowl. Data Eng. 15(3), 529–540 (2003)

    Article  Google Scholar 

  20. Cormode, G., Muthukrishnan, S.: An improved data stream summary: the count-min sketch and its applications. J. Algorithms 55(1), 58–75 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  21. Cormode, G., Muthukrishnan, S.: What’s hot and what’s not: tracking most frequent items dynamically. ACM Trans. Database Syst. (TODS) 30(1), 249–278 (2005)

    Article  Google Scholar 

  22. Deng, F., Rafiei, D.: New estimation algorithms for streaming data: count-min can do more (2007)

    Google Scholar 

  23. Dietzfelbinger, M., Hagerup, T., Katajainen, J., Penttonen, M.: A reliable randomized algorithm for the closest-pair problem. J, Algorithms 25(1), 19–51 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  24. Durand, M., Flajolet, P.: Loglog counting of large cardinalities. In: Di Battista, G., Zwick, U. (eds.) ESA 2003. LNCS, vol. 2832, pp. 605–617. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-39658-1_55

    Chapter  Google Scholar 

  25. Ertl, O.: New cardinality estimation algorithms for HyperLogLog sketches. arXiv preprint arXiv:1702.01284 (2017)

  26. Estan, C., Varghese, G., Fisk, M.: Bitmap algorithms for counting active flows on high speed links. In: Proceedings of the 3rd ACM SIGCOMM Conference on Internet Measurement, pp. 153–166. ACM (2003)

    Google Scholar 

  27. Fan, L., Cao, P., Almeida, J., Broder, A.Z.: Summary cache: a scalable wide-area web cache sharing protocol. IEEE/ACM Trans. Netw. (TON) 8(3), 281–293 (2000)

    Article  Google Scholar 

  28. Feigenbaum, J., Kannan, S., McGregor, A., Suri, S., Zhang, J.: On graph problems in a semi-streaming model. Theoret. Comput. Sci. 348(2–3), 207–216 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  29. Flajolet, P., Fusy, É., Gandouet, O., Meunier, F.: HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. In: Discrete Mathematics and Theoretical Computer Science. pp. 137–156 (2007)

    Google Scholar 

  30. Flajolet, P., Martin, G.N.: Probabilistic counting algorithms for data base applications. J. Comput. Syst. Sci. 31(2), 182–209 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  31. Goyal, A., Daumé III, H., Cormode, G.: Sketch algorithms for estimating point queries in NLP. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1093–1103. Association for Computational Linguistics (2012)

    Google Scholar 

  32. Green, O., McColl, R., Bader, D.A.: A fast algorithm for streaming betweenness centrality. In: 2012 International Conference on Privacy, Security, Risk and Trust (PASSAT) and 2012 International Conference on Social Computing (SocialCom), pp. 11–20. IEEE (2012)

    Google Scholar 

  33. Guha, S., McGregor, A.: Graph streams and sketches: resources (2018). https://people.cs.umass.edu/~mcgregor/graphs/

  34. Gupta, P., Goel, A., Lin, J., Sharma, A., Wang, D., Zadeh, R.: WTF: the who to follow service at Twitter. In: Proceedings of the 22nd International Conference on World Wide Web, pp. 505–514. ACM (2013)

    Google Scholar 

  35. Hayashi, T., Akiba, T., Yoshida, Y.: Fully dynamic betweenness centrality maintenance on massive networks. Proc. VLDB Endow. 9(2), 48–59 (2015)

    Article  Google Scholar 

  36. Heule, S., Nunkesser, M., Hall, A.: HyperLogLog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm. In: Proceedings of the 16th International Conference on Extending Database Technology, pp. 683–692. ACM (2013)

    Google Scholar 

  37. Indyk, P.: Stable distributions, pseudorandom generators, embeddings, and data stream computation. J. ACM (JACM) 53(3), 307–323 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  38. Indyk, P., Woodruff, D.: Optimal approximations of the frequency moments of data streams. In: Proceedings of the Thirty-Seventh Annual ACM Symposium on Theory of Computing, pp. 202–208. ACM (2005)

    Google Scholar 

  39. Johnson, W.B., Lindenstrauss, J.: Extensions of Lipschitz mappings into a Hilbert space. Contemp. Math. 26(189–206), 1 (1984)

    MathSciNet  MATH  Google Scholar 

  40. Jowhari, H., Sağlam, M., Tardos, G.: Tight bounds for Lp samplers, finding duplicates in streams, and related problems. In: Proceedings of the Thirtieth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 49–58. ACM (2011)

    Google Scholar 

  41. Kane, D.M., Nelson, J., Woodruff, D.P.: An optimal algorithm for the distinct elements problem. In: Proceedings of the Twenty-Ninth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 41–52. ACM (2010)

    Google Scholar 

  42. Kang, C., Kraus, S., Molinaro, C., Spezzano, F., Subrahmanian, V.: Diffusion centrality: a paradigm to maximize spread in social networks. Artif. Intell. 239, 70–96 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  43. Kapralov, M., Lee, Y.T., Musco, C., Musco, C., Sidford, A.: Single pass spectral sparsification in dynamic streams. SIAM J. Comput. 46(1), 456–477 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  44. Kourtellis, N., Alahakoon, T., Simha, R., Iamnitchi, A., Tripathi, R.: Identifying high betweenness centrality nodes in large social networks. Soc. Netw. Anal. Min. 3(4), 899–914 (2013)

    Article  Google Scholar 

  45. Kourtellis, N., Morales, G.D.F., Bonchi, F.: Scalable online betweenness centrality in evolving graphs. IEEE Trans. Knowl. Data Eng. 27(9), 2494–2506 (2015)

    Article  Google Scholar 

  46. Li, Y., Nguyen, H.L., Woodruff, D.P.: On sketching matrix norms and the top singular vector. In: Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1562–1581. Society for Industrial and Applied Mathematics (2014)

    Google Scholar 

  47. Li, Y., Nguyen, H.L., Woodruff, D.P.: Turnstile streaming algorithms might as well be linear sketches. In: Proceedings of the Forty-Sixth Annual ACM Symposium on Theory of Computing, pp. 174–183. ACM (2014)

    Google Scholar 

  48. Li, Y., Woodruff, D.P.: Tight bounds for sketching the operator norm, Schatten norms, and subspace embeddings. In: LIPIcs-Leibniz International Proceedings in Informatics, vol. 60. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2016)

    Google Scholar 

  49. Mahoney, M.W., et al.: Randomized algorithms for matrices and data. Found. Trends® Mach. Learn. 3(2), 123–224 (2011)

    Google Scholar 

  50. Malewicz, G., et al.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 135–146. ACM (2010)

    Google Scholar 

  51. McGregor, A.: Graph mining on streams. In: Liu, L., Özsu, M.T. (eds.) Encyclopedia of Database Systems, pp. 1271–1275. Springer, Boston (2009). https://doi.org/10.1007/978-0-387-39940-9_184

    Chapter  Google Scholar 

  52. Monemizadeh, M., Woodruff, D.P.: 1-pass relative-error Lp-sampling with applications. In: Proceedings of the Twenty-First Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1143–1160. SIAM (2010)

    Google Scholar 

  53. Muthukrishnan, S., et al.: Data streams: algorithms and applications. Found. Trends® Theor. Comput. Sci. 1(2), 117–236 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  54. Myers, S.A., Sharma, A., Gupta, P., Lin, J.: Information network or social network?: the structure of the Twitter follow graph. In: Proceedings of the 23rd International Conference on World Wide Web, pp. 493–498. ACM (2014)

    Google Scholar 

  55. Nelson, J., Nguyên, H.L.: OSNAP: faster numerical linear algebra algorithms via sparser subspace embeddings. In: 2013 IEEE 54th Annual Symposium on Foundations of Computer Science (FOCS), pp. 117–126. IEEE (2013)

    Google Scholar 

  56. Nelson, J., Nguyn, H.L., Woodruff, D.P.: On deterministic sketching and streaming for sparse recovery and norm estimation. Linear Algebra Appl. 441, 152–167 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  57. Nisan, N.: Pseudorandom generators for space-bounded computation. Combinatorica 12(4), 449–461 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  58. Palmer, C.R., Gibbons, P.B., Faloutsos, C.: ANF: a fast and scalable tool for data mining in massive graphs. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 81–90. ACM (2002)

    Google Scholar 

  59. Pearce, R.: Triangle counting for scale-free graphs at scale in distributed memory. In: 2017 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–4. IEEE (2017)

    Google Scholar 

  60. Pearce, R., Gokhale, M., Amato, N.M.: Faster parallel traversal of scale free graphs at extreme scale with vertex delegates. In: SC14: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 549–559. IEEE (2014)

    Google Scholar 

  61. Priest, B.W.: Semi-streaming approximation of centrality indices in massive graphs. Ph.D. thesis, Dartmouth College (2019)

    Google Scholar 

  62. Priest, B.W., Pearce, R., Sanders, G.: Estimating edge-local triangle count heavy hitters in edge-linear time and almost-vertex-linear space. In: 2018 IEEE High Performance Extreme Computing Conference (HPEC). IEEE (2018)

    Google Scholar 

  63. Pătraşcu, M., Thorup, M.: The power of simple tabulation hashing. J. ACM (JACM) 59(3), 14 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  64. Qin, J., Kim, D., Tung, Y.: LogLog-beta and more: a new algorithm for cardinality estimation based on LogLog counting. arXiv preprint arXiv:1612.02284 (2016)

  65. Riondato, M., Kornaropoulos, E.M.: Fast approximation of betweenness centrality through sampling. Data Min. Knowl. Disc. 30(2), 438–475 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  66. Sun, X., Dai, J., Liu, P., Singhal, A., Yen, J.: Using bayesian networks for probabilistic identification of zero-day attack paths. IEEE Trans. Inf. Forensics Secur. 13(10), 2506–2521 (2018)

    Article  Google Scholar 

  67. Ting, D.: Streamed approximate counting of distinct elements: Beating optimal batch methods. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 442–451. ACM (2014)

    Google Scholar 

  68. Upstill, T., Craswell, N., Hawking, D.: Predicting fame and fortune: PageRank or indegree. In: Proceedings of the Australasian Document Computing Symposium, ADCS, pp. 31–40 (2003)

    Google Scholar 

  69. Vu, H.: Data stream algorithms for large graphs and high dimensional data (2018)

    Google Scholar 

  70. Wegman, M.N., Carter, J.L.: New hash functions and their use in authentication and set equality. J. Comput. Syst. Sci. 22(3), 265–279 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  71. Wei, W., Carley, K.: Real time closeness and betweenness centrality calculations on streaming network data. In: Proceedings of the 2014 ASE Big-Data/SocialCom/Cybersecurity Conference, Stanford University (2014)

    Google Scholar 

  72. Whang, K.Y., Vander-Zanden, B.T., Taylor, H.M.: A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. (TODS) 15(2), 208–229 (1990)

    Article  Google Scholar 

  73. Woodruff, D.P., et al.: Sketching as a tool for numerical linear algebra. Found. Trends® Theor. Comput. Sci. 10(1–2), 1–157 (2014)

    Google Scholar 

  74. Xiao, Q., Zhou, Y., Chen, S.: Better with fewer bits: improving the performance of cardinality estimation of large data streams. In: INFOCOM 2017-IEEE Conference on Computer Communications, pp. 1–9. IEEE (2017)

    Google Scholar 

  75. Yoshida, Y.: Almost linear-time algorithms for adaptive betweenness centrality using hypergraph sketches. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1416–1425. ACM (2014)

    Google Scholar 

  76. Zhang, Q., Pell, J., Canino-Koning, R., Howe, A.C., Brown, C.T.: These are not the k-mers you are looking for: efficient online k-mer counting using a probabilistic data structure. PLoS ONE 9(7), e101271 (2014)

    Article  Google Scholar 

Download references

Acknowledgements

The work presented in this chapter was supported by the Army Research Office under grant W911NF-13-1-0421.

Author information

Authors and Affiliations

  1. Dartmouth College, Hanover, NH, USA

    Benjamin W. Priest & George Cybenko

  2. University of Michigan, Ann Arbor, MI, USA

    Satinder Singh

  3. George Mason University, Fairfax, VA, USA

    Massimiliano Albanese

  4. Penn State, State College, PA, USA

    Peng Liu

Authors
  1. Benjamin W. Priest
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George Cybenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Satinder Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Massimiliano Albanese
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benjamin W. Priest .

Editor information

Editors and Affiliations

  1. George Mason University, Fairfax, VA, USA

    Sushil Jajodia

  2. Dartmouth College, Hanover, NH, USA

    George Cybenko

  3. Pennsylvania State University, University Park, PA, USA

    Peng Liu

  4. Army Research Laboratory, Triangle Park, NC, USA

    Cliff Wang

  5. University of Michigan, Ann Arbor, MI, USA

    Michael Wellman

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Priest, B.W., Cybenko, G., Singh, S., Albanese, M., Liu, P. (2019). Online and Scalable Adaptive Cyber Defense. In: Jajodia, S., Cybenko, G., Liu, P., Wang, C., Wellman, M. (eds) Adversarial and Uncertain Reasoning for Adaptive Cyber Defense. Lecture Notes in Computer Science(), vol 11830. Springer, Cham. https://doi.org/10.1007/978-3-030-30719-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30719-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30718-9

  • Online ISBN: 978-3-030-30719-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation