Skip to main content
SpringerLink
Log in

Pattern Matching for Perception Streams

  • Conference paper
  • First Online:
Runtime Verification (RV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14245))

Included in the following conference series:

Abstract

We introduce Spatial Regular Expressions (SpREs) as a novel querying language for pattern matching over perception streams containing spatial and temporal data. To highlight the capabilities of SpREs, we developed the Strem tool as a matching framework that works in both the offline and online domain. We demonstrate the tool through an offline example with an AV dataset, an online example through an integration with the ROS and CARLA simulators, and an initial set of performance benchmarks on various SpRE queries. From our designed matching framework, we are able to find over 20,000 matches within 296 ms making it highly usable in runtime monitoring applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://crates.io/crates/strem.

References

  1. Aho, A.V.: Pattern matching in strings. In: Formal Language Theory, pp. 325–347. Elsevier (1980)

    Google Scholar 

  2. Aho, A.V., Lam, M.S., Sethi, R., Ullman, J.D.: Compilers: Principles, Techniques and Tools. Addison-Wesley, Pearson (2020)

    MATH  Google Scholar 

  3. Alfred, V.: Algorithms for finding patterns in strings. In: Algorithms and Complexity, vol. 1, p. 255 (2014)

    Google Scholar 

  4. Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983)

    Article  MATH  Google Scholar 

  5. Bai, Z., et al.: Cyber mobility mirror: a deep learning-based real-world object perception platform using roadside LiDAR. IEEE Trans. Intell. Transp. Syst. 24, 9476–9489 (2023)

    Article  Google Scholar 

  6. Balakrishnan, A., Deshmukh, J., Hoxha, B., Yamaguchi, T., Fainekos, G.: PerceMon: online monitoring for perception systems. In: Feng, L., Fisman, D. (eds.) RV 2021. LNCS, vol. 12974, pp. 297–308. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88494-9_18

    Chapter  Google Scholar 

  7. Beer, I., Ben-David, S., Eisner, C., Fisman, D., Gringauze, A., Rodeh, Y.: The temporal logic sugar. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 363–367. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44585-4_33

    Chapter  Google Scholar 

  8. Boyer, R.S., Moore, J.S.: A fast string searching algorithm. Commun. ACM 20(10), 762–772 (1977)

    Article  MATH  Google Scholar 

  9. Caesar, H., et al.: nuScenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11621–11631 (2020)

    Google Scholar 

  10. Del Bimbo, A., Vicario, E., Zingoni, D.: Symbolic description and visual querying of image sequences using spatio-temporal logic. IEEE Trans. Knowl. Data Eng. 7(4), 609–622 (1995)

    Article  Google Scholar 

  11. Dokhanchi, A., Amor, H.B., Deshmukh, J.V., Fainekos, G.: Evaluating perception systems for autonomous vehicles using quality temporal logic. In: Colombo, C., Leucker, M. (eds.) RV 2018. LNCS, vol. 11237, pp. 409–416. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03769-7_23

    Chapter  Google Scholar 

  12. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Conference on Robot Learning, pp. 1–16. PMLR (2017)

    Google Scholar 

  13. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88, 303–338 (2010)

    Article  Google Scholar 

  14. Fang, W., et al.: Computer vision applications in construction safety assurance. Autom. Constr. 110, 103013 (2020)

    Article  Google Scholar 

  15. Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 63–78 (2019)

    Google Scholar 

  16. Friedl, J.E.: Mastering Regular Expressions. O’Reilly Media Inc., Sebastopol (2006)

    Google Scholar 

  17. Gabelaia, D., Kontchakov, R., Kurucz, A., Wolter, F., Zakharyaschev, M.: Combining spatial and temporal logics: expressiveness vs. complexity. J. Artif. Intell. Res. 23, 167–243 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  18. Gallant, A.: regex-automata (2023). https://github.com/rust-lang/regex

  19. Hekmatnejad, M., Hoxha, B., Deshmukh, J.V., Yang, Y., Fainekos, G.: Formalizing and evaluating requirements of perception systems for automated vehicles using spatio-temporal perception logic. arXiv preprint arXiv:2206.14372 (2022)

  20. Janai, J., Güney, F., Behl, A., Geiger, A., et al.: Computer vision for autonomous vehicles: problems, datasets and state of the art. Found. Trends® Comput. Graph. Vis. 12(1–3), 1–308 (2020)

    Google Scholar 

  21. Kapach, K., Barnea, E., Mairon, R., Edan, Y., Ben-Shahar, O.: Computer vision for fruit harvesting robots-state of the art and challenges ahead. Int. J. Comput. Vis. Robot. 3(1–2), 4–34 (2012)

    Article  Google Scholar 

  22. Kesten, R., et al.: Woven planet perception dataset 2020 (2019). https://woven.toyota/en/perception-dataset

  23. Kim, E., et al.: Querying labelled data with scenario programs for sim-to-real validation. In: 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS), pp. 34–45. IEEE (2022)

    Google Scholar 

  24. Knuth, D.E., Morris, J.H., Jr., Pratt, V.R.: Fast pattern matching in strings. SIAM J. Comput. 6(2), 323–350 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  25. Kontchakov, R., Kurucz, A., Wolter, F., Zakharyaschev, M.: Spatial logic+ temporal logic=?. In: Handbook of Spatial Logics, pp. 497–564 (2007)

    Google Scholar 

  26. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  27. Lu, D., et al.: CAROM air-vehicle localization and traffic scene reconstruction from aerial videos. arXiv preprint arXiv:2306.00075 (2023)

  28. Matsakis, N.D., Klock, F.S.: The rust language. ACM SIGAda Ada Lett. 34(3), 103–104 (2014)

    Article  Google Scholar 

  29. Meng, T., Huang, J., Chew, C.M., Yang, D., Zhong, Z.: Configuration and design schemes of environmental sensing and vehicle computing systems for automated driving: a review. IEEE Sens. J. 23, 15305–15320 (2023)

    Article  Google Scholar 

  30. Pitropov, M., et al.: Canadian adverse driving conditions dataset. Int. J. Robot. Res. 40(4–5), 681–690 (2021)

    Article  Google Scholar 

  31. Pnueli, A.: The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science (SFCS 1977), pp. 46–57. IEEE (1977)

    Google Scholar 

  32. Quigley, M., et al.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software, p. 5. No. 3.2 in 3, Kobe, Japan (2009)

    Google Scholar 

  33. Roşu, G., Bensalem, S.: Allen linear (interval) temporal logic – translation to LTL and monitor synthesis. In: Ball, T., Jones, R.B. (eds.) CAV 2006. LNCS, vol. 4144, pp. 263–277. Springer, Heidelberg (2006). https://doi.org/10.1007/11817963_25

    Chapter  Google Scholar 

  34. Sun, P., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2446–2454 (2020)

    Google Scholar 

  35. Thomas, G., Gade, R., Moeslund, T.B., Carr, P., Hilton, A.: Computer vision for sports: current applications and research topics. Comput. Vis. Image Underst. 159, 3–18 (2017)

    Article  Google Scholar 

  36. Turtiainen, H., Costin, A., Lahtinen, T., Sintonen, L., Hamalainen, T.: Towards large-scale, automated, accurate detection of CCTV camera objects using computer vision. applications and implications for privacy, safety, and cybersecurity. arXiv preprint arXiv:2006.03870 (2020)

  37. Ward, T.M., et al.: Computer vision in surgery. Surgery 169(5), 1253–1256 (2021)

    Article  Google Scholar 

  38. Wolper, P.: Temporal logic can be more expressive. Inf. Control 56(1–2), 72–99 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  39. Xiao, P., et al.: PandaSet: advanced sensor suite dataset for autonomous driving. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3095–3101. IEEE (2021)

    Google Scholar 

  40. Xu, Z., Julius, A.A.: Census signal temporal logic inference for multiagent group behavior analysis. IEEE Trans. Autom. Sci. Eng. 15(1), 264–277 (2016)

    Article  Google Scholar 

  41. Yadav, P., Curry, E.: VidCEP: complex event processing framework to detect spatiotemporal patterns in video streams. In: 2019 IEEE International conference on big data (big data), pp. 2513–2522. IEEE (2019)

    Google Scholar 

  42. Yu, F., et al.: Bdd100k: a diverse driving dataset for heterogeneous multitask learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2636–2645 (2020)

    Google Scholar 

  43. Zhang, Y., Carballo, A., Yang, H., Takeda, K.: Perception and sensing for autonomous vehicles under adverse weather conditions: a survey. ISPRS J. Photogrammetry Remote Sens. 196, 146–177 (2023)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Toyota Motor North America, Research and Development, Ann Arbor, MI, USA

    Jacob Anderson, Georgios Fainekos, Bardh Hoxha, Hideki Okamoto & Danil Prokhorov

Authors
  1. Jacob Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Georgios Fainekos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bardh Hoxha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hideki Okamoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Danil Prokhorov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jacob Anderson .

Editor information

Editors and Affiliations

  1. Dept. of Informatics, Aristotle Univ. of Thessaloniki, Thessaloniki, Greece

    Panagiotis Katsaros

  2. University of Trieste, Trieste, Italy

    Laura Nenzi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Anderson, J., Fainekos, G., Hoxha, B., Okamoto, H., Prokhorov, D. (2023). Pattern Matching for Perception Streams. In: Katsaros, P., Nenzi, L. (eds) Runtime Verification. RV 2023. Lecture Notes in Computer Science, vol 14245. Springer, Cham. https://doi.org/10.1007/978-3-031-44267-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-44267-4_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-44266-7

  • Online ISBN: 978-3-031-44267-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation