Skip to main content
SpringerLink
Log in

Spatio-temporal querying recurrent multimedia databases using a semantic sequence state graph

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

We present an indexing method for spatiotemporal data: semantic sequence state graph (S3G). S3G maintains objects with their locations as states and events as transitions. The spatial information is maintained in states while the semantic events result in temporal ordering between the states. If the objects visit the same locations repeatedly, we call such databases as recurrent databases. Our querying interface supports queries based on spatio-temporal logic that includes operators such as ‘next’ and ‘eventually’. The interactive querying interface enables the user to build the query interactively and see the intermediate results of the query.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Algorithm 1.
Algorithm 2.
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Allen, J.F.: Maintaining knowledge about temporal intervals. In: Communications of the ACM. 26/11/1983. ACM Press (1983)

  2. Assfalg, J., Bertini, M., Colombo, C., Del Bimbo, A.: Semantic annotation of sports videos. IEEE Multimed. 9(2), 52–60 (2002)

    Article  Google Scholar 

  3. Athitsos, V., Papapetrou, P., Potamias, M., Kollios, G., Gunopulos, D.: Approximate embedding-based subsequence matching of time series. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. Vancouver, Canada, June 09–12, 2008, SIGMOD ’08 (2008)

  4. Aygun, R.S., Yazici, A.: Modeling and management of fuzzy information in multimedia database application. Multimed. Tools Appl. 24(1), 29–56 (2004)

    Article  Google Scholar 

  5. Berchtold, S., Keim, D.A., Kriegel, H.-P.: The X-tree: An Index Structure for High-Dimensional Data, Readings in Multimedia Computing and Networking, pp. 451–462. Morgan Kaufmann, San Francisco (2002)

    Google Scholar 

  6. Bozanis, P., Foteinos, P.: WeR-trees. Data Knowl. Eng. 63(2), 397–413 (2007)

    Article  Google Scholar 

  7. Böhm, C., Berchtold, S., Keim, D.A.: Searching in high-dimensional spaces: index structures for improving the performance of multimedia databases. ACM Comput. Surv. 33(3), 322–373 (2001)

    Article  Google Scholar 

  8. Chen, S., Ooi, B., Tan, K., Nascimento, M.A. (2008) ST2B-tree: a self-tunable spatio-temporal b+-tree index for moving objects. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. Vancouver, Canada, June 09–12, 2008, SIGMOD ’08

  9. Del Bimbo, A., Vicario, E., Zingoni, D.: Symbolic description and visual querying of image sequences using spatio-temporal logic. IEEE Trans. Knowl. Data Eng. 7(4), 609–622 (1995)

    Article  Google Scholar 

  10. Google Video (2011) Google Video. http://video.google.com

  11. Hadjieleftheriou, M., Kollios, G., Bakalov, P., Tsotras, V.J.: Complex spatio-temporal pattern queries. In: Proceedings of the 31st International Conference on Very Large Data Bases (VLDB ’05). VLDB Endowment, pp. 877–888 (2005)

  12. Hongeng, S., Nevatia, R., Bremond, F.: Video-based event recognition: activity representation and probabilistic recognition methods. Comput. Vis. Image Underst. 96(2), 129–162 (2004)

    Article  Google Scholar 

  13. Internet Archive (2011) Internet archive. http://www.archive.org

  14. Jain, V., Aygun, R.S.: SMART: A grammar-based semantic video modeling and representation. In: IEEE SouthEast Con 2008

  15. Jain, V., Aygün, R.S.: Spatio-temporal querying of video content using SQL for quantizable video databases. J. Multimed. 4(4), 215–227 (2009)

    Google Scholar 

  16. Jensen, C.S., Lin, D., Ooi, B.C.: Query and update efficient B+-tree based indexing of moving objects. In: VLDB. pp. 768–779 (2004)

  17. Jensen, C.S., Snodgrass, R.T.: Temporal data management. IEEE Trans. Knowl. Data Eng. 11(1), 36–44 (1999)

    Article  Google Scholar 

  18. Kim, C., Vasudev, B.: Spatiotemporal sequence matching for efficient video copy detection. IEEE Trans. Circuits Syst. Video Technol. 15(1), 127–132 (2005)

    Article  Google Scholar 

  19. Koprulu, M., Cicekli, N.K., Yazici, A.: Spatio-temporal querying in video databases. Inf. Sci. Inf. Comput. Sci. 160(1–4), 131–152 (2004)

    Google Scholar 

  20. Lagogiannis, G., Lorentzos, N., Sioutas, S., Theodoridis, E.: A time efficient indexing scheme for complex spatiotemporal retrieval. SIGMOD Rec. 38(3), 11–16 (2010)

    Article  Google Scholar 

  21. Lay, J.A., Guan, L.: Semantic retrieval of multimedia by concept languages: treating semantic concepts like words. Signal Process. Mag. IEEE 23(2), 115–123 (2006)

    Article  Google Scholar 

  22. Lee, M., Yoon, H., Kim, Y.J., Lee, Y.: SMILE tree: a stream data multi-query indexing technique with level-dimension nodes and extended-range nodes. In: Proceedings of the 2nd International Conference on Ubiquitous Information Management and Communication. Suwon, Korea, January 31–February 01, 2008, ICUIMC ‘08

  23. Lejsek, H., Ásmundsson, F.H., Jónsson, B.Þ., Amsaleg, L.: NV-Tree: an efficient disk-based index for approximate search in very large high-dimensional collections. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 869–883 (2009)

    Article  Google Scholar 

  24. Li, J.Z., Ozsu, M.T., Szafron, D.: Modeling of moving objects in a video database. In: International Conference on Multimedia Computing and Systems, pp. 336 (1997)

  25. Lin, D., Jensen, C.S., Ooi, B.C., Šaltenis, S.: Efficient indexing of the historical, present, and future positions of moving objects. In: Proceedings of the 6th International Conference on Mobile Data Management, Ayia Napa, Cyprus, May 09–13, 2005, MDM ‘05. ACM, New York, NY, pp. 59–66 (2005)

  26. Min, J.S., Kim, D.H., Ryu, K.H.: A spatiotemporal data and indexing, electrical and electronic technology, 2001. TENCON. In: Proceedings of IEEE Region 10th International Conference on, vol. 1, pp. 110–113 (2001)

  27. Naik, M., Jain, V., Aygun, R.S.: S3G: A semantic sequence state graph for indexing spatio-temporal data—a tennis video database application. In: ICSC, 2008 IEEE International Conference on Semantic Computing, pp. 66–73 (2008)

  28. Park, D.-J., Heu, S., Kim, H.-J.: The RS-tree: an efficient data structure for distance browsing queries. Inf. Process. Lett. 80(4), 195–203 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  29. Patel, J.M., Chen, Y., Chakka, V.P.: STRIPES: an efficient index for predicted trajectories. In: Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, Paris, France, June 13–18, 2004. SIGMOD ’04. ACM, New York, NY, pp. 635–646 (2004)

  30. Pissinou, N., Radev, I., Makki, K., Campbell, W.J.: Spatio-temporal composition of video objects: representation and querying in video database systems. IEEE Trans. Knowl. Data Eng. 13(16), 1033–1040 (2001)

    Article  Google Scholar 

  31. Ren, W., Singh, S., Singh, M., Zhu, Y.S.: State-of-the-art on spatio-temporal information-based video retrieval. Pattern Recognit. 42(2), 267–282 (2009)

    Article  MATH  Google Scholar 

  32. Šaltenis, S., Jensen, C.S., Leutenegger, S.T., Lopez, M.A.: Indexing the positions of continuously moving objects. SIGMOD Rec. 29(2), 331–342 (2000)

    Article  Google Scholar 

  33. Saltenis, S., Jensen, C.S.: Indexing of moving objects for location-based services. ICDE 2002, 463–472 (2002)

    Google Scholar 

  34. Salzberg, B., Tsotras, V.J.: Comparison of access methods for time-evolving data. ACM Comput. Surv. 31(2), 158–221 (1999)

    Article  Google Scholar 

  35. Tao, Y., Papadias, D., Sun, J.: The TPR*-tree: an optimized spatio-temporal access method for predictive queries. In: Proceedings of the 29th International Conference on Very Large Data Bases (VLDB), Berlin, Germany, 2003, pp. 790–801 (2003)

  36. Valle, E., Cord, M., Philipp-Foliguet, S.: High-dimensional descriptor indexing for large multimedia databases. In: Proceeding of the 17th ACM Conference on Information and Knowledge Management. CIKM '08. ACM (2008)

  37. Vardi, M.Y. Branching vs. linear time: final showdown. In: Proceedings of the 7th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’01), 2031 (2001), pp. 1–22

  38. Wattamwar, S.S., Ghosh, H.: Spatio-temporal query for multimedia databases. In: Proceeding of the 2nd ACM Workshop on Multimedia Semantics. Vancouver, British Columbia, Canada, October 31–31, 2008, MS ’08. pp. 48–55, ACM, New York, NY

  39. Ye, H., Luo, H., Song, K., Xiang, H., Chen, J.: Indexing moving objects based on 2n index tree. In: Proceedings of the 6th Conference on 6th WSEAS International Conference on Artificial intelligence, Knowledge Engineering and Data Bases, vol. 6, Corfu Island, Greece, February 16–19, 2007

  40. Yiu, M.L., Tao, Y., Mamoulis, N.: The Bdual-tree: indexing moving objects by space filling curves in the dual space. VLDB J. 17(3), 379–400 (2008)

    Article  Google Scholar 

  41. YouTube (2011) YouTube—Broadcast Yourself. http://www.youtube.com

Download references

Acknowledgments

We thank Vineetha Bettaiah for improving the code for searching states, incorporating rank of clips, and editing of the paper.

Author information

Authors and Affiliations

  1. Computer Science Department, University of Alabama in Huntsville, Huntsville, AL, USA

    M. M. Naik, M. Sigdel & R. S. Aygun

Authors
  1. M. M. Naik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Sigdel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. S. Aygun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to R. S. Aygun.

Additional information

Communicated by B. Prabhakaran.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 9 kb)

Appendix

Appendix

In Appendix, we provide information about our user interface for querying.

1.1 A user interface

In this section, we provide some snapshots from our user interface.

1.1.1 Appendix 1: Building S3G

Figure 20 displays the user interface where SMART strings are read from the database. When “Proceed to State Conversion” button is clicked, the S3G is built.

Fig. 20
figure 20

Interface showing the strings obtained from GSMART database

Full size image

1.1.2 Appendix 2: Displaying states

The interface is prepared based on design in Fig. 21 where player1, player2, and the ball images are placed on all possible locations on the tennis court image. All these object images are hidden except the ones that represent the location values for the given state when displaying a state.

Fig. 21
figure 21

Player1, player2 and ball images on all possible locations on the tennis court

Full size image

1.1.3 Appendix 3: Location selector interface

Figure 22 shows location selector interface filled with location values for a desired state. In this example, player1 location has a location as 7, player2 has a location as 10, and the ball has a location as 4. To search for a particular state with corresponding location values in the ‘Location Selector’ interface, the search option in Fig. 15 is selected. To retrieve states based on an event, the user needs to follow the links to retrieve the corresponding links. For example, if the user wants to retrieve the clips that result after an event such as forehand shot by player1, the transition for this event followed and the clips from the relevant states are retrieved.

Fig. 22
figure 22

Location selector interface state with entries filled for player1, player2 and ball locations

Full size image

1.1.4 Appendix 4: Viewing clips

A particular clip can be selected to view from the list of clips given for that state (Fig. 23). The clip will be displayed after pressing the view button. If the selected state does not exist in the S3G, a message will be displayed saying that the specified state does not exist. Figure 23 also shows player1, player2, and ball locations.

Fig. 23
figure 23

Interface showing details of 40710

Full size image

1.1.5 Appendix 5: Interface for querying next states

A state can be chosen by selecting the player1, the player2 and the ball locations in the tennis court and then corresponding ‘next’ states in the S3G can be obtained (Fig. 24). The aim is to obtain the clips having the current state and the selected next state.

Fig. 24
figure 24

Location selector interface with next ‘state’

Full size image

Figure 25 shows an interface showing the possible ‘next’ states in the S3G for the transition, when player2 hits a forehand shot. A ‘next’ state can be selected by choosing the ‘Select State’ options button associated with that state. If the next state does not exist in the S3G for a transition then a message will be displayed stating that there is no next state for that transition. Once a current state and a next state are selected, the relevant clips can then be viewed (Fig. 26).

Fig. 25
figure 25

Interface showing ‘next’ states when player2 hits a forehand shot

Full size image
Fig. 26
figure 26

Interface showing the relevant clip having the two states (current and ‘next’ states on top right corner)

Full size image

Only the next states that have common clips with the relevant clip set (clips common to all selected states) are displayed to select. There may be a case where there can be a next state existing for a state in the S3G but the next state has no common clip with all previously selected states. In such a case, a message stating that “the next state exists in the S3G, but no common clip exists” is displayed.

1.1.6 Appendix 6: Interface for querying eventually

The result, whether the ‘eventually’ link exists between the chosen initial and final states in the S3G or not, is obtained by selecting the “Result” option as in Fig. 27. Figure 27 shows that an ‘eventually’ link exists for the chosen initial and final states in the S3G.

Fig. 27
figure 27

Result of ‘eventually’ check

Full size image

Rights and permissions

Reprints and permissions

About this article

Cite this article

Naik, M.M., Sigdel, M. & Aygun, R.S. Spatio-temporal querying recurrent multimedia databases using a semantic sequence state graph. Multimedia Systems 18, 263–281 (2012). https://doi.org/10.1007/s00530-011-0255-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-011-0255-8

Keywords

Search

Navigation