Skip to main content
SpringerLink
Log in

Maximizing a deep submodular function optimization with a weighted MAX-SAT problem for trajectory clustering and motion segmentation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Computer vision models are commonly defined for maximum constrained submodular functions lies at the core of low-level and high-level models. In such that, the pixels that are to be grouped or segmenting moving object remains a challenging task. This paper proposes a joint framework for maximizing submodular energy subject to a matroid constraint using Deep Submodular Function (DSF) optimization approximately to solve the weighted MAX-SAT (Maximum Satisfiability) problem and a new trajectory clustering method called Simple Slice Linear clustering (SSLIC) and motion cue method for trajectory clustering and motion segmentation. In this objective function, the illustrative trajectories of a small number are selected automatically by deep submodular maximization. Although, the exploitation of monotone and submodular properties are further maximized and the complexity is reduced by a continuous greedy algorithm. The bound guarantees a fully sliced curve of (1- S/e) to (1–1/e) with less running time. Lastly, the motion is segmented by the motion cue method to accurately differentiate the set of frames for different scenes. Experiments on the Hopkins 155, Berkley Motion Segmentation (BMS) and FBMS-59 datasets display the trajectory clustering and motion segmentation result over its superior performance with respect to 14 quality evaluation metrics. Hence the simulation result shows that the proposed joint framework attains better performance than existing methods on trajectory clustering and motion segmentation task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Buchbinder N, Feldman M, Seffi J, Schwartz R (2015) A tight linear time (1/2)-approximation for unconstrained submodular maximization. SIAM J Comput 44(5):1384–1402

    Article  MathSciNet  Google Scholar 

  2. Calinescu G, Chekuri C, Pal M, Vondrák J (2011) Maximizing a monotone submodular function subject to a matroid constraint. SIAM J Comput 40(6):1740–1766

    Article  MathSciNet  Google Scholar 

  3. Lin H, Bilmes J (2011) A class of submodular functions for document summarization. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 510-520)

  4. Narasimhan M, Jojic N, Bilmes JA (2006) Q-clustering. In: Advances in Neural Information Processing Systems (pp. 979-986)

  5. Liu D, Zhou KS, Bernhardt D, Comaniciu D (2010) Search strategies for multiple landmark detection by submodular maximization. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 2831-2838). IEEE

  6. Zhu F, Jiang Z, Shao L (2014) Submodular object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2457-2464)

  7. Xu J, Mukherjee L, Li Y, Warner J, Rehg JM, Singh V (2015) Gaze-enabled egocentric video summarization via constrained submodular maximization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2235-2244)

  8. Gygli M, Grabner H, Van Gool L (2015) Video summarization by learning submodular mixtures of objectives. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3090-3098)

  9. Pokutta S, Singh M, Torrico A (2018) Efficient algorithms for robust submodular maximization under matroid constraints. arXiv preprint arXiv:1807.09405

  10. de Veciana G, Hashemi A, Vikalo H (2019) Stochastic-greedy++: closing the optimality gap in exact weak submodular maximization. arXiv preprint arXiv:1907.09064

  11. Mirzasoleiman B, Badanidiyuru A, Karbasi A (2016) Fast constrained submodular maximization: personalized data summarization. In: ICML (pp. 1358-1367)

  12. Chakrabarti A, Kale S (2015) Submodular maximization meets streaming: Matchings, matroids, and more. Math Program 154(1–2):225–247

    Article  MathSciNet  Google Scholar 

  13. Badanidiyuru A, Mirzasoleiman B, Karbasi A, Krause A (2014) Streaming submodular maximization: massive data summarization on the fly. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 671-680)

  14. Bateni M, Esfandiari H, Mirrokni V (2018) Optimal distributed submodular optimization via sketching. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1138-1147)

  15. Huang CC, Kakimura N, Yoshida Y (2017) Streaming algorithms for maximizing monotone submodular functions under a knapsack constraint. In: Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM 2017). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik

  16. Norouzi-Fard A, Tarnawski J, Mitrović S, Zandieh A, Mousavifar A, Svensson O (2018) Beyond $1/2$-Approximation for Submodular Maximization on Massive Data Streams. arXiv preprint arXiv:1808.01842

  17. Agrawal S, Shadravan M, Stein C (2018) Submodular secretary problem with shortlists. arXiv preprint arXiv:1809.05082

  18. Kumar R, Moseley B, Vassilvitskii S, Vattani A (2015) Fast greedy algorithms in mapreduce and streaming. ACM Trans Parallel Comput (TOPC) 2(3):1–22

    Article  Google Scholar 

  19. Yu Q, Xu EL, Cui S (2016) Streaming algorithms for news and scientific literature recommendation: submodular maximization with a d-knapsack constraint. arXiv preprint arXiv:1603.05614

  20. Iyer R, Bilmes J (2012) Algorithms for approximate minimization of the difference between submodular functions, with applications. arXiv preprint arXiv:1207.0560

  21. Lin H, Bilme J (2010) Multi-document summarization via budgeted maximization of submodular functions. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (pp. 912–920)

  22. Buchbinder N, Feldman M, Schwartz R (2017) Comparing apples and oranges: query trade-off in submodular maximization. Math Oper Res 42(2):308–329

    Article  MathSciNet  Google Scholar 

  23. Kuhnle A (2019) Interlaced greedy algorithm for maximization of submodular functions in nearly linear time. Adv Neural Inf Process Syst 2374-2384

  24. Brox T, Malik J (2010) Object segmentation by long term analysis of point trajectories. In European conference on computer vision (pp. 282-295). Springer, Berlin, Heidelberg

  25. Shi F, Zhou Z, Xiao J, Wu W (2013) Robust trajectory clustering for motion segmentation. In: Proceedings of the IEEE International Conference on Computer Vision (pp. 3088-3095)

  26. Wang G (2016) Robust structure from motion in the presence of outliers and missing data. arXiv preprint arXiv:1609.02638

  27. Zhang Y, Luo B, Zhang L (2017) Permutation preference based alternate sampling and clustering for motion segmentation. IEEE Signal Processing Letters 25(3):432–436

    Article  Google Scholar 

  28. Liu L, Zhou F, Bai XR, Tao ML, Zhang ZJ (2016) Joint cross-range scaling and 3D geometry reconstruction of ISAR targets based on factorization method. IEEE Trans Image Process 25(4):1740–1750

    Article  MathSciNet  Google Scholar 

  29. Zhang CC, Liu ZL (2017) Prior-free dependent motion segmentation using Helmholtz-Hodge decomposition based object-motion oriented map. J Comput Sci Technol 32(3):520–535

    Article  MathSciNet  Google Scholar 

  30. Chen W, Zhang E, Zhang Z (2016) A Laplacian structured representation model in subspace clustering for enhanced motion segmentation. Neurocomputing. 208:174–182

    Article  Google Scholar 

  31. Chen L, Shen J, Wang W, Ni B (2015) Video object segmentation via dense trajectories. IEEE Trans Multimedia 17(12):2225–2234

    Article  Google Scholar 

  32. Lee CM, Cheong LF (2017) Minimal basis subspace representation: a unified framework for rigid and non-rigid motion segmentation. Int J Comput Vis 121(2):209–233

    Article  Google Scholar 

  33. Sarpatwar KK, Schieber B, Shachnai H (2019) Constrained submodular maximization via greedy local search. Oper Res Lett 47(1):1–6

    Article  MathSciNet  Google Scholar 

  34. Robey A, Adibi A, Schlotfeldt B, Pappas GJ, Hassani H (2019) Optimal algorithms for submodular maximization with distributed constraints. arXiv preprint arXiv:1909.13676

  35. Sakaue S (2019) Approximation guarantees of stochastic greedy algorithms for non-monotone submodular maximization with a size constraint. arXiv preprint arXiv:1908.06242

  36. Qian C, Yu Y, Tang K, Yao X, Zhou ZH (2019) Maximizing submodular or monotone approximately submodular functions by multi-objective evolutionary algorithms. Artif Intell 275:279–294

    Article  MathSciNet  Google Scholar 

  37. Adouani Y, Jarboui B, Masmoudi M (2019) A matheuristic for the 0–1 generalized quadratic multiple knapsack problem. Optim Lett 1-22

  38. Jiang Y, Wang Y, Xu D, Yang R, Zhang Y (2019) Streaming algorithm for maximizing a monotone non-submodular function under d-knapsack constraint. Optim Lett 1–14

  39. Shen J, Liang Z, Liu J, Sun H, Shao L, Tao D (2018) Multiobject tracking by submodular optimization. IEEE Trans Cybern 49(6):1990–2001

    Article  Google Scholar 

  40. Shen J, Peng J, Shao L (2018) Submodular trajectories for better motion segmentation in videos. IEEE Trans Image Process 27(6):2688–2700

    Article  MathSciNet  Google Scholar 

  41. Zhang Y, Chen X, Li J, Teng W, Song H (2018) Exploring weakly labeled images for video object segmentation with submodular proposal selection. IEEE Trans Image Process 27(9):4245–4259

    Article  MathSciNet  Google Scholar 

  42. Liu F, Lin G, Qiao R, Shen C (2018) Structured learning of tree potentials in CRF for image segmentation. IEEE Trans Neural Netw Learn Syst 29(6):2631–2637

    Article  MathSciNet  Google Scholar 

  43. Verter V (2011) Uncapacitated and capacitated facility location problems. In: Foundations of location analysis (pp. 25–37). New York: Springer

  44. Wang Y, Liu Y, Blasch E, Ling H (2017) Simultaneous trajectory association and clustering for motion segmentation. IEEE Signal Process Lett 25(1):145–149

    Article  Google Scholar 

  45. Zhang Y, Lu H, Zhang L, Ruan X (2016) Combining motion and appearance cues for anomaly detection. Pattern Recogn 51:443–452

    Article  Google Scholar 

  46. Mesbahi SC, Mahraz MA, Riffi J, Tairi H (2017) Head gesture recognition using optical flow based background subtraction. In: Proceedings of the Mediterranean Symposium on Smart City Applications (pp. 200-211). Cham: Springer

  47. Tron R, Vidal R (2007) A benchmark for the comparison of 3-d motion segmentation algorithms. In: 2007 IEEE conference on computer vision and pattern recognition (pp. 1-8). IEEE

  48. Ochs P, Malik J, Brox T (2013) Segmentation of moving objects by long term video analysis. IEEE Trans Pattern Anal Mach Intell 36(6):1187–1200

    Article  Google Scholar 

  49. Jian YD, Chen CS (2010) Two-view motion segmentation with model selection and outlier removal by ransac-enhanced dirichlet process mixture models. Int J Comput Vis 88(3):489–501

    Article  MathSciNet  Google Scholar 

  50. Elhamifar E, Vidal R (2013) Sparse subspace clustering: algorithm, theory, and applications. IEEE Trans Pattern Anal Mach Intell 35(11):2765–2781

    Article  Google Scholar 

  51. Lai T, Wang H, Yan Y, Chin TJ, Zhao WL (2016) Motion segmentation via a sparsity constraint. IEEE Trans Intell Transp Syst 18(4):973–983

    Article  Google Scholar 

  52. Trobo IP, Díaz VG, Espada JP, Crespo RG, Moreno-Ger P (2019) Rapid modeling of human-defined AI behavior patterns in games. J Ambient Intell Humaniz Comput 10(7):2683–2692

    Article  Google Scholar 

  53. Wang X, Dai Z (2019) Vision-based vehicle behavior monitoring method using a novel clustering algorithm. J Ambient Intell Humaniz Comput 28:1–9

    Google Scholar 

  54. Chin TJ, Suter D, Wang H (2010) Multi-structure model selection via kernel optimisation. In: 2010 IEEE computer society conference on computer vision and pattern recognition (pp. 3586-3593). IEEE

  55. Liu G, Lin Z, Yan S, Sun J, Yu Y, Ma Y (2012) Robust recovery of subspace structures by low-rank representation. IEEE Trans Pattern Anal Mach Intell 35(1):171–184

    Article  Google Scholar 

  56. Hu H, Feng J, Zhou J (2014) Exploiting unsupervised and supervised constraints for subspace clustering. IEEE Trans Pattern Anal Mach Intell 37(8):1542–1557

    Article  Google Scholar 

  57. Russell C, Yu R, Agapito L (2014) Video pop-up: monocular 3d reconstruction of dynamic scenes. In: European conference on computer vision (pp. 583-598). Cham: Springer

  58. Abdelwahab MA, Abdelwahab MM, Uchiyama H, Shimada A, Taniguchi RI. (2016) Video object segmentation based on superpixel trajectories. In: International conference on image analysis and recognition (pp. 191-197). Cham: Springer

  59. Keuper M, Andres B, Brox T (2017) Motion trajectory segmentation via minimum cost multicuts. In: Proceedings of the IEEE international conference on computer vision (pp. 3271–3279)

  60. Keuper M (2017) Higher-order minimum cost lifted multicuts for motion segmentation. In: Proceedings of the IEEE International Conference on Computer Vision (pp. 4242–4250)

  61. Keuper M, Tang S, Andres B, Brox T, Schiele B (2018) Motion segmentation & multiple object tracking by correlation co-clustering. IEEE Trans Pattern Anal Mach Intell 42(1):140–153

    Article  Google Scholar 

  62. Dong W, Wu XJ, Kittler J, Yin HF (2019) Sparse subspace clustering via nonconvex approximation. Pattern Anal Applic 22(1):165–176

    Article  MathSciNet  Google Scholar 

  63. Gong YJ, Zhou Y (2017) Differential evolutionary superpixel segmentation. IEEE Trans Image Process 27(3):1390–1404

    Article  MathSciNet  Google Scholar 

  64. Chai D (2019) SQL: Superpixels via quaternary labeling. Pattern Recogn 92:52–63

    Article  Google Scholar 

  65. Xu L, Luo B, Pei Z (2018) Weak boundary preserved superpixel segmentation based on directed graph clustering. Signal Process Image Commun 65:231–239

    Article  Google Scholar 

Download references

Funding

There is no funding for this study.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, Visveswaraya Technological University, Bengaluru, Karnataka, India

    Kiran Kumar Chandriah

  2. Department of Mechanical Engineering, National Institute of Engineering (NIE), Mysore, India

    Raghavendra V Naraganahalli

Authors
  1. Kiran Kumar Chandriah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raghavendra V Naraganahalli
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors have participated in writing the manuscript and have revised the final version. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Kiran Kumar Chandriah.

Ethics declarations

Ethical approval

This article does not contain any studies with human participants and/or animals performed by any of the authors.

Informed consent

There is no informed consent for this study.

Conflict of interest

Authors declares that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chandriah, K.K., Naraganahalli, R.V. Maximizing a deep submodular function optimization with a weighted MAX-SAT problem for trajectory clustering and motion segmentation. Appl Intell 51, 8192–8211 (2021). https://doi.org/10.1007/s10489-021-02276-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02276-8

Keywords

Search

Navigation