Skip to main content
Springer Nature Link
Log in

Multi-sensor based object tracking using enhanced particle swarm optimized multi-cue granular fusion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In the discipline of computer vision, object tracking is one of the progressive and prominent areas of research with its application in the field of medical imaging, vehicle navigation, surveillance etc. Many of the proposed object tracking algorithms has shown success in the recent years. In this manuscript, we introduced a novel approach for object tracking that can develop an efficient framework of various features from different sensors. If we contemplate a RGB (red–green–blue) image it has better distinction of colors from human eye standpoint but is degraded by shadows and noise caused by illumination. Unlike RGB images thermal images are less receptive to such type of noise factors yet environmental condition can alter its distinction. To overcome this distinction issue of the two sensors a fusion of these two modalities is introduced, considering their interdependent advantages. This proposed technique is focused at enhancing the information collected from the fusion of visible imaging and thermal imaging sensors. It can also be implemented if the number of sensor are increased which in turn increases the number of features. With the use of features from two different sensors the proposed scheme utilizes the six information cues for the estimation of single output. The EPSO (enhanced particle swarm optimization) based particle filtering was adjusted with the concept of using multi-cue granular computing to weigh the particles and estimate the ultimate tracking result. After conducting attribute weight adaptation, the same approach is expanded to produce source-level fusion. The experimental performance of the method has been demonstrated on publicly available standard video sequences. After comparing it against state-of-the-art approaches, the findings show that it outperforms the trackers mentioned in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available in the Ohio State University repository, BRS, CDS, SPS, TRS. These are all publicly available.

References

  1. Arulampalam MS, Maskell S, Gordon N, Clapp T (2002) A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans Signal Process 50(2):174–188

    Google Scholar 

  2. Bebis SSG, Gyaourova A, Pavlidis I (2006) Face Recoginition by Fusion Thermal Infrared and Visible Imagery. Image Vis Comput 24(7):727–742

    MATH  Google Scholar 

  3. Bertinetto L, Valmadre J, Henriques JF, Vedaldi A, Torr PH (2016) Fully-convolutional siamese networks for object tracking In European conference on computer vision (pp. 850-865). Springer, Cham

    Google Scholar 

  4. Bewley A, Ge Z, Ott L, Ramos F, Upcroft B (2016) Simple online and realtime tracking. In: 2016 IEEE international conference on image processing (ICIP). IEEE, pp 3464–3468

  5. Bhattacharyya A (1943) On a measure of divergence between two statistical populations defined by their probability distributions. Bull Calcutta Math Soc 35:99–109

    MathSciNet  MATH  Google Scholar 

  6. Brasnett P, Mihaylova L, Bull D, Canagarajah N (2007) Sequential Monte Carlo tracking by fusing multiple cues in video sequences. Image Vis Comput 25(8):1217–1227

    Google Scholar 

  7. Bristol Eden Project Multi-Sensor Data Set (2019) https://www.cis.rit.edu/pelz/scanpaths/data/bristol-eden.htm/. Accessed 29 Dec 2019

  8. Cielniak G, Duckett T, Lilienthal AJ (2007) “Improved data association and occlusion handling for vision-based people tracking by mobile robots”, in. IEEE/RSJ Int Conf Intell Robots Syst 2007:3436–3441

    Google Scholar 

  9. Davis JW, Sharma V (2007) Background-subtraction using contour-based fusion of thermal and visible imagery. Comput Vis Image Underst 106(2–3):162–182

    Google Scholar 

  10. Eberhart R, Kennedy J (1995) Particle swarm optimization. Proceed IEEE Int Conf Neural Netw 4:39–43

    Google Scholar 

  11. Hanif M, Ali U (2006) Optimized visual and thermal image fusion for efficient face recognition. In: 2006 9th International Conference on Information Fusion. IEEE, pp 1–6

  12. Henriques JF, Caseiro R, Martins P, Batista J (2014) High-speed tracking with kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596

    Google Scholar 

  13. Heo J, Kong SG, Abidi BR, Abidi MA (2004) Fusion of visual and thermal signatures with eyeglass removal for robust face recognition. In: 2004 Conference on Computer Vision and Pattern Recognition Workshop. IEEE, pp 122–122

  14. Hsu C, Dai GT (2012) Multiple object tracking using particle swarm optimization. World Acad Sci Eng Technol 68:41–44

    Google Scholar 

  15. Isard M, Blake A (1998) Condensation—conditional density propagation for visual tracking. Int J Comput Vision 29(1):5–28

    Google Scholar 

  16. John V, Tsuchizawa S, Liu Z, Mita S (2017) Fusion of thermal and visible cameras for the application of pedestrian detection. SIViP 11(3):517–524

    Google Scholar 

  17. Kapoor R, Dhamija A (2012) Fast tracking algorithm using modified potential function. IET Comput Vision 6(2):111–120

    MathSciNet  Google Scholar 

  18. Kapoor R, Rohilla R (2015) Modified foreground segmentation for object tracking using wavelets in a tensor framework. Int J Electron 102(9):1560–1582

    Google Scholar 

  19. Kennedy J (2006) Swarm intelligence. Handbook of nature-inspired and innovative computing. Springer, Berlin

  20. Kong SG, Heo J, Boughorbel F, Zheng Y, Abidi BR, Koschan A, Abidi MA (2007) Multiscale fusion of visible and thermal IR images for illumination-invariant face recognition. Int J Comput Vision 71(2):215–233

    Google Scholar 

  21. Kumar DA, Meher SK, Kumari KP (2019) Fusion of progressive granular neural networks for pattern classification. Soft Comput 23(12):4051–4064

    Google Scholar 

  22. Kumar A, Walia GS, Sharma K (2020) Recent trends in multicue based visual tracking: A review. Expert Syst Appl 162:113711

    Google Scholar 

  23. Li T, Bolic M, Djuric PM (2015) Resampling methods for particle filtering: classification, implementation, and strategies. IEEE Signal Process Mag 32(3):70–86

    Google Scholar 

  24. Li C, Wu X, Zhao N, Cao X, Tang J (2017) Fusing two-stream convolutional neural networks for RGB-T object tracking. Neurocomputing 281:78–85

    Google Scholar 

  25. Liu M, Jin CB, Yang B, Cui X, Kim H (2019) Online multiple object tracking using confidence score-based appearance model learning and hierarchical data association. IET Comput Vision 13(3):312–318

    Google Scholar 

  26. Ning Y, Peng Z, Dai Y, Bi D, Wang J (2019) Enhanced particle swarm optimization with multi-swarm and multi-velocity for optimizing high-dimensional problems. Appl Intell 49(2):335–351

    Google Scholar 

  27. Nummiaro K, Koller-Meier E, Van Gool L (2003) An adaptive color-based particle filter. Image Vis Comput 21(1):99–110

    MATH  Google Scholar 

  28. Palmerini GB (2014) “Combining thermal and visual imaging in spacecraft proximity operations.” Int Conf Control Autom Robot Vis (ICARCV) 2014:383–388

    Google Scholar 

  29. Pedrycz W (2002) Relational and directional aspects in the construction of information granules. IEEE Trans Syst Man Cybern-Part A: Syst Hum 32(5):605–614

    Google Scholar 

  30. Pedrycz W, Bargiela A (2002) Granular clustering: a granular signature of data. IEEE Trans Syst Man, Cybern, Part B (Cybern) 32(2):212–224

    Google Scholar 

  31. Qian Y, Liang J, Dang C (2009) Knowledge structure, knowledge granulation and knowledge distance in a knowledge base. Int J Approx Reason 50(1):174–188

    MathSciNet  MATH  Google Scholar 

  32. Qian Y, Liang J, Wei-zhi ZW, Dang C (2011) Information granularity in fuzzy binary GrC model. IEEE Trans Fuzzy Syst 19(2):253–264

    Google Scholar 

  33. Qian Y, Cheng H, Wang J, Liang J, Pedrycz W, Dang C (2017) Grouping granular structures in human granulation intelligence. Inf Sci 382:150–169

    Google Scholar 

  34. Qian X, Han L, Wang Y, Ding M (2018) Deep learning assisted robust visual tracking with adaptive particle filtering. Signal Process Image Commun 60:183–192

    Google Scholar 

  35. Rohilla R, Sikri V, Kapoor R (2016) Spider monkey optimisation assisted particle filter for robust object tracking. IET Comput Vision 11(3):207–219

    Google Scholar 

  36. Sardari F, Moghaddam ME (2017) A hybrid occlusion free object tracking method using particle filter and modified galaxy based search meta-heuristic algorithm. Appl Soft Comput 50:280–299

    Google Scholar 

  37. Singh S, Kapoor R, Khosla A (2018) Cross-domain usage in real-time video-based tracking. In: Handbook of Research on Advanced Concepts in Real-Time Image and Video Processing. IGI Global, pp 105–129

  38. Singh S, Khosla A, Kapoor R (2019) Object Tracking with a Novel Visual-Thermal Sensor Fusion Method in Template Matching. Int J Image, Graph Signal Process 9(7):39

    Google Scholar 

  39. Smeulders AW, Chu DM, Cucchiara R, Calderara S, Dehghan A, Shah M (2013) Visual tracking: An experimental survey. IEEE Trans Pattern Anal Mach Intell 36(7):1442–1468

    Google Scholar 

  40. St-Laurent L, Prévost D, Maldague X (2010) “Fast and accurate calibration-based thermal/colour sensors registration,” Quant Infrared Thermography

  41. Talha M, Stolkin R (2014) Particle Filter Tracking of Camouflaged Targets by Adaptive Fusion of Thermal and Visible Spectra Camera Data. IEEE Sens J 14(1):159–166

    Google Scholar 

  42. Tong Y, Liu L, Zhao M, Chen J, Li H (2016) Adaptive fusion algorithm of heterogeneous sensor networks under different illumination conditions. Signal Process 126:149–158

    Google Scholar 

  43. Ursine W, Calado F, Teixeira G, Diniz H, Silvino S, De Andrade R (2012) Thermal/visible autonomous stereo visio system calibration methodology for non-controlled environments. In: 11th International Conference on Quantitative Infrared Thermography, pp 1–10

  44. Vidas S, Lakemond R, Denman S, Fookes C, Sridharan S, Wark T (2012) A mask-based approach for the geometric calibration of thermal-infrared cameras. IEEE Trans Instrum Meas 61(6):1625–1635

    Google Scholar 

  45. Video Analytics Dataset (2021) https://www.ino.ca/en/video-analytics-dataset/. Accessed 12 Mar 2021

  46. Walia GS, Kapoor R (2014) Intelligent video target tracking using an evolutionary particle filter based upon improved cuckoo search. Expert Syst Appl 41(14):6315–6326

    Google Scholar 

  47. Walia GS, Kapoor R (2016) Robust object tracking based upon adaptive multi-cue integration for video surveillance. Multimed Tools Appl 75(23):15821–15847

    Google Scholar 

  48. Walia GS, Kapoor R (2016) Recent advances on multicue object tracking: a survey. Artif Intell Rev 46(1):1–39

    Google Scholar 

  49. Walia GS, Kapoor R (2018) Online Object Tracking via Novel Adaptive Multicue Based Particle Filter Framework for Video Surveillance. Int J Artif Intell Tools 27(06):1850023

    Google Scholar 

  50. Walia GS, Kumar A, Saxena A, Sharma K, Singh K (2020) Robust object tracking with crow search optimized multi-cue particle filter. Pattern Anal Appl 23(3):1439–1455

    MathSciNet  Google Scholar 

  51. Wang YD, Wu JK, Kassim AA (2007) Adaptive particle filter for data fusion of multiple cameras. J VLSI Signal Process Syst Signal, Image, Vid Technol 49(3):363–376

    Google Scholar 

  52. Wilhelm T, Böhme HJ, Gross HM (2004) A multi-modal system for tracking and analyzing faces on a mobile robot. Robot Auton Syst 48(1):31–40

    Google Scholar 

  53. Wojke N, Bewley A, Paulus D (2017) Simple online and realtime tracking with a deep association metric. In: 2017 IEEE international conference on image processing (ICIP). IEEE, pp 3645–3649

  54. Xiao J, Stolkin R, Oussalah M, Leonardis A (2016) Continuously adaptive data fusion and model relearning for particle filter tracking with multiple features. IEEE Sens J 16(8):2639–2649

    Google Scholar 

  55. Xiao G, Yun X, Wu J (2016) A new tracking approach for visible and infrared sequences based on tracking-before-fusion. Int J Dyn Control 4(1):40–51

    Google Scholar 

  56. Xiao J, Stolkin R, Gao Y, Leonardis A (2017) Robust fusion of color and depth data for RGB-D target tracking using adaptive range-invariant depth models and spatio-temporal consistency constraints. IEEE Trans Cybern 48(8):2485–2499

    Google Scholar 

  57. Yang H, Shao L, Zheng F, Wang L, Song Z (2011) Recent advances and trends in visual tracking: A review. Neurocomputing 74(18):3823–3831

    Google Scholar 

  58. Zadeh LA (1996) Fuzzy logic= computing with words. IEEE Trans Fuzzy Syst 4(2):103–111

    Google Scholar 

  59. Zheng Y, Meng Y (2007) “The PSO-based adaptive window for people tracking,” In 2007 IEEE Symposium on Computational Intelligence in Security and Defense Applications, pp. 23–29. IEEE

  60. Zhu P, Wen Q (2011) Homomorphisms between fuzzy information systems revisited. Appl Math Lett 24(9):1548–1553

    MathSciNet  MATH  Google Scholar 

  61. Zhu X, Guo K, Ren S, Hu B, Hu M, Fang H (2021) Lightweight image super-resolution with expectation-maximization attention mechanism. IEEE Trans Circ Syst Vid Technol 32(3):1273–1284

    Google Scholar 

  62. Zhu, X, Guo, K, Fang, H, Chen, L, Ren, S, Hu, B (2021) Cross view capture for stereo image super-resolution. IEEE Trans Multimed

Download references

Funding

DRDO has funded this project (Approval No: ERIP/ER/202205001/M/01/1811. Authors acknowledge and admire the research grant provided by DRDO.

Author information

Authors and Affiliations

  1. Department of ECE, DTU, New Delhi, India

    Rajiv Kapoor & Nikhil Singh

  2. Department of CSE, Thapar Institute of Engineering & Technology, Patiala, Punjab, India

    Nikhil Singh & Aarchishya Kapoor

Authors
  1. Rajiv Kapoor
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Nikhil Singh
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Aarchishya Kapoor
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rajiv Kapoor.

Ethics declarations

Conflicts of interest

All authors declare that there is no conflict of interest among us.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kapoor, R., Singh, N. & Kapoor, A. Multi-sensor based object tracking using enhanced particle swarm optimized multi-cue granular fusion. Multimed Tools Appl 82, 42417–42438 (2023). https://doi.org/10.1007/s11042-023-15164-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15164-9

Keywords