Skip to main content
SpringerLink
Log in

PPIS-JOIN: A Novel Privacy-Preserving Image Similarity Join Method

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Recently, massive multimedia data (especially images) is moved to the cloud environment for analysis and retrieval, which makes data security issue become particularly significant. Image similarity join has attracted more and more attention in the community of multimedia retrieval. However, few researches have investigated the privacy-preserving problem of image similarity join. To tackle this challenge, this paper proposes a novel privacy-preserving image similarity join method, called PPIS-JOIN. Different from the existing schemes, this approach aims to combine deep image hashing method and a novel affine transformation method to conceal sensitive information at feature level and generate high quality hash codes. Meanwhile, based on secure hash codes, a privacy-preserving similarity query model is proposed, which includes a secure image hash codes based inverted index, called ISH-Index, to support efficient and accuracy similarity search. We conduct comprehensive experiments on three common used benchmarks, and the results demonstrate the performance of the proposed PPIS-JOIN outperforms baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Cao D, Chu J, Zhu N, Nie L (2020) Cross-modal recipe retrieval via parallel- and cross-attention networks learning. Knowl Based Syst 193:105428

    Google Scholar 

  2. Li T, Liu W, Wang T, Zhao M, Li X, Ma M (2020) Trust data collections via vehicles joint with unmanned aerial vehicles in the smart Internet of Things. Trans Emerg Telecommun Technol. https://doi.org/10.1002/ett.3956

    Article  Google Scholar 

  3. Huang M, Zhang K, Zeng Z, Wang T, Liu Y (2020) An AUV-assisted data gathering scheme based on clustering and matrix completion for smart ocean. IEEE Internet Things J 7:9904

    Google Scholar 

  4. Jiang B, Huang G, Wang T, Gui J, Zhu X (2020) Trust based energy efficient data collection with unmanned aerial vehicle in edge network. Trans Emerg Telecommun Technol. https://doi.org/10.1002/ett.3942

    Article  Google Scholar 

  5. Zhang H, Sheng H (2008) A novel image authentication robust to geometric transformations. In: Congress on image and signal processing

  6. Xu C, Sun J, Ca Wang (2020) A novel image encryption algorithm based on bit-plane matrix rotation and hyper chaotic systems. Multimed Tools Appl 79:5573

    Google Scholar 

  7. Zhu X, Zhu Y, Zheng W (2020) Spectral rotation for deep one-step clustering. Pattern Recognit 105:107175

    Google Scholar 

  8. Shen H, Zhu Y, Zheng W, Zhu X (2020) Half-quadratic minimization for unsupervised feature selection on incomplete data. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/tnnls.2020.3009632

    Article  Google Scholar 

  9. Cao D, Han N, Wei X, He X (2020) Video-based recipe retrieval. Inf Sci 514:302

    Google Scholar 

  10. Yuan X, Wang X, Wang C, Yu C, Nutanong S (2017) Privacy-preserving similarity joins over encrypted data. IEEE Trans Inf Forensics Secur 12(11):2763–2775

    Google Scholar 

  11. Wu L, Wang Y, Gao J, Wang M, Zha Z, Tao D (2020) Deep co-attention based comparators for relative representation learning on person re-identification. IEEE Trans Neural Netw Learn Syst 32:722

    Google Scholar 

  12. Fang L, Liu Z, Song W (2019) Deep hashing neural networks for hyperspectral image feature extraction. IEEE Geosci Remote Sens Lett 16:1412

    Google Scholar 

  13. Liu Y, Xiao Y (2013) A robust image hashing algorithm resistant against geometrical attacks. Radioengineering 22:1072

    Google Scholar 

  14. Zhang C, Zhang Y, Zhang W, Lin X (2016) Inverted linear quadtree: efficient top k spatial keyword search. IEEE Trans Knowl Data Eng 28(7):1706–1721

    Google Scholar 

  15. Chen L, Shang S, Jensen CS, Yao B, Kalnis P (2020) Parallel semantic trajectory similarity join. In: 2020 IEEE 36th international conference on data engineering (ICDE), IEEE, pp 997–1008

  16. Zhu L, Yu W, Zhang C, Zhang Z, Huang F, Yu H (2019) SVS-JOIN: efficient spatial visual similarity join for geo-multimedia. IEEE Access 7:158389–158408

    Google Scholar 

  17. Ta N, Li G, Xie Y, Li C, Hao S, Feng J (2017) Signature-based trajectory similarity join. IEEE Trans Knowl Data Eng 29(4):870–883

    Google Scholar 

  18. Christiani T, Pagh R, Sivertsen J (2018) Scalable and robust set similarity join. In: 2018 IEEE 34th international conference on data engineering (ICDE), IEEE, pp 1240–1243

  19. Xiao C, Wang W, Lin X, Yu JX, Wang G (2011) Efficient similarity joins for near-duplicate detection. ACM Trans Database Syst 36(3):1–41

    Google Scholar 

  20. Shang Z, Liu Y, Li G, Feng J (2016) K-join: knowledge-aware similarity join. IEEE Trans Knowl Data Eng 28(12):3293–3308

    Google Scholar 

  21. Wang J, Li G, Fe J (2011) Fast-join: an efficient method for fuzzy token matching based string similarity join. In: 2011 IEEE 27th international conference on data engineering, IEEE, pp 458–469

  22. Wang J, Li G, Feng J (2012) Can we beat the prefix filtering? An adaptive framework for similarity join and search. In: Proceedings of the 2012 ACM SIGMOD international conference on management of data, pp 85–96

  23. Rong C, Lu W, Wang X, Du X, Chen Y, Tung AK (2012) Efficient and scalable processing of string similarity join. IEEE Trans Knowl Data Eng 25(10):2217–2230

    Google Scholar 

  24. Xiong Y, Zhu Y, Philip SY (2014) Top-k similarity join in heterogeneous information networks. IEEE Trans Knowl Data Eng 27(6):1710–1723

    Google Scholar 

  25. Li R, Zhao X, Shang H, Chen Y, Xiao W (2017) Fast top-k similarity join for SimRank. Inf Sci 381:1–19

    MATH  Google Scholar 

  26. Wang H, Yang L, Xiao Y (2020) SETJoin: a novel top-k similarity join algorithm. Soft Comput 24:1–16

    Google Scholar 

  27. Zheng W, Zou L, Chen L, Zhao D (2017) Efficient simrank-based similarity join. ACM Trans Database Syst 42(3):1–37

    MathSciNet  MATH  Google Scholar 

  28. Zhang J, Tang J, Ma C, Tong H, Jing Y, Li J, Moens MF (2017) Fast and flexible top-k similarity search on large networks. ACM Trans Inf Syst 36(2):1–30

    Google Scholar 

  29. Wu L, Wang Y, Gao J, Li X (2019) Where-and-when to look: deep siamese attention networks for video-based person re-identification. IEEE Trans Multimedia PP:1412–1424

    Google Scholar 

  30. Wang H, Li Z, Li Y, Gupta BB, Choi C (2020) Visual saliency guided complex image retrieval. Pattern Recognit Lett 130:64–72

    Google Scholar 

  31. Zhu L, Song J, Yu W, Zhang C, Yu H, Zhang Z (2020) Reverse spatial visual Top-\(k\) query. IEEE Access 8:21770–21787

    Google Scholar 

  32. Lu H, Zhang M, Xu X, Li Y, Shen HT (2020) Deep fuzzy hashing network for efficient image retrieval. IEEE Trans Fuzzy Syst 29:166

    Google Scholar 

  33. Wang Y, Wu L, Lin X, Gao J (2018) Multi-view spectral clustering via structured low-rank matrix factorization. IEEE Trans Neural Netw Learn Syst 29:4833–4843

    Google Scholar 

  34. Hu R, Zhu X, Zhu Y, Gan J (2020) Robust SVM with adaptive graph learning. World Wide Web 23:1945

    Google Scholar 

  35. Wang Y (2020) Survey on deep multi-modal data analytics: collaboration, rivalry and fusion. arXiv preprint arXiv:2006.08159

  36. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Google Scholar 

  37. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks?. In: Advances in neural information processing systems, pp 3320–3328

  38. Wan J, Wang D, Hoi SCH, Wu P, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of the 22nd ACM international conference on Multimedia, pp 157–166

  39. Zhang C, Zhu L, Zhang S, Yu W (2020) PAC-GAN: an effective pose augmentation scheme for unsupervised cross-view person re-identification. Neurocomputing 387:22–39

    Google Scholar 

  40. Zhu L, Long J, Zhang C, Yu W, Yuan X, Sun L (2019) An efficient approach for geo-multimedia cross-modal retrieval. IEEE Access 7:180571–180589

    Google Scholar 

  41. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Google Scholar 

  42. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  43. Gordo A, Almazán J, Revaud J, Larlus D (2016) Deep image retrieval: learning global representations for image search. In: European conference on computer vision, Springer, Cham, pp 241–257

  44. Liu P, Guo JM, Wu CY, Cai D (2017) Fusion of deep learning and compressed domain features for content-based image retrieval. IEEE Trans Image Process 26(12):5706–5717

    MathSciNet  MATH  Google Scholar 

  45. Seddati O, Dupont S, Mahmoudi S, Parian M (2017) Towards good practices for image retrieval based on CNN features. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1246–1255

  46. Yang J, Liang J, Shen H, Wang K, Rosin PL, Yang MH (2018) Dynamic match kernel with deep convolutional features for image retrieval. IEEE Trans Image Process 27(11):5288–5302

    MathSciNet  Google Scholar 

  47. Zhou Z, Zheng Y, Ye H, Pu J, Sun G (2018) Satellite image scene classification via ConvNet with context aggregation. In: Pacific rim conference on multimedia, Springer, Cham, pp 329–339

  48. Radenović F, Tolias G, Chum O (2018) Fine-tuning CNN image retrieval with no human annotation. IEEE Trans Pattern Anal Mach Intell 41(7):1655–1668

    Google Scholar 

  49. Bhattarai M, Oyen D, Castorena J, Yang L, Wohlberg B (2020) Diagram image retrieval using sketch-based deep learning and transfer learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 174–175

  50. Mukherjee A, Sil J, Sahu A, Chowdhury AS (2020) A bag of constrained informative deep visual words for image retrieval. Pattern Recognit Lett 129:158–165

    Google Scholar 

  51. Wang Y, Huang F, Zhang Y, Feng R, Zhang T, Fan W (2020) Deep cascaded cross-modal correlation learning for fine-grained sketch-based image retrieval. Pattern Recognit 100:107148

    Google Scholar 

  52. Luo X, Chen C, Zhong H, Zhang H, Deng M, Huang J, Hua X (2020) A survey on deep hashing methods. arXiv preprint arXiv:2003.03369

  53. Wang J, Liu W, Kumar S, Chang SF (2016) Learning to hash for indexing big data—a survey. Proc IEEE 104(1):34–57

    Google Scholar 

  54. Ouyang J, Liu Y, Shu H (2019) Robust hashing for image authentication using SIFT feature and quaternion Zernike moments. Multimed Tools Appl 76:2609

    Google Scholar 

  55. Liu Y, Xin G, Yong X (2016) Robust image hashing using radon transform and invariant features. Radioengineering 25:556–564

    Google Scholar 

  56. Wang Y (2020) Survey on deep multi-modal data analytics: collaboration, rivalry, and fusion. ACM Trans Multimed Comput Commun Appl 17:1–25

    Google Scholar 

  57. Lu J, Liong VE, Zhou J (2017) Deep hashing for scalable image search. IEEE Trans Image Process 26(5):2352–2367

    MathSciNet  MATH  Google Scholar 

  58. Yang HF, Tu CH, Chen CS (2019) Adaptive labeling for deep learning to hash. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

  59. Eghbali S, Tahvildari L (2019) Deep spherical quantization for image search. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 11690–11699

  60. Ghasedi Dizaji K, Zheng F, Sadoughi N, Yang Y, Deng C, Huang H (2018) Unsupervised deep generative adversarial hashing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3664–3673

  61. Xu J, Guo C, Liu Q, Qin J, Wang Y, Liu L (2019) DHA: Supervised deep learning to hash with an adaptive loss function. In: Proceedings of the IEEE international conference on computer vision workshops

  62. Gattupalli V, Zhuo Y, Li B (2019) Weakly supervised deep image hashing through tag embeddings. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 10375–10384

  63. Peng Y, Zhang J, Ye Z (2019) Deep reinforcement learning for image hashing. IEEE Trans Multimed 22:2061

    Google Scholar 

  64. Cui H, Zhu L, Li J, Yang Y, Nie L (2019) Scalable deep hashing for large-scale social image retrieval. IEEE Trans Image Process 29:1271–1284

    MathSciNet  Google Scholar 

  65. Shashank J, Kowshik P, Srinathan K, Jawahar CV (2008) Private content based image retrieval. In: 2008 IEEE conference on computer vision and pattern recognition, IEEE, pp 1–8

  66. Zhang C, Zhu L, Zhang S, Yu W (2020) TDHPPIR: an efficient deep hashing based privacy-preserving image retrieval method. Neurocomputing 406:386

    Google Scholar 

  67. Mohassel P, Zhang Y (2017) Secureml: a system for scalable privacy-preserving machine learning. In: 2017 IEEE symposium on security and privacy (SP), IEEE, pp 19–38

  68. Aono Y, Hayashi T, Wang L, Moriai S (2017) Privacy-preserving deep learning via additively homomorphic encryption. IEEE Trans Inf Forensics Secur 13(5):1333–1345

    Google Scholar 

  69. Shen M, Deng Y, Zhu L, Du X, Guizani N (2019) Privacy-preserving image retrieval for medical IoT systems: a blockchain-based approach. IEEE Netw 33(5):27–33

    Google Scholar 

  70. Yu L, Zheng Q, Liao X, Wu J (2020) Cryptanalysis and enhancement of an image encryption scheme based on a 1-D coupled Sine map. Nonlinear Dyn 100:1–15

    Google Scholar 

  71. Lu W, Varna AL, Swaminathan A, Wu M (2009) Secure image retrieval through feature protection. In: 2009 IEEE international conference on acoustics, speech and signal processing, IEEE, pp 1533–1536

  72. Abdulsada AI, Ali ANM, Abduljabbar ZA, Hashim HS (2013) Secure image retrieval over untrusted cloud servers. Int J Eng Adv Technol 3(1):2249

    Google Scholar 

  73. Ferreira B, Rodrigues J, Leitao J, Domingos H (2017) Practical privacy-preserving content-based retrieval in cloud image repositories. IEEE Trans Cloud Comput 7:784

    Google Scholar 

  74. Weng L, Amsaleg L, Morton A, Marchand-Maillet S (2014) A privacy-preserving framework for large-scale content-based information retrieval. IEEE Trans Inf Forensics Secur 10(1):152–167

    Google Scholar 

  75. Xia Z, Wang X, Zhang L, Qin Z, Sun X, Ren K (2016) A privacy-preserving and copy-deterrence content-based image retrieval scheme in cloud computing. IEEE Trans Inf Forensics Secur 11(11):2594–2608

    Google Scholar 

  76. Xu Y, Gong J, Xiong L, Xu Z, Wang J, Shi YQ (2017) A privacy-preserving content-based image retrieval method in cloud environment. J Vis Commun Image Represent 43:164–172

    Google Scholar 

  77. Shen M, Cheng G, Zhu L, Du X, Hu J (2020) Content-based multi-source encrypted image retrieval in clouds with privacy preservation. Future Gener Comput Syst 109:621–632

    Google Scholar 

  78. Rahim N, Ahmad J, Muhammad K, Sangaiah AK, Baik SW (2018) Privacy-preserving image retrieval for mobile devices with deep features on the cloud. Comput Commun 127:75–85

    Google Scholar 

  79. Razeghi B, Voloshynovskiy S (2018) Privacy-preserving outsourced media search using secure sparse ternary codes. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 1992–1996

  80. Dusmanu M, Schönberger JL, Sinha SN, Pollefeys M (2020) Privacy-preserving visual feature descriptors through adversarial affine subspace embedding. arXiv preprint arXiv:2006.06634

  81. Curtmola R, Garay J, Kamara S, Ostrovsky R (2011) Searchable symmetric encryption: improved definitions and efficient constructions. J Comput Secur 19(5):895–934

    Google Scholar 

  82. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images

  83. Chua TS, Tang J, Hong R, Li H, Luo Z, Zheng Y (2009) NUS-WIDE: a real-world web image database from National University of Singapore. In: Proceedings of the ACM international conference on image and video retrieval, pp 1–9

  84. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Zitnick CL (2014) Microsoft coco: common objects in context. In: European conference on computer vision, Springer, Cham, pp 740–755

  85. Weiss Y, Torralba A, Fergus R (2009) Spectral hashing. In: Advances in neural information processing systems, pp 1753–1760

  86. He K, Wen F, Sun J (2013) K-means hashing: An affinity-preserving quantization method for learning binary compact codes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2938–2945

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (61702560, 61472450, 61972203), the Key Research Program of Hunan Province (2016JC2018), project (2018JJ3691) of Science and Technology Plan of Hunan Province.

Author information

Authors and Affiliations

  1. College of Computer Science and Electronic Engineering, Hunan University, Changsha, People’s Republic of China

    Chengyuan Zhang

  2. School of Computer Science and Engineering, Central South University, Changsha, People’s Republic of China

    Fangxin Xie & Hao Yu

  3. College of Computer Science, National University of Defense Technology, Changsha, People’s Republic of China

    Jianfeng Zhang

  4. College of Information and Intelligence, Hunan Agricultural University, Changsha, People’s Republic of China

    Lei Zhu

  5. Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha, People’s Republic of China

    Yangding Li

Authors
  1. Chengyuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fangxin Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianfeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lei Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yangding Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Lei Zhu or Yangding Li.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, C., Xie, F., Yu, H. et al. PPIS-JOIN: A Novel Privacy-Preserving Image Similarity Join Method. Neural Process Lett 54, 2783–2801 (2022). https://doi.org/10.1007/s11063-021-10537-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10537-3

Keywords

Search

Navigation