Abstract
Machine learning models are now being widely deployed in clouds, but serious data leakage problems are also exposed when dealing with sensitive data. Homomorphic encryption (HE) has been used in the secure inference on unimodal private data because of its ability to calculate encrypted data. Although the privacy protection of multimodal data is of great significance, there is still no privacy-preserving inference scheme for multimodal data. In this work, we propose a lightweight and efficient homomorphic-encryption based framework that enables privacy-preserving multimodal representation inference. Firstly, we propose an HE scheme based on Tensor Fusion Network, which can perform encrypted multimodal feature fusion. Then we propose a pre-expansion method and a packaging method for multimodal data, which can effectively reduce the time delay and data traffic of homomorphic computing. The experimental results show that our encryption inference method has almost no loss of accuracy and obtains an F1 score of 0.7697, while using less than 220KB of data throughput and about 0.91 s of evaluation time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Albrecht, M.R., et al.: Homomorphic encryption standard. IACR Cryptol. ePrint Arch., 939 (2019). https://eprint.iacr.org/2019/939
Badawi, A.A., Hoang, L., Mun, C.F., Laine, K., Aung, K.M.M.: PrivFT: private and fast text classification with homomorphic encryption. IEEE Access 8, 226544–226556 (2020). https://doi.org/10.1109/ACCESS.2020.3045465
Badawi, A.A., et al.: Towards the AlexNet moment for homomorphic encryption: HCNN, the first homomorphic CNN on encrypted data with GPUs. IEEE Trans. Emerg. Top. Comput. 9(3), 1330–1343 (2021). https://doi.org/10.1109/TETC.2020.3014636
Baltrusaitis, T., Ahuja, C., Morency, L.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2019). https://doi.org/10.1109/TPAMI.2018.2798607
Benaissa, A., Retiat, B., Cebere, B., Belfedhal, A.E.: TenSEAL: a library for encrypted tensor operations using homomorphic encryption. CoRR abs/2104.03152 (2021). https://arxiv.org/abs/2104.03152
Bengio, Y., Courville, A.C., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013). https://doi.org/10.1109/TPAMI.2013.50
Brakerski, Z., Gentry, C., Vaikuntanathan, V.: (Leveled) fully homomorphic encryption without bootstrapping. ACM Trans. Comput. Theor. 6(3), 13:1-13:36 (2014). https://doi.org/10.1145/2633600
Brutzkus, A., Gilad-Bachrach, R., Elisha, O.: Low latency privacy preserving inference. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019, Long Beach, California, USA. Proceedings of Machine Learning Research, vol. 97, pp. 812–821. PMLR (2019). http://proceedings.mlr.press/v97/brutzkus19a.html
Cai, C., Sang, Y., Tian, H.: A multimodal differential privacy framework based on fusion representation learning. Connect. Sci. 34(1), 2219–2239 (2022). https://doi.org/10.1080/09540091.2022.2111406
Cheon, J.H., Kim, A., Kim, M., Song, Y.: Homomorphic encryption for arithmetic of approximate numbers. In: Takagi, T., Peyrin, T. (eds.) ASIACRYPT 2017. LNCS, vol. 10624, pp. 409–437. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70694-8_15
Chou, E.J., Gururajan, A., Laine, K., Goel, N.K., Bertiger, A., Stokes, J.W.: Privacy-preserving phishing web page classification via fully homomorphic encryption. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, 4–8 May 2020, pp. 2792–2796. IEEE (2020). https://doi.org/10.1109/ICASSP40776.2020.9053729
Deldjoo, Y., Schedl, M., Hidasi, B., Wei, Y., He, X.: Multimedia recommender systems: algorithms and challenges, 3rd edn. In: Recommender Systems Handbook (2020)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Volume 1 (Long and Short Papers), Minneapolis, MN, USA, 2–7 June 2019, pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/n19-1423
Fan, J., Vercauteren, F.: Somewhat practical fully homomorphic encryption. IACR Cryptol. ePrint Arch., 144 (2012). http://eprint.iacr.org/2012/144
Gentry, C.: Fully homomorphic encryption using ideal lattices. In: Mitzenmacher, M. (ed.) Proceedings of the 41st Annual ACM Symposium on Theory of Computing, STOC 2009, Bethesda, MD, USA, 31 May–2 June 2009, pp. 169–178. ACM (2009). https://doi.org/10.1145/1536414.1536440
Gilad-Bachrach, R., Dowlin, N., Laine, K., Lauter, K.E., Naehrig, M., Wernsing, J.: CryptoNets: applying neural networks to encrypted data with high throughput and accuracy. In: Balcan, M., Weinberger, K.Q. (eds.) Proceedings of the 33nd International Conference on Machine Learning, JMLR Workshop and Conference Proceedings, ICML 2016, New York City, NY, USA, 19–24 June 2016, vol. 48, pp. 201–210. JMLR.org (2016). http://proceedings.mlr.press/v48/gilad-bachrach16.html
Halevi, S., Shoup, V.: Algorithms in HElib. IACR Cryptol. ePrint Arch. 2014, 106 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 770–778. IEEE Computer Society (2016). https://doi.org/10.1109/CVPR.2016.90
Huynh, D.: Cryptotree: fast and accurate predictions on encrypted structured data. CoRR abs/2006.08299 (2020). https://arxiv.org/abs/2006.08299
McKeen, F., et al.: Innovative instructions and software model for isolated execution. In: Lee, R.B., Shi, W. (eds.) The Second Workshop on Hardware and Architectural Support for Security and Privacy, HASP 2013, Tel-Aviv, Israel, 23–24 June 2013, p. 10. ACM (2013). https://doi.org/10.1145/2487726.2488368
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Getoor, L., Scheffer, T. (eds.) Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, 28 June–2 July 2011, pp. 689–696. Omnipress (2011). https://icml.cc/2011/papers/399_icmlpaper.pdf
Rahulamathavan, Y.: Privacy-preserving similarity calculation of speaker features using fully homomorphic encryption. CoRR abs/2202.07994 (2022). https://arxiv.org/abs/2202.07994
Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Sig. Process. Mag. 34(6), 96–108 (2017). https://doi.org/10.1109/MSP.2017.2738401
Rivest, R.L., Dertouzos, M.L.: On Data Banks and Privacy Homomorphisms (1978)
Microsoft SEAL (release 4.0). Microsoft Research, Redmond, WA, March 2022. https://github.com/Microsoft/SEAL
Sun, L., Wang, J., Zhang, K., Su, Y., Weng, F.: RpBERT: a text-image relation propagation-based BERT model for multimodal NER. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, 2–9 February 2021, pp. 13860–13868. AAAI Press (2021). https://ojs.aaai.org/index.php/AAAI/article/view/17633
Wang, D., Xiong, D.: Efficient object-level visual context modeling for multimodal machine translation: masking irrelevant objects helps grounding. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, 2–9 February 2021, pp. 2720–2728. AAAI Press (2021).https://ojs.aaai.org/index.php/AAAI/article/view/16376
Yao, A.C.: Protocols for secure computations (extended abstract). In: 23rd Annual Symposium on Foundations of Computer Science, Chicago, Illinois, USA, 3–5 November 1982, pp. 160–164. IEEE Computer Society (1982). https://doi.org/10.1109/SFCS.1982.38
Yu, F., et al.: ERNIE-ViL: knowledge enhanced vision-language representations through scene graphs. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, 2–9 February 2021, pp. 3208–3216. AAAI Press (2021). https://ojs.aaai.org/index.php/AAAI/article/view/16431
Yu, W., et al.: CH-SIMS: a Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, 5–10 July 2020, pp. 3718–3727. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.acl-main.343
Yu, W., Xu, H., Yuan, Z., Wu, J.: Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, 2–9 February 2021, pp. 10790–10797. AAAI Press (2021). https://ojs.aaai.org/index.php/AAAI/article/view/17289
Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.: Tensor fusion network for multimodal sentiment analysis. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, 9–11 September 2017, pp. 1103–1114. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/d17-1115
Zadeh, A., Zellers, R., Pincus, E., Morency, L.: Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell. Syst. 31(6), 82–88 (2016). https://doi.org/10.1109/MIS.2016.94
Acknowledgement
This work was supported by the Key-Area Research and Development Program of Guangdong Province (No. 2020B010164003), China.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, Z., Sang, Y., Deng, X., Tian, H. (2023). Lightweight and Efficient Privacy-Preserving Multimodal Representation Inference via Fully Homomorphic Encryption. In: Nguyen, N.T., et al. Intelligent Information and Database Systems. ACIIDS 2023. Lecture Notes in Computer Science(), vol 13995. Springer, Singapore. https://doi.org/10.1007/978-981-99-5834-4_25
Download citation
DOI: https://doi.org/10.1007/978-981-99-5834-4_25
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-5833-7
Online ISBN: 978-981-99-5834-4
eBook Packages: Computer ScienceComputer Science (R0)