research-article

Open access

Secure Softmax/Sigmoid for Machine-learning Computation

Authors:

Sherman S. M. Chow,

Shan YinAuthors Info & Claims

ACSAC '23: Proceedings of the 39th Annual Computer Security Applications Conference

Pages 463 - 476

https://doi.org/10.1145/3627106.3627175

Published: 04 December 2023 Publication History

All formats PDF

Abstract

Softmax and sigmoid, composing exponential functions (ex) and division (1/x), are activation functions often required in training. Secure computation on non-linear, unbounded 1/x and ex is already challenging, let alone their composition. Prior works aim to compute softmax by its exact formula via iteration (CrypTen, NeurIPS ’21) or with ASM approximation (Falcon, PoPETS ’21). They fall short in efficiency and/or accuracy. For sigmoid, existing solutions such as ABY2.0 (Usenix Security ’21) compute it via piecewise functions, incurring logarithmic communication rounds.

We study a rarely-explored approach to secure computation using ordinary differential equations and Fourier series for numerical approximation of rational/trigonometric polynomials over composition rings. Our results include 1) the first constant-round protocol for softmax and 2) the first 1-round error-bounded protocol for approximating sigmoid. They reduce communication by and, respectively, shortening the private training process of state-of-the-art frameworks or platforms, namely, CryptGPU (S&P ’21), Piranha (Usenix Security ’22), and quantized training from MP-SPDZ (ICML ’22), while maintaining competitive accuracy.

References

[1]

Jianli Bai, Xiaowu Zhang, Xiangfu Song, Hang Shao, Qifan Wang, Shujie Cui, and Giovanni Russello. 2023. CryptoMask: Privacy-preserving Face Recognition. In ICICS. 333–350.

[2]

Christina Boura, Ilaria Chillotti, Nicolas Gama, Dimitar Jetchev, Stanislav Peceny, and Alexander Petric. 2018. High-Precision Privacy-Preserving Real-Valued Function Evaluation. In FC. 183–202.

[3]

John Charles Butcher. 2016. Numerical Methods for Ordinary Differential Equations. John Wiley & Sons, New Zealand.

[4]

Paul L. Butzer and Rolf J. Nessel. 1971. Fourier Analysis and Approximation: One Dimensional Theory. Birkhäuser Basel, Switzerland.

[5]

Yuanfeng Chen, Gaofeng Huang, Junjie Shi, Xiang Xie, and Yilin Yan. 2020. Rosetta: A Privacy-Preserving Framework Based on TensorFlow. https://github.com/LatticeX-Foundation/Rosetta. Also presented at the Privacy Preserving Machine Learning Workshop at ACM CCS 2021.

[6]

Morten Dahl, Jason Mancuso, Yann Dupis, Ben Decoste, Morgan Giraud, Ian Livingstone, Justin Patriquin, and Gavin Uhma. 2018. Private Machine Learning in TensorFlow using Secure Computation. arXiV 1810.08130. Also presented at the Privacy Preserving Machine Learning Workshop at NeurIPS 2018.

[7]

Anders P. K. Dalskov, Daniel Escudero, and Marcel Keller. 2021. Fantastic Four: Honest-Majority Four-Party Secure Computation with Malicious Security. In Usenix Security. 2183–2200.

[8]

Li Deng. 2012. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]. IEEE Signal Process. Mag. 29, 6 (2012), 141–142.

[9]

Minxin Du, Xiang Yue, Sherman S. M. Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. 2023. DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass. In ACM CCS. 18 pages. To appear, also available at arXiV 2309.06746.

[10]

Simeon Ola Fatunla. 1988. Numerical Methods for Initial Value Problems in Ordinary Differential Equations. Elsevier, Boston.

[11]

Kanav Gupta, Deepak Kumaraswamy, Nishanth Chandran, and Divya Gupta. 2022. LLAMA: A Low Latency Math Library for Secure Inference. Proc. Priv. Enhancing Technol. 2022, 4 (2022), 274–294.

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR. 770–778.

[13]

Zhicong Huang, Wen-jie Lu, Cheng Hong, and Jiansheng Ding. 2022. Cheetah: Lean and Fast Secure Two-Party Deep Neural Network Inference. In Usenix Security. 809–826.

[14]

Marcel Keller and Ke Sun. 2022. Secure Quantized Training for Deep Learning. In ICML. 10912–10938.

[15]

Brian Knott, Shobha Venkataraman, Awni Y. Hannun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. 2021. CrypTen: Secure Multi-Party Computation Meets Machine Learning. In NeurIPS. 4961–4973.

[16]

Alex Krizhevsky and Geoffrey Hinton. 2009. Learning Multiple Layers of Features from Tiny Images. https://www.cs.toronto.edu/ kriz/cifar.html.

[17]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS. 1106–1114.

[18]

Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Comput. 1, 4 (1989), 541–551.

Digital Library

[19]

Wen-jie Lu, Zhicong Huang, Qizhi Zhang, Yuchen Wang, and Cheng Hong. 2023. Squirrel: A Scalable Secure Two-Party Computation Framework for Training Gradient Boosting Decision Tree. In Usenix Security. 6435–6451.

[20]

Jack P. K. Ma, Raymond K. H. Tai, Yongjun Zhao, and Sherman S. M. Chow. 2021. Let’s Stride Blindfolded in a Forest: Sublinear Multi-Client Decision Trees Evaluation. In NDSS. 18 pages.

[21]

Payman Mohassel and Peter Rindal. 2018. ABY3: A Mixed Protocol Framework for Machine Learning. In ACM CCS. 35–52.

[22]

Payman Mohassel and Yupeng Zhang. 2017. SecureML: A System for Scalable Privacy-Preserving Machine Learning. In IEEE S&P. 19–38.

[23]

Lucien K. L. Ng and Sherman S. M. Chow. 2021. GForce: GPU-Friendly Oblivious and Rapid Neural Network Inference. In Usenix Security. 2147–2164.

[24]

Lucien K. L. Ng and Sherman S. M. Chow. 2023. SoK: Cryptographic Neural-Network Computation. In IEEE S&P. 497–514.

[25]

Lucien K. L. Ng, Sherman S. M. Chow, Anna P. Y. Woo, Donald P. H. Wong, and Yongjun Zhao. 2021. Goten: GPU-Outsourcing Trusted Execution of Neural Network Training. In AAAI. 14876–14883.

[26]

Arpita Patra, Thomas Schneider, Ajith Suresh, and Hossein Yalame. 2021. ABY2.0: Improved Mixed-Protocol Secure Two-Party Computation. In Usenix Security. 2165–2182.

[27]

Deevashwer Rathee, Anwesh Bhattacharya, Rahul Sharma, Divya Gupta, Nishanth Chandran, and Aseem Rastogi. 2022. SecFloat: Accurate Floating-Point meets Secure 2-Party Computation. In IEEE S&P. 576–595.

[28]

Deevashwer Rathee, Mayank Rathee, Rahul Kranti Kiran Goli, Divya Gupta, Rahul Sharma, Nishanth Chandran, and Aseem Rastogi. 2021. SiRnn: A Math Library for Secure RNN Inference. In IEEE S&P. 1003–1020.

[29]

Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. 2020. CrypTFlow2: Practical 2-Party Secure Inference. In ACM CCS. 325–342.

[30]

Théo Ryffel, Pierre Tholoniat, David Pointcheval, and Francis R. Bach. 2022. AriaNN: Low-Interaction Privacy-Preserving Deep Learning via Function Secret Sharing. Proc. Priv. Enhancing Technol. 2022, 1 (2022), 291–316.

[31]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR. 14 pages.

[32]

Sijun Tan, Brian Knott, Yuan Tian, and David J. Wu. 2021. CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU. In IEEE S&P. 1021–1038.

[33]

Sameer Wagh, Divya Gupta, and Nishanth Chandran. 2019. SecureNN: 3-Party Secure Computation for Neural Network Training. Proc. Priv. Enhancing Technol. 2019, 3 (2019), 26–49.

[34]

Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateek Mittal, and Tal Rabin. 2021. Falcon: Honest-Majority Maliciously Secure Framework for Private Deep Learning. Proc. Priv. Enhancing Technol. 2021, 1 (2021), 188–208.

[35]

Jean-Luc Watson, Sameer Wagh, and Raluca Ada Popa. 2022. Piranha: A GPU Platform for Secure Computation. In Usenix Security. 827–844.

[36]

Harry W. H. Wong, Jack P. K. Ma, Donald P. H. Wong, Lucien K. L. Ng, and Sherman S. M. Chow. 2020. Learning Model with Error - Exposing the Hidden Model of BAYHENN. In IJCAI. 3529–3535.

[37]

Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Peng Hao, Tongliang Liu, and Bo Han. 2023. FedFed: Feature Distillation against Data Heterogeneity in Federated Learning. In NeurIPS. 32 pages.

[38]

Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. 2021. Dive into Deep Learning. arXiv:2106.11342.

[39]

Mengxin Zheng, Qian Lou, and Lei Jiang. 2023. Primer: A Privacy-preserving Transformer on Encrypted Data. In DAC. 6 pages.

[40]

Yu Zheng, Wei Song, Minxin Du, Sherman S. M. Chow Qian Lou, Yongjun Zhao, and Xiuhua Wang. 2023. Cryptography-Inspired Federated Learning for Generative Adversarial Networks and Meta Learning. In ADMA. 393–407.

Cited By

Wu YLiao CSun XShen YWu T(2025)Communication Efficient Secure Three-Party Computation Using Lookup Tables for RNN InferenceElectronics10.3390/electronics1405098514:5(985)Online publication date: 28-Feb-2025
https://doi.org/10.3390/electronics14050985
Zheng YZhang WSong WWang XFu C(2024)Encrypted Video Search with Single/Multiple WritersACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3643887Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1145/3643887
He GRen YHe GFeng GZhang X(2024)PPNNI: Privacy-Preserving Neural Network Inference against Adversarial Example AttackIEEE Transactions on Services Computing10.1109/TSC.2024.3399648(1-14)Online publication date: 2024
https://doi.org/10.1109/TSC.2024.3399648

Index Terms

Secure Softmax/Sigmoid for Machine-learning Computation
1. Computing methodologies
  1. Machine learning
2. Security and privacy
  1. Cryptography
  2. Security services
    1. Privacy-preserving protocols

Recommendations

Complete fairness in secure two-party computation
STOC '08: Proceedings of the fortieth annual ACM symposium on Theory of computing

In the setting of secure two-party computation, two mutually distrusting parties wish to compute some function of their inputs while preserving, to the extent possible, various security properties such as privacy, correctness, and more. One desirable ...
Improvements to Secure Computation with Penalties
CCS '16: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security

Motivated by the impossibility of achieving fairness in secure computation [Cleve, STOC 1986], recent works study a model of fairness in which an adversarial party that aborts on receiving output is forced to pay a mutually predefined monetary penalty ...
Zero-knowledge from secure multiparty computation
STOC '07: Proceedings of the thirty-ninth annual ACM symposium on Theory of computing

We present a general construction of a zero-knowledge proof for an NP relation R(x,w) which only makes a black-box use of a secure protocol for a related multi-partyfunctionality f. The latter protocol is only required to be secure against a small ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACSAC '23: Proceedings of the 39th Annual Computer Security Applications Conference

December 2023

836 pages

ISBN:9798400708862

DOI:10.1145/3627106

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Research Grants Council, University Grants Committee

Conference

ACSAC '23

ACSAC '23: Annual Computer Security Applications Conference

December 4 - 8, 2023

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 104 of 497 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
1,657
Total Downloads

Downloads (Last 12 months)1,477
Downloads (Last 6 weeks)165

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu YLiao CSun XShen YWu T(2025)Communication Efficient Secure Three-Party Computation Using Lookup Tables for RNN InferenceElectronics10.3390/electronics1405098514:5(985)Online publication date: 28-Feb-2025
https://doi.org/10.3390/electronics14050985
Zheng YZhang WSong WWang XFu C(2024)Encrypted Video Search with Single/Multiple WritersACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3643887Online publication date: 5-Feb-2024
https://dl.acm.org/doi/10.1145/3643887
He GRen YHe GFeng GZhang X(2024)PPNNI: Privacy-Preserving Neural Network Inference against Adversarial Example AttackIEEE Transactions on Services Computing10.1109/TSC.2024.3399648(1-14)Online publication date: 2024
https://doi.org/10.1109/TSC.2024.3399648
Byun JChoi YLee JPark S(2024)Privacy-preserving inference resistant to model extraction attacksExpert Systems with Applications10.1016/j.eswa.2024.124830(124830)Online publication date: Jul-2024
https://doi.org/10.1016/j.eswa.2024.124830

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten