Skip to main content
SpringerLink
Log in

Unified Enclave Abstraction and Secure Enclave Migration on Heterogeneous Security Architectures

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Nowadays, application migration becomes more and more attractive. For example, it can make computation closer to data sources or make service closer to end-users, which may significantly decrease latency in edge computing. Yet, migrating applications among servers that are controlled by different platform owners raises security issues. We leverage hardware-secured trusted execution environment (TEE, aka., enclave) technologies, such as Intel SGX, AMD SEV, and ARM TrustZone, for protecting critical computations on untrusted servers. However, these hardware TEEs propose non-uniform programming abstractions and are based on heterogeneous architectures, which not only forces programmers to develop secure applications targeting some specific abstraction but also hinders the migration of protected applications. Therefore, we propose UniTEE which gives a unified enclave programming abstraction across the above three hardware TEEs by using a microkernel-based design and enables the secure enclave migration by integrating heterogeneous migration techniques. We have implemented the prototype on real machines. The evaluation results show the migration support incurs nearly-zero runtime overhead and the migration procedure is also efficient.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Park H, Zhai S, Lu L, Lin F X. StreamBox-TZ: Secure stream analytics at the edge with TrustZone. In Proc. the 2019 USENIX Annual Technical Conference, July 2019, pp.537-554.

  2. Shi W, Cao J, Zhang Q, Li Y, Xu L. Edge computing: Vision and challenges. IEEE Internet of Things Journal, 2016, 3(5): 637-646. DOI: https://doi.org/10.1109/JIOT.2016.2579198.

    Article  Google Scholar 

  3. Hu Y C, Patel M, Sabella D, Sprecher N, Young V. Mobile edge computing-A key technology towards 5G. Technical Report, European Telecommunications Standards Institute, 2015. https://infotech.report/Resources/Whitepaper-s/f205849d-0109-4de3-8c47-be52f4e4fb27_etsi_wp11_mec_a_key_technology_towards_5g.pdf, Dec. 2021.

  4. Satyanarayanan M. The emergence of edge computing. Computer, 2017, 50(1): 30-39. DOI: https://doi.org/10.1109/MC.2017.9.

    Article  Google Scholar 

  5. Shi W, Dustdar S. The promise of edge computing. Computer, 2016, 49(5): 78-81. DOI: https://doi.org/10.1109/MC.2016.145.

    Article  Google Scholar 

  6. Stojkoska B L R, Trivodaliev K V. A review of Internet of Things for smart home: Challenges and solutions. Journal of Cleaner Production, 2017, 140: 1454-1464. DOI: https://doi.org/10.1016/j.jclepro.2016.10.006.

    Article  Google Scholar 

  7. Nastic S, Rausch T, Scekic O, Dustdar S, Gusev M, Koteska B, Kostoska M, Jakimovski B, Ristov S, Prodan R. A server- less real-time data analytics platform for edge computing. IEEE Internet Computing, 2017, 21(4): 64-71. DOI: https://doi.org/10.1109/MIC.2017.2911430.

    Article  Google Scholar 

  8. Machen A, Wang S, Leung K K, Ko B J, Salonidis T. Live service migration in mobile edge clouds. IEEE Wireless Communications, 2017, 25(1): 140-147. DOI: https://doi.org/10.1109/MWC.2017.1700011.

    Article  Google Scholar 

  9. Wang S, Xu J, Zhang N, Liu Y. A survey on service migration in mobile edge computing. IEEE Access, 2018, 6: 23511-23528. DOI: https://doi.org/10.1109/ACCESS.2018.2828102.

    Article  Google Scholar 

  10. Islam M, Razzaque A, Islam J. A genetic algorithm for virtual machine migration in heterogeneous mobile cloud computing. In Proc. the 2016 International Conference on Net- working Systems and Security, Jan. 2016. DOI: https://doi.org/10.1109/N-SysS.2016.7400696.

  11. Barbalace A, Karaoui M L, Wang W, Xing T, Olivier P, Ravindran B. Edge computing: The case for heterogeneous-ISA container migration. In Proc. the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, Mar. 2020, pp.73-87. DOI: https://doi.org/10.1145/3381052.3381321.

  12. Rodrigues T G, Suto K, Nishiyama H, Kato N, Temma K. Cloudlets activation scheme for scalable mobile edge computing with transmission power control and virtual machine migration. IEEE Transactions on Computers, 2018, 67(9): 1287-1300. DOI: https://doi.org/10.1109/TC.2018.2818144.

    Article  MathSciNet  Google Scholar 

  13. Roman R, Lopez J, Mambo M. Mobile edge computing, fog et al.: A survey and analysis of security threats and challenges. Future Generation Computer Systems, 2018, 78: 680-698. DOI: https://doi.org/10.1016/j.future.2016.11.009.

    Article  Google Scholar 

  14. Ning Z, Liao J, Zhang F, Shi W. Preliminary study of trusted execution environments on heterogeneous edge platforms. In Proc. the 2018 IEEE/ACM Symposium on Edge Computing, Dec. 2018, pp.421-426. DOI: https://doi.org/10.1109/SEC.2018.00057.

  15. Costan V, Devadas S. Intel SGX explained. IACR Cryptol. ePrint Arch., 2016, 2016: Article No. 86.

  16. Kaplan D, Powell J, Woller T. AMD memory encryption. https://developer.amd.com/wordpress/media/2013/12/A-MD_Memory_Encryption_Whitepaper_v7-Public.pdf, Dec. 2021.

  17. Ngabonziza B, Martin D, Bailey A, Cho H, Martin S. Trust- Zone explained: Architectural features and use cases. In Proc. the 2nd IEEE International Conference on Collab- oration and Internet Computing, Nov. 2016, pp.445-451. DOI: https://doi.org/10.1109/CIC.2016.065.

  18. Kim T, Park J, Woo J, Jeon S, Huh J. ShieldStore: Shielded in-memory key-value storage with SGX. In Proc. the 14th EuroSys Conference 2019, Mar. 2019, Article No. 14. DOI: https://doi.org/10.1145/3302424.3303951.

  19. Arnautov S, Trach B, Gregor F et al. SCONE: Secure Linux containers with intel SGX. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, Nov. 2016, pp.689-703.

  20. Priebe C, Vaswani K, Costa M. EnclaveDB: A secure database using SGX. In Proc. the 2018 IEEE Symposium on Security and Privacy, May 2018, pp.264-278. DOI: https://doi.org/10.1109/SP.2018.00025.

  21. Tsai C C, Porter D E, Vij M. Graphene-SGX: A practical library OS for unmodified applications on SGX. In Proc. the 2017 USENIX Annual Technical Conference, July 2017, pp.645-658.

  22. Barbalace A, Lyerly R, Jelesnianski C, Carno A, Chuang H R, Legout V, Ravindran B. Breaking the boundaries in heterogeneous-ISA datacenters. ACM SIGARCH Computer Architecture News, 2017, 45(1): 645-659. DOI: https://doi.org/10.1145/3093337.3037738.

    Article  Google Scholar 

  23. Barbalace A, Sadini M, Ansary S, Jelesnianski C, Ravichandran A, Kendir C, Murray A, Ravindran B. Popcorn: Bridging the programmability gap in heterogeneous-ISA platforms. In Proc. the 10th European Conference on Computer Systems, Apr. 2015, Article No. 29. DOI: https://doi.org/10.1145/2741948.2741962.

  24. Gu J, Hua Z, Xia Y, Chen H, Zang B, Guan H, Li J. Secure live migration of SGX enclaves on untrusted cloud. In Proc. the 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, June 2017, pp.225-236. DOI: https://doi.org/10.1109/DSN.2017.37.

  25. Choy S, Wong B, Simon G, Rosenberg C. The brewing storm in cloud gaming: A measurement study on cloud to end-user latency. In Proc. the 11th Annual Workshop on Network and Systems Support for Games, Nov. 2012. DOI: https://doi.org/10.1109/NetGames.2012.6404024.

  26. Furlong M, Quinn A, Flinn J. The case for determinism on the edge. In Proc. the 2nd USENIX Workshop on Hot Topics in Edge Computing, July 2019.

  27. Ha K, Abe Y, Eiszler T, Chen Z, Hu W, Amos B, Upadhyaya R, Pillai P, Satyanarayanan M. You can teach elephants to dance: Agile VM handoff for edge computing. In Proc. the 2nd ACM/IEEE Symposium on Edge Computing, Oct. 2017, Article No. 12. DOI: https://doi.org/10.1145/3132211.3134453.

  28. Nadgowda S, Suneja S, Bila N, Isci C. Voyager: Complete container state migration. In Proc. the 37th IEEE International Conference on Distributed Computing Systems, June 2017, pp.2137-2142. DOI: https://doi.org/10.1109/ICDCS.2017.91.

  29. Jamshidi P, Ahmad A, Pahl C. Cloud migration research: A systematic review. IEEE Transactions on Cloud Computing, 2013, 1(2): 142-157. DOI: https://doi.org/10.1109/TCC.2013.10.

    Article  Google Scholar 

  30. Zhu J, Hou R, Wang X et al. Enabling rack-scale confidential computing using heterogeneous trusted execution environment. In Proc. the 2020 IEEE Symposium on Security and Privacy, May 2020, pp.1450-1465. DOI: https://doi.org/10.1109/SP40000.2020.00054.

  31. Hua Z, Gu J, Xia Y, Chen H, Zang B, Guan H. vTZ: Virtualizing ARM TrustZone. In Proc. the 26th USENIX Security Symposium, Aug. 2017, pp.541-556.

  32. Nightingale E B, Hodson O, McIlroy R, Hawblitzel C, Hunt G. Helios: Heterogeneous multiprocessing with satellite kernels. In Proc. the 22nd ACM SIGOPS Symposium on Operating Systems Principles, Oct. 2009, pp.221-234. DOI: https://doi.org/10.1145/1629575.1629597.

  33. Piraghaj S F, Dastjerdi A V, Calheiros R N, Buyya R. A framework and algorithm for energy efficient container consolidation in cloud data centers. In Proc. the 2015 IEEE International Conference on Data Science and Data Intensive Systems, Dec. 2015, pp.368-375. DOI: https://doi.org/10.1109/DS-DIS.2015.67.

  34. Wang H, Shi P, Zhang Y. JointCloud: A cross-cloud cooperation architecture for integrated internet service customization. In Proc. the 37th IEEE International Conference on Distributed Computing Systems, June 2017, pp.1846-1855. DOI: https://doi.org/10.1109/ICDCS.2017.237.

  35. Baumann A, Peinado M, Hunt G. Shielding applications from an untrusted cloud with Haven. ACM Transactions on Computer Systems, 2015, 33(3): Article No. 8. DOI: https://doi.org/10.1145/2799647.

  36. Hunt T, Zhu Z, Xu Y, Peter S, Witchel E. Ryoan: A distributed sandbox for untrusted computation on secret data. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, Nov. 2016, pp.533-549.

  37. Ohrimenko O, Schuster F, Fournet C, Mehta A, Nowozin S, Vaswani K, Costa M. Oblivious multi-party machine learning on trusted processors. In Proc. the 25th USENIX Conference on Security Symposium, August 2016, pp.619-636.

  38. Shinde S, Le Tien D, Tople S, Saxena P. Panoply: Low-TCB Linux applications with SGX enclaves. In Proc. the 24th Annual Network and Distributed System Security Symp., Feb. 26-Mar. 1, 2017. DOI: https://doi.org/10.14722/ndss.2017.23500.

  39. Schuster F, Costa M, Fournet C, Gkantsidis C, Peinado M, Mainar-Ruiz G, Russinovich M. VC3: Trustworthy data analytics in the cloud using SGX. In Proc. the 2015 IEEE Symposium on Security and Privacy, May 2015, pp.38-54. DOI: https://doi.org/10.1109/SP.2015.10.

  40. Li M, Zhang Y, Lin Z, Solihin Y. Exploiting unprotected I/O operations in AMD’s secure encrypted virtualization. In Proc. the 28th USENIX Security Symposium, Aug. 2019, pp.1257-1272.

  41. Morbitzer M, Huber M, Horsch J. Extracting secrets from encrypted virtual machines. In Proc. the 9th ACM Conference on Data and Application Security and Privacy, Mar. 2019, pp.221-230. DOI: https://doi.org/10.1145/3292006.3300022.

  42. Alves T, Felton D. TrustZone: Integrated hardware and software security. ARM White Paper, 2004, 3(4): 18-24.

    Google Scholar 

  43. Sun H, Sun K, Wang Y, Jing J. TrustOTP: Transforming smartphones into secure one-time password tokens. In Proc. the 22nd ACM SIGSAC Conference on Computer and Communications Security, Oct. 2015, pp.976-988. DOI: https://doi.org/10.1145/2810103.2813692.

  44. Santos N, Raj H, Saroiu S, Wolman A. Using ARM TrustZone to build a trusted language runtime for mobile applications. In Proc. the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, Feb. 2014, pp.67-80. DOI: https://doi.org/10.1145/2541940.2541949.

  45. Zhang N, Sun K, Lou W, Hou Y T. CaSE: Cache-assisted secure execution on ARM processors. In Proc. the 2016 IEEE Symposium on Security and Privacy, May 2016, pp.72-90. DOI: https://doi.org/10.1109/SP.2016.13.

  46. Guan L, Liu P, Xing X, Ge X, Zhang S, Yu M, Jaeger T. TrustShadow: Secure execution of unmodified applications with ARM TrustZone. In Proc. the 15th Annual International Conference on Mobile Systems, Applications, and Services, June 2017, pp.488-501. DOI: https://doi.org/10.1145/3081333.3081349.

  47. Zhao S, Zhang Q, Qin Y, Feng W, Feng D. SecTEE: A software-based approach to secure enclave architecture using TEE. In Proc. the 2019 ACM SIGSAC Conference on Computer and Communications Security, Nov. 2019, pp.1723-1740. DOI: https://doi.org/10.1145/3319535.3363205.

  48. Lind J, Priebe C, Muthukumaran D et al. Glamdring: Automatic application partitioning for Intel SGX. In Proc. the 2017 USENIX Annual Technical Conference, July 2017, pp.285-298.

  49. Soares L, Stumm M. FlexSC: Flexible system call scheduling with exception-less system calls. In Proc. the 9th USENIX Conference on Operating Systems Design and Implementation, Oct. 2010, pp.33-46.

  50. Rott J. Intel® advanced encryption standard instructions (AES-NI). https://www.intel.com/content/www/us/en/d-eveloper/articles/technical/advanced-encryption-standard- instructions-aes-ni.html, Dec. 2021.

  51. McCune J M, Li Y, Qu N, Zhou Z, Datta A, Gligor V, Perrig A. TrustVisor: Efficient TCB reduction and attestation. In Proc. the 2010 IEEE Symposium on Security and Privacy, May 2010, pp.143-158. DOI: https://doi.org/10.1109/SP.2010.17.

  52. Zhang F, Chen J, Chen H, Zang B. CloudVisor: Retrofitting protection of virtual machines in multi-tenant cloud with nested virtualization. In Proc. the 23rd ACM Symposium on Operating Systems Principles, Oct. 2011, pp.203-216. DOI: https://doi.org/10.1145/2043556.2043576.

  53. Dautenhahn N, Kasampalis T, Dietz W, Criswell J, Adve V. Nested kernel: An operating system architecture for intra-kernel privilege separation. ACM SIGPLAN Notices, 2015, 50(4): 191-206. DOI: https://doi.org/10.1145/2694344.2694386.

    Article  Google Scholar 

  54. Nelson L, Sigurbjarnarson H, Zhang K, Johnson D, Bornholt J, Torlak E, Wang X. Hyperkernel: Push-button verification of an OS kernel. In Proc. the 26th Symposium on Operating Systems Principles, Oct. 2017, pp.252-269. DOI: https://doi.org/10.1145/3132747.3132748.

  55. Klein G, Elphinstone K, Heiser G et al. sel4: Formal verification of an OS kernel. In Proc. the 22nd ACM SIGOPS Symposium on Operating Systems Principles, Oct. 2009, pp.207-220. DOI: https://doi.org/10.1145/1629575.1629596.

  56. Baumann A. Hardware is the new software. In Proc. the 16th Workshop on Hot Topics in Operating Systems, May 2017, pp.132-137. DOI: https://doi.org/10.1145/3102980.3103002.

  57. Ferraiuolo A, Baumann A, Hawblitzel C, Parno B. Komodo: Using verification to disentangle secure-enclave hardware from software. In Proc. the 26th Symposium on Operating Systems Principles, Oct. 2017, pp.287-305. DOI: https://doi.org/10.1145/3132747.3132782.

  58. Brasser F, Gens D, Jauernig P, Sadeghi A R, Stapf E. SANCTUARY: ARMing TrustZone with user-space enclaves. In Proc. the 26th Annual Network and Distributed System Security Symposium, Feb. 2019. DOI: https://doi.org/10.14722/ndss.2019.23448.

  59. Gu J, Wu X, Zhu B, Xia Y, Zang B, Guan H, Chen H. Enclavisor: A hardware-software co-design for enclaves on untrusted cloud. IEEE Transactions on Computers, 2021, 70(10): 1598-1611. DOI: https://doi.org/10.1109/TC.2020.3019704.

    Article  MATH  Google Scholar 

  60. Levin R, Cohen E, Corwin W, Pollack F, Wulf W. Policy/mechanism separation in Hydra. In Proc. the 5th ACM Symposium on Operating Systems Principles, Nov. 1975, pp.132-140. DOI: https://doi.org/10.1145/800213.806531

  61. Liedtke J. Improving IPC by kernel design. In Proc. the 14th ACM Symposium on Operating Systems Principles, Dec. 1993, pp.175-188. DOI: https://doi.org/10.1145/168619.168633.

  62. David F M, Chan E, Carlyle J C, Campbell R H. CuriOS: Improving reliability through operating system structure. In Proc. the 8th USENIX Conference on Operating Systems Design and Implementation, Dec. 2008, pp.59-72.

  63. Gu J, Wu X, Li W, Liu N, Mi Z, Xia Y, Chen H. Harmonizing performance and isolation in microkernels with efficient intra-kernel isolation and communication. In Proc. the 2020 USENIX Annual Technical Conference, July 2020, pp.401- 417.

  64. Hildebrand D. An architectural overview of QNX. In Proc. the Workshop on Micro-Kernels and Other Kernel Architectures, Apr. 1992, pp.113-126.

  65. Ji D, Zhang Q, Zhao S, Shi Z, Guan Y. MicroTEE: Designing TEE OS based on the microkernel architecture. In Proc. the 18th IEEE International Conference on Trust, Security and Privacy in Computing and Communications/13th IEEE International Conference on Big Data Science and Engineering, Aug. 2019, pp.26-33. DOI: https://doi.org/10.1109/Trust-Com/BigDataSE.2019.00014.

  66. DeVuyst M, Venkat A, Tullsen D M. Execution migration in a heterogeneous-ISA chip multiprocessor. In Proc. the 17th International Conference on Architectural Support for Programming Languages and Operating Systems, Mar. 2012, pp.261-272. DOI: https://doi.org/10.1145/2150976.2151004.

  67. Gordon M S, Jamshidi D A, Mahlke S, Mao Z M, Chen X. COMET: Code offload by migrating execution transparently. In Proc. the 10th USENIX Symposium on Operating Systems Design and Implementation, Oct. 2012, pp.93-106.

Download references

Author information

Authors and Affiliations

  1. Engineering Research Center for Domain-Specific Operating Systems, Ministry of Education, Shanghai, 200240, China

    Jin-Yu Gu, Hao Li, Yu-Bin Xia & Hai-Bo Chen

  2. Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, Shanghai, 200240, China

    Jin-Yu Gu, Hao Li, Yu-Bin Xia & Hai-Bo Chen

  3. Ant Group, Hangzhou, 310099, China

    Cheng-Gang Qin & Zheng-Yu He

Authors
  1. Jin-Yu Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu-Bin Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hai-Bo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cheng-Gang Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zheng-Yu He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu-Bin Xia.

Supplementary Information

ESM 1

(PDF 279 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gu, JY., Li, H., Xia, YB. et al. Unified Enclave Abstraction and Secure Enclave Migration on Heterogeneous Security Architectures. J. Comput. Sci. Technol. 37, 468–486 (2022). https://doi.org/10.1007/s11390-021-1083-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-1083-8

Keywords

Search

Navigation