skip to main content
research-article

Lazy Load Scheduling for Mixed-criticality Applications in Heterogeneous MPSoCs

Published: 13 May 2023 Publication History

Abstract

Newly emerging multiprocessor system-on-a-chip (MPSoC) platforms provide hard processing cores with programmable logic (PL) for high-performance computing applications. In this article, we take a deep look into these commercially available heterogeneous platforms and show how to design mixed-criticality applications such that different processing components can be isolated to avoid contention on the shared resources such as last-level cache and main memory.
Our approach involves software/hardware co-design to achieve isolation between the different criticality domains. At the hardware level, we use a scratchpad memory (SPM) with dedicated interfaces inside the PL to avoid conflicts in the main memory. At the software level, we employ a hypervisor to support cache-coloring such that conflicts at the shared L2 cache can be avoided. In order to move the tasks in/out of the SPM memory, we rely on a DMA engine and propose a new CPU-DMA co-scheduling policy, called Lazy Load, for which we also derive the response time analysis. The results of a case study on image processing demonstrate that the contention on the shared memory subsystem can be avoided when running with our proposed architecture. Moreover, comprehensive schedulability evaluations show that the newly proposed Lazy Load policy outperforms the existing CPU-DMA scheduling approaches and is effective in mitigating the main memory interference in our proposed architecture.

References

[1]
Ahmed Alhammad and Rodolfo Pellizzoni. 2014. Time-predictable execution of multithreaded applications on multicore systems. In 2014 Design, Automation Test in Europe Conference Exhibition (DATE’14). 1–6. DOI:
[2]
Muhammad Ali Awan, Konstantinos Bletsas, Pedro F. Souto, Benny Akesson, and Eduardo Tovar. 2018. Mixed-criticality scheduling with dynamic memory bandwidth regulation. In 2018 IEEE 24th International Conference on Embedded and Real-time Computing Systems and Applications (RTCSA’18). 111–117. DOI:
[3]
Matthias Becker, Dakshina Dasari, Borislav Nicolic, Benny Akesson, Vincent Nélis, and Thomas Nolte. 2016. Contention-free execution of automotive applications on a clustered many-core platform. In 2016 28th Euromicro Conference on Real-time Systems (ECRTS’16). 14–24. DOI:
[4]
Enrico Bini and Giorgio C. Buttazzo. 2005. Measuring the performance of schedulability tests. Real-time Systems 30, 1–2 (2005), 129–154. DOI:
[5]
Frédéric Boniol, Hugues Cassé, Eric Noulard, and Claire Pagetti. 2012. Deterministic execution model on COTS hardware. In Architecture of Computing Systems (ARCS’12). Springer, 98–110. DOI:
[6]
Reinder J. Bril, Johan J. Lukkien, and Wim F. J. Verhaegh. 2007. Worst-case response time analysis of real-time tasks under fixed-priority scheduling with deferred preemption revisited. In 19th Euromicro Conference on Real-time Systems (ECRTS’07). 269–279. DOI:
[7]
Paolo Burgio, Andrea Marongiu, Paolo Valente, and Marko Bertogna. 2015. A memory-centric approach to enable timing-predictability within embedded many-core accelerators. In 2015 CSI Symposium on Real-time and Embedded Systems and Technologies (RTEST’15). 1–8. DOI:
[8]
Alan Burns and Robert I. Davis. 2017. A survey of research into mixed criticality systems. ACM Comput. Surv. 50, 6, Article 82 (Nov.2017), 37 pages. DOI:
[9]
Daniel Casini, Paolo Pazzaglia, Alessandro Biondi, Marco Di Natale, and Giorgio Buttazzo. 2020. Predictable memory-CPU co-scheduling with support for latency-sensitive tasks. In 2020 57th ACM/IEEE Design Automation Conference (DAC’20). 1–6. DOI:
[10]
Jon Perez Cerrolaza, Roman Obermaisser, Jaume Abella, Francisco J. Cazorla, Kim Grüttner, Irune Agirre, Hamidreza Ahmadian, and Imanol Allende. 2020. Multi-core devices for safety-critical systems: A survey. ACM Comput. Surv. 53, 4, Article 79 (Aug.2020), 38 pages. DOI:
[11]
Robert I. Davis, Alan Burns, Reinder J. Bril, and Johan J. Lukkien. 2007. Controller area network (CAN) schedulability analysis: Refuted, revisited and revised. Real-time Systems 35, 3 (April2007), 239–272. DOI:
[12]
Guy Durrieu, Madeleine Faugère, Sylvain Girbal, Daniel Gracia Pérez, Claire Pagetti, and Wolfgang Puffitsch. 2014. Predictable flight management system implementation on a multicore processor. In Embedded Real-time Software (ERTS’14). https://hal.archives-ouvertes.fr/hal-01121700.
[13]
Evidence. 2020. Erika Enterprise RTOS v3. http://www.erika-enterprise.com/.
[14]
Björn Forsberg, Luca Benini, and Andrea Marongiu. 2018. HePREM: Enabling predictable GPU execution on heterogeneous SoC. In 2018 Design, Automation Test in Europe Conference Exhibition (DATE’18). 539–544. DOI:
[15]
Giovani Gracioli, Ahmed Alhammad, Renato Mancuso, Antônio Augusto Fröhlich, and Rodolfo Pellizzoni. 2015. A survey on cache management mechanisms for real-time embedded systems. ACM Comput. Surv. 48, 2, Article 32 (Nov.2015), 36 pages. DOI:
[16]
Giovani Gracioli and Antônio Augusto Fröhlich. 2017. Two-phase colour-aware multicore real-time scheduler. IET Computers & Digital Techniques 11, 4 (July2017), 133–139(6).
[17]
Giovani Gracioli, Rohan Tabish, Renato Mancuso, Reza Mirosanlou, Rodolfo Pellizzoni, and Marco Caccamo. 2019. Designing mixed criticality applications on modern heterogeneous MPSoC platforms. In 31st Euromicro Conference on Real-time Systems (ECRTS’19)(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 133), Sophie Quinton (Ed.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 27:1–27:25. DOI:
[18]
Heechul Yun. 2019. Latency and Bandwidth Utilities. https://github.com/heechul/misc.
[19]
Denis Hoornaert, Shahin Roozkhosh, and Renato Mancuso. 2021. A memory scheduling infrastructure for multi-core systems with re-programmable logic. In 33rd Euromicro Conference on Real-time Systems (ECRTS’21)(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 196), Björn B. Brandenburg (Ed.). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 2:1–2:22. DOI:
[20]
Hyoseung Kim, Dionisio de Niz, Björn Andersson, Mark Klein, Onur Mutlu, and Ragunathan Rajkumar. 2014. Bounding memory interference delay in COTS-based multi-core systems. In 2014 IEEE 19th Real-time and Embedded Technology and Applications Symposium (RTAS’14). 145–154. DOI:
[21]
Hyoseung Kim, Arvind Kandhalu, and Ragunathan Rajkumar. 2013. A coordinated approach for practical OS-level cache management in multi-core real-time systems. In 2013 25th Euromicro Conference on Real-time Systems. 80–89. DOI:
[22]
Hyoseung Kim and Ragunathan (Raj) Rajkumar. 2017. Predictable shared cache management for multi-core real-time virtualization. ACM Trans. Embed. Comput. Syst. 17, 1, Article 22 (Dec.2017), 27 pages. DOI:
[23]
Namhoon Kim, Bryan C. Ward, Micaiah Chisholm, Cheng-Yang Fu, James H. Anderson, and F. Donelson Smith. 2016. Attacking the one-out-of-m multicore problem by combining hardware management with mixed-criticality provisioning. In 2016 IEEE Real-time and Embedded Technology and Applications Symposium (RTAS’16). 1–12. DOI:
[24]
Tomasz Kloda, Marco Solieri, Renato Mancuso, Nicola Capodieci, Paolo Valente, and Marko Bertogna. 2019. Deterministic memory hierarchy and virtualization for modern multi-core embedded systems. In 2019 IEEE Real-time and Embedded Technology and Applications Symposium (RTAS’19). 1–14. DOI:
[25]
John P. Lehoczky. 1990. Fixed priority scheduling of periodic task sets with arbitrary deadlines. In Proceedings of the 11th Real-time Systems Symposium. 201–209. DOI:
[26]
C. L. Liu and James W. Layland. 1973. Scheduling algorithms for multiprogramming in a hard-real-time environment. J. ACM 20, 1 (Jan.1973), 46–61. DOI:
[27]
Méndez Miguel Macías, José L. Gutierrez, David Fernández, and Javier Díaz. 2013. Open platform for mixed-criticality applications. In Proceedings of the Workshop on Industry-driven Approaches for Cost-effective Certification of Safety-critical, Mixed-criticality Systems (WICERT’13). 1–7. http://atcproyectos.ugr.es/wicert/downloads/wicert_papers/wicert2013_submission_8.pdf.
[28]
Renato Mancuso, Roman Dudko, Emiliano Betti, Marco Cesati, Marco Caccamo, and Rodolfo Pellizzoni. 2013. Real-time cache management framework for multi-core architectures. In 2013 IEEE 19th Real-time and Embedded Technology and Applications Symposium (RTAS’13). 45–54. DOI:
[29]
Joel Matějka, Björn Forsberg, Michal Sojka, Luca Benini, Zdeněk Hanzálek, and Andrea Marongiu. 2019. Combining PREM compilation and static scheduling for high-performance and predictable MPSoC execution. Parallel Comput. 85 (2019), 27–44. DOI:
[30]
Alessandra Melani, Marko Bertogna, Vincenzo Bonifaci, Alberto Marchetti-Spaccamela, and Giorgio Buttazzo. 2015. Memory-processor co-scheduling in fixed priority systems. In Proceedings of the 23rd International Conference on Real-time and Networks Systems (RTNS’15). Association for Computing Machinery, New York, NY,87–96. DOI:
[31]
Paolo Modica, Alessandro Biondi, Giorgio Buttazzo, and Anup Patel. 2018. Supporting temporal and spatial isolation in a hypervisor for ARM multicore platforms. In 2018 IEEE International Conference on Industrial Technology (ICIT’18). 1651–1657. DOI:
[32]
Tiago Mück, Antonio A. Fröhlich, Giovani Gracioli, Amir M. Rahmani, João Gabriel Reis, and Nikil Dutt. 2018. CHIPS-AHOy: A predictable holistic cyber-physical hypervisor for MPSoCs. In Proceedings of the 18th International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation (SAMOS’18). Association for Computing Machinery, New York, NY, 73–80. DOI:
[33]
Anup Patel, Mai Daftedar, Mohamed Shalan, and M. Watheq El-Kharashi. 2015. Embedded hypervisor xvisor: A comparative analysis. In 2015 23rd Euromicro International Conference on Parallel, Distributed, and Network-based Processing. 682–691. DOI:
[34]
Rodolfo Pellizzoni, Emiliano Betti, Stanley Bak, Gang Yao, John Criswell, Marco Caccamo, and Russell Kegley. 2011. A predictable execution model for COTS-based embedded systems. In 2011 17th IEEE Real-time and Embedded Technology and Applications Symposium. 269–279. DOI:
[35]
Ralf Ramsauer, Jan Kiszka, Daniel Lohmann, and Wolfgang Mauerer. 2017. Look mum, no VM exits! (Almost). In Proceedings of the 13th Annual Workshop on Operating Systems Platforms for Embedded Real-time Applications (OSPERT’17). http://arxiv.org/abs/1705.06932.
[36]
Juan M. Rivas, Joël Goossens, Xavier Poczekajlo, and Antonio Paolillo. 2019. Implementation of memory centric scheduling for COTS multi-core real-time systems. In 31st Euromicro Conference on Real-time Systems (ECRTS’19)(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 133), Sophie Quinton (Ed.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 7:1–7:23. DOI:
[37]
Shahin Roozkhosh, Denis Hoornaert, and Renato Mancuso. 2022. CAESAR: Coherence-aided elective and seamless alternative routing via on-chip FPGA. In 2022 IEEE Real-time Systems Symposium (RTSS’22). 356–369. DOI:
[38]
Shahin Roozkhosh, Denis Hoornaert, Ju Hyoung Mun, Tarikul Islam Papon, Ulrich Drepper, Renato Mancuso, and Manos Athanassoulis. 2023. Relational memory: Native in-memory accesses on rows and columns. In 2023 International Conference on Extending Database Technology (EDBT’23). DOI:
[39]
Shahin Roozkhosh and Renato Mancuso. 2020. The potential of programmable logic in the middle: Cache bleaching. In 2020 IEEE Real-time and Embedded Technology and Applications Symposium (RTAS’20). 296–309. DOI:
[40]
Benjamin Rouxel, Steven Derrien, and Isabelle Puaut. 2017. Tightening contention delays while scheduling parallel applications on multi-core architectures. ACM Trans. Embed. Comput. Syst. 16, 5s, Article 164 (Sept.2017), 20 pages. DOI:
[41]
Benjamin Rouxel, Stefanos Skalistis, Steven Derrien, and Isabelle Puaut. 2019. Hiding communication delays in contention-free execution for SPM-based multi-core architectures. In 31st Euromicro Conference on Real-time Systems (ECRTS’19)(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 133), Sophie Quinton (Ed.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 25:1–25:24. DOI:
[42]
Gero Schwäricke, Tomasz Kloda, Giovani Gracioli, Marko Bertogna, and Marco Caccamo. 2020. Fixed-priority memory-centric scheduler for COTS-based multiprocessors. In 32nd Euromicro Conference on Real-time Systems (ECRTS’20)(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 165), Marcus Völp (Ed.). Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 1:1–1:24. DOI:
[43]
Parul Sohal, Rohan Tabish, Ulrich Drepper, and Renato Mancuso. 2020. E-WarP: A system-wide framework for memory bandwidth profiling and management. In 2020 IEEE Real-time Systems Symposium (RTSS’20). 345–357. DOI:
[44]
Parul Sohal, Rohan Tabish, Ulrich Drepper, and Renato Mancuso. 2022. Profile-driven memory bandwidth management for accelerators and CPUs in QoS-enabled platforms. Real-time Syst. 58, 3 (Sept.2022), 235–274. DOI:
[45]
Muhammad R. Soliman, Giovani Gracioli, Rohan Tabish, Rodolfo Pellizzoni, and Marco Caccamo. 2019. Segment streaming for the three-phase execution model: Design and implementation. In 2019 IEEE Real-time Systems Symposium (RTSS’19). 260–273. DOI:
[46]
Muhammad R. Soliman and Rodolfo Pellizzoni. 2019. PREM-based optimal task segmentation under fixed priority scheduling. In 31st Euromicro Conference on Real-time Systems (ECRTS’19)(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 133), Sophie Quinton (Ed.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 4:1–4:23. DOI:
[47]
The KITTI Vision Benchmark Suite. 2020. KITTI. http://www.cvlibs.net/datasets/kitti/.
[48]
Rohan Tabish, Renato Mancuso, Saud Wasly, Ahmed Alhammad, Sujit S. Phatak, Rodolfo Pellizzoni, and Marco Caccamo. 2016. A real-time scratchpad-centric os for multi-core embedded systems. In 2016 IEEE Real-time and Embedded Technology and Applications Symposium (RTAS’16). IEEE, 1–11. DOI:
[49]
Rohan Tabish, Renato Mancuso, Saud Wasly, Rodolfo Pellizzoni, and Marco Caccamo. 2019. A real-time scratchpad-centric OS with predictable inter/intra-core communication for multi-core embedded systems. Real-time Systems 55, 4 (2019), 850–888. DOI:
[50]
Prathap Kumar Valsan, Heechul Yun, and Farzad Farshchi. 2016. Taming non-blocking caches to improve isolation in multicore real-time systems. In 2016 IEEE Real-time and Embedded Technology and Applications Symposium (RTAS’16). 1–12. DOI:
[51]
Sravanthi Kota Venkata, Ikkjin Ahn, Donghwan Jeon, Anshuman Gupta, Christopher Louie, Saturnino Garcia, Serge Belongie, and Michael Bedford Taylor. 2009. SD-VBS: The San Diego vision benchmark suite. In 2009 IEEE International Symposium on Workload Characterization (IISWC’09). 55–64. DOI:
[52]
Bryan C. Ward, Jonathan L. Herman, Christopher J. Kenna, and James H. Anderson. 2013. Outstanding paper award: Making shared caches more predictable on multicore platforms. In 2013 25th Euromicro Conference on Real-time Systems. 157–167. DOI:
[53]
Saud Wasly and Rodolfo Pellizzoni. 2014. Hiding memory latency using fixed priority scheduling. In 2014 IEEE 19th Real-time and Embedded Technology and Applications Symposium (RTAS’14). 75–86. DOI:
[54]
Jack Whitham and Neil C. Audsley. 2012. Explicit reservation of local memory in a predictable, preemptive multitasking real-time system. In 2012 IEEE 18th Real-time and Embedded Technology and Applications Symposium. 3–12. DOI:
[55]
Jack Whitham, Neil C. Audsley, and Robert I. Davis. 2014. Explicit reservation of cache memory in a predictable, preemptive multitasking real-time system. ACM Trans. Embed. Comput. Syst. 13, 4s, Article 120 (Apr.2014), 25 pages. DOI:
[56]
Jack Whitham, Robert I. Davis, Neil C. Audsley, Sebastian Altmeyer, and Claire Maiza. 2012. Investigation of scratchpad memory for preemptive multitasking. In 2012 IEEE 33rd Real-time Systems Symposium. 3–13. DOI:
[57]
Xilinx. 2019. Zynq UltraScale+ Device - Technical Reference Manual. https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf.
[58]
Meng Xu, Linh Thi, Xuan Phan, Hyon-Young Choi, and Insup Lee. 2017. vCAT: Dynamic cache management using CAT virtualization. In 2017 IEEE Real-time and Embedded Technology and Applications Symposium (RTAS’17). 211–222. DOI:
[59]
Gang Yao, Rodolfo Pellizzoni, Stanley Bak, Emiliano Betti, and Marco Caccamo. 2012. Memory-centric scheduling for multicore hard real-time systems. Real-time Systems 48, 6 (Nov.2012), 681–715. DOI:
[60]
Gang Yao, Rodolfo Pellizzoni, Stanley Bak, Heechul Yun, and Marco Caccamo. 2016. Global real-time memory-centric scheduling for multicore systems. IEEE Trans. Comput. 65, 9 (2016), 2739–2751. DOI:
[61]
Ying Ye, Richard West, Jingyi Zhang, and Zhuoqun Cheng. 2016. MARACAS: A real-time multicore VCPU scheduling framework. In 2016 IEEE Real-time Systems Symposium (RTSS’16). 179–190. DOI:
[62]
Heechul Yun, Renato Mancuso, Zheng-Pei Wu, and Rodolfo Pellizzoni. 2014. PALLOC: DRAM bank-aware memory allocator for performance isolation on multicore platforms. In 2014 IEEE 19th Real-time and Embedded Technology and Applications Symposium (RTAS’14). 155–166. DOI:
[63]
Heechul Yun, Gang Yao, Rodolfo Pellizzoni, Marco Caccamo, and Lui Sha. 2013. MemGuard: Memory bandwidth reservation system for efficient performance isolation in multi-core platforms. In 2013 IEEE 19th Real-time and Embedded Technology and Applications Symposium (RTAS’13). 55–64. DOI:
[64]
Alfons Crespo, Patricia Balbastre, José Simó, Javier Coronel, Daniel Gracia Pérez, and Philippe Bonnot. 2018. Hypervisor-based multicore feedback control of mixed-criticality systems. IEEE Access 6, (2018), 50627–50640. DOI:

Cited By

View all
  • (2024)Implementation and Evaluation of MemGuard in the Bao Hypervisor2024 XIV Brazilian Symposium on Computing Systems Engineering (SBESC)10.1109/SBESC65055.2024.10771909(1-6)Online publication date: 26-Nov-2024
  • (2024)Minimizing cache usage with fixed-priority and earliest deadline first schedulingReal-Time Systems10.1007/s11241-024-09423-760:4(625-664)Online publication date: 1-Dec-2024
  • (2023)CVA6 RISC-V Virtualization: Architecture, Microarchitecture, and Design Space ExplorationIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.330283731:11(1713-1726)Online publication date: 28-Aug-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Embedded Computing Systems
ACM Transactions on Embedded Computing Systems  Volume 22, Issue 3
May 2023
519 pages
ISSN:1539-9087
EISSN:1558-3465
DOI:10.1145/3592782
  • Editor:
  • Tulika Mitra
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

Publication History

Published: 13 May 2023
Online AM: 14 March 2023
Accepted: 04 March 2023
Revised: 03 March 2023
Received: 22 November 2021
Published in TECS Volume 22, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Mixed-criticality real-time systems
  2. heterogeneous multiprocessor systems-on-chip
  3. schedulability analysis

Qualifiers

  • Research-article

Funding Sources

  • National Science Foundation and ONR
  • Red Hat Research program
  • Alexander von Humboldt Professorship endowed by the German Federal Ministry of Education and Research
  • Fundação de Desenvolvimento da Pesquisa - Fundep

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)97
  • Downloads (Last 6 weeks)8
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Implementation and Evaluation of MemGuard in the Bao Hypervisor2024 XIV Brazilian Symposium on Computing Systems Engineering (SBESC)10.1109/SBESC65055.2024.10771909(1-6)Online publication date: 26-Nov-2024
  • (2024)Minimizing cache usage with fixed-priority and earliest deadline first schedulingReal-Time Systems10.1007/s11241-024-09423-760:4(625-664)Online publication date: 1-Dec-2024
  • (2023)CVA6 RISC-V Virtualization: Architecture, Microarchitecture, and Design Space ExplorationIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.330283731:11(1713-1726)Online publication date: 28-Aug-2023

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media