Performance Prediction for Deep Learning Models With Pipeline Inference Strategy | IEEE Journals & Magazine | IEEE Xplore

Performance Prediction for Deep Learning Models With Pipeline Inference Strategy


Abstract:

For heterogeneous multiprocessor system-on-chips (HMPSoCs), a reasonable pipeline design can significantly improve the inference performance of deep learning (DL) models....Show More

Abstract:

For heterogeneous multiprocessor system-on-chips (HMPSoCs), a reasonable pipeline design can significantly improve the inference performance of deep learning (DL) models. The pipeline design optimization can be modeled as a search problem where an accurate prediction model can efficiently speed up the search process. However, the performance prediction of DL models for the pipeline inference strategy is challenging because of the interlayer effect, inference details, and variety of model structures. In this article, we propose TPPNet, a transformer-based model for predicting the inference performance of various DL models with the pipeline inference strategy. TPPNet represents the DL model as an execution sequence with operators and hardware details to extract the hidden factors between layers. Moreover, we apply the multitask learning (MTL) method to accurately predict throughput and latency metrics by constructing a predictive model. To the best of our knowledge, this is the first study dedicated to pipeline inference performance prediction for the DL model on HMPSoCs. We evaluate TPPNet on six well-known DL models using RK3399. The experimental outcomes affirm the high accuracy of TPPNet and its capability to significantly reduce the time overhead associated with pipeline exploration.
Published in: IEEE Internet of Things Journal ( Volume: 11, Issue: 2, 15 January 2024)
Page(s): 2964 - 2978
Date of Publication: 11 July 2023

ISSN Information:

Funding Agency:


References

References is not available for this document.