Loading web-font TeX/Main/Regular
Trainer: An Energy-Efficient Edge-Device Training Processor Supporting Dynamic Weight Pruning | IEEE Journals & Magazine | IEEE Xplore

Trainer: An Energy-Efficient Edge-Device Training Processor Supporting Dynamic Weight Pruning


Abstract:

Transfer learning, which transfers knowledge from source datasets to target datasets, is practical for adaptive deep neural network (DNN) applications. When considering u...Show More

Abstract:

Transfer learning, which transfers knowledge from source datasets to target datasets, is practical for adaptive deep neural network (DNN) applications. When considering user privacy and communication bandwidth issues, edge devices’ training is essential for transfer learning. Nevertheless, training requires repeating feedforward (FF), backpropagation (BP), and weight gradient (WG) millions of times, introducing prohibitive computation for edge devices. A promising method to reduce training computation is sparse DNN training (SDT), which dynamically prunes weights during training iterations and performs FF, BP, and WG only with unpruned weights. However, SDT suffers implicit redundancy and reuse imbalance for convolution layers. Besides, it turns bottlenecks into batch normalization (BN) layers. Therefore, it is challenging to achieve energy-efficient SDT computing. This article proposes a processor, Trainer, solving the above challenges with three features. First, a speculation mechanism removes implicit redundant operations, which have nonzeros’ input, weight, or output, but are ineffective for training. Second, a dynamic sparsity adaptive dataflow tackles the reuse imbalance, improving energy efficiency (EE) for dynamic sparse convolution in SDT. Third, a computational dependence decoupled BN unit eliminates BN’s repeated data access to reduce training energy and time. Trainer is fabricated in 28-nm CMOS technology and occupies 20.96 mm2 of area. It achieves a peak EE of 173.28TFLOPS/W@FP16 (276.55TFLOPS/W@FP8) for a 90% activation sparsity and 90% weight sparsity. The sparsity to EE conversion ratio is 80.9, outperforming the previous work by 1.55 \times . When training a ResNet18 model with SDT, Trainer reduces energy by 2.23 \times and time by 1.76 \times than the state-of-the-art sparse training processor.
Published in: IEEE Journal of Solid-State Circuits ( Volume: 57, Issue: 10, October 2022)
Page(s): 3164 - 3178
Date of Publication: 20 May 2022

ISSN Information:

Funding Agency:


References

References is not available for this document.