Energy and Loss-aware Selective Updating for SplitFed Learning with Energy Harvesting-Powered Devices

Abstract

SplitFed learning (SFL) is a promising data-privacy preserving decentralized learning framework for IoT devices that has low computation requirement but high communication overhead. To reduce the communication overhead, we present a selective model update method that sends/receives activations/gradients only in selected epochs. However for IoT devices that are powered by harvested energy, the client-side model computations can take place only when the harvested energy can support it. So in this paper, we propose an energy+loss-aware selective updating method for SFL systems where the client-side model is updated based on both the clients’ energy and loss changes. When all clients have the same energy harvesting capability, we show that the proposed method can save energy by 43.7% to 80.5% with 0.5% drop in accuracy compared to an energy-aware method for VGG11 and ResNet20 models on CIFAR-10 and CIFAR-100 datasets. When the energy harvesting capability of the clients are different, the proposed method can save energy by up to 28.8% to 70.0% with 0.5% drop in accuracy.

Appendix

1.1 Energy Computation

Computation Energy

The overall client computation energy during training is a function of the number of epochs of computing forward and backward propagation:

$$\begin{aligned} E^{total}_{comp} = T_{act,k} \times E_{comp\_for} + T_{grad,k} \times E_{comp\_back} \end{aligned}$$

(8)

where $T_{act,k}$ and $T_{grad,k}$ is the number of epochs in which client k computes the forward propagation and backward propagation, respectively. $E_{comp\_for}$ is the forward computation energy per epoch, and $E_{comp\_back}$ is the backward computation energy per epoch. $E_{comp\_for}$ and $E_{comp\_back}$ are a function of the number of samples processed by client k, the number of operations in the forward and backward computations and the energy efficiency of the device:

$$\begin{aligned} E_{comp\_for} = n_k \times {OP_{comp\_for}} \times {e_{comp}} \end{aligned}$$

(9)

$$\begin{aligned} E_{comp\_back} = n_k \times {OP_{comp\_back}} \times {e_{comp}} \end{aligned}$$

(10)

where $n_k$ is the number of samples in client k. $OP_{comp\_for}$ and $OP_{comp\_back}$ are the number of operations per sample necessary for forward and backward propagation. The computation operations of backward propagation is approximately twice that of forward propagation [37]. $e_{comp}$ (Joule per operation) is the computation energy-efficiency of client-side devices.

Consider the SFL training on VGG11 with 3 convolution layers in client side with $n_k= 2500$. The computation overhead of the client side model is $OP_{comp\_for}=0.8\times 10^{-4}$ TOP/sample and $OP_{comp\_back}=1.6\times 10^{-4}$ TOP/sample. When $\tau _k=20$, the client-side model computes back propagation for 19 epochs, specifically in epochs 1-10, 21, 41,..., 181, thus $T_{grad,k}=19$. The clients need to compute forward propagation for a total of 29 epochs, specifically in epochs 1-11, 21, 22, 41, 42,...,181, 182, thus $T_{act,k}=29$. We assume the client-side devices is DNN accelerator with energy consumption of 1.5TOPs/W, that is 0.67 J/TOP [32]. As a result, $E_{comp\_for}=0.13$ J, $E_{comp\_back}=0.27$ J, $E^{total}_{comp}=10$ J.

Communication Energy

The communication energy is a function of the number of epochs of sending activations and receiving gradients:

$$\begin{aligned} E^{total}_{comm} = T_{act,k} \times E_{comm\_{act}} + T_{grad,k} \times E_{comm\_{grad}} \end{aligned}$$

(11)

$E_{comm\_{act}}$ and $E_{comm\_{grad}}$ is the communication energy per epoch for sending activations and receiving gradients, respectively. And they are the function of the number of samples in client k, and the size of activations and gradients:

$$\begin{aligned} E_{comm\_{act}} = n_k \times act\_size \times {e_{TX}} \end{aligned}$$

(12)

$$\begin{aligned} E_{comm\_{grad}} = n_k \times grad\_size \times {e_{RX}} \end{aligned}$$

(13)

where $T_{act,k}$ and $T_{grad,k}$ is the number of epochs of client k to send activations and receive gradients, $act\_size$ and $grad\_size$ (bits) is the size in bits of activations and gradients per sample, and $n_k$ is the number of samples in client k. $e_{TX}$ and $e_{RX}$ (J/bit) is the energy efficiency of transmitting and receiving data.

For the same example of SFL training on VGG11 with 3 convolution layers in client side with $n_k= 2500$, the size of activations/gradients of client side model is $256\times 8\times 8\times 32\text {bits}=0.5243$ Mbits per sample. When $\tau _k=20$, $T_{grad,k}=19$ and $T_{act,k}=29$, as mentioned above. We calculate the energy consumption of three different communication protocols:

• Bluetooth with 1Mbps@10mW [33]: $e_{TX}$ = $e_{RX}$ = $10^{-8}$ J/bit, $E_{comm\_{act}}=13.1$ J, $E_{comm\_{grad}}=13.1$ J, $E^{total}_{comm}=628$ J;

• LoRaWan with 0.05Mbps@60mW [34]: $e_{TX}$ = $e_{RX}$ = $1.2\times 10^{-6}$ J/bit, $E_{comm\_{act}}=1573$ J, $E_{comm\_{grad}}=1573$ J, $E^{total}_{comm}=75504$ J;

• NB-IoT with 0.25Mbps@200mW [35]: $e_{TX}$ = $e_{RX}$ = $0.8\times 10^{-6}$ J/bit, $E_{comm\_{act}}=1049$ J, $E_{comm\_{grad}}=1049$ J, $E^{total}_{comm}=50352$ J;