A Deep Learning Approach for the Design of Narrow Transition-Band FIR Filter

Abstract

Deep neural network (DNN), being an important member of machine learning family, has been employed to serve a wide range of applications in the area of signal and image processing like pattern recognition, speech recognition, language processing, image segmentation, etc. To this aim, this paper concentrates on the design of a narrow transition-band finite impulse response (FIR) filter with the aid of back-propagation-based deep learning approach. The proposed deep learning-based approach offers a unified design framework for a variety of FIR filters. Convergence behaviour of the proposed algorithm has been proved analytically in situations when weights between adjacent layers are updated continuously. Simulation results have shown the frequency response characteristics of several FIR filters with narrow transition-band, designed with the help of proposed approach. Advantage of our design strategy has also been established in terms of magnitude response over a number of state-of-the-art techniques of recent interest. Simulation results have shown noticeable improvement in terms of transition bandwidth when compared with few existing works. Designed filter is subsequently implemented on Altera’s Cyclone IV field programmable gate array (FPGA) chip, and hardware efficiency of the suggested design has strongly been established by correlating its hardware cost with many of the state-of-the-art FIR filters.

Appendices

Appendix A

Define a Lyapunov function:

$$\begin{aligned}&V_{k}=\frac{1}{2}E_{k}^2\Rightarrow \varDelta V_{k}=\frac{1}{2}E_{k+1}^2-\frac{1}{2}E_{k}^2 \end{aligned}$$

(27)

$$\begin{aligned}&Now,~~ E_{k+1}=E_{k}+\varDelta E_{k}=E_{k}+\frac{d E_{k}}{d W_{qr}}\varDelta W_{qr} ~,~where \nonumber \\&\quad \varDelta W_{qr}=-\eta _{1}\frac{\partial \mathscr {J}}{\partial W_{qr}}=\eta _{1}\delta _{r}.H_{q};~\nonumber \\&\frac{d E_{k}}{d W_{qr}}=-H_{q}\nonumber \\&Therefore, ~E_{k+1}=E_{k}-\eta _{1}\delta _{r}(H_{q})^2~~~~\forall q \in \mathscr {Q} \end{aligned}$$

(28)

According to Eqs. (27) and (28), we can write

$$\begin{aligned} \varDelta V_{k}\!=\!\frac{1}{2}\Bigg [E_{k}-\eta _{1}\delta _{r}(H_{q})^2\Bigg ]^2-\frac{1}{2}E_{k}^2=\Bigg [-E_{k}\eta _{1}\delta _{r}(H_{q})^2+\frac{1}{2}\eta _{1}^2\delta _{r}^2(H_{q})^4\Bigg ]\nonumber \\=\delta _{r}^2(H_{q})^2\Bigg \{-\eta _{1}+\frac{\eta _{1}^2}{2}(H_{q})^2\Bigg \}~~~~\forall q \in \mathscr {Q}\qquad \end{aligned}$$

(29)

In order to let the algorithm be convergent, $\varDelta V_{k}<0$ (i.e. $\delta _{r}^2(H_{q})^2>0$). Therefore, we have

$$\begin{aligned} -\eta _{1}+\frac{\eta _{1}^2}{2}(H_{q})^2<0~~~~\forall q \in \mathscr {Q} \end{aligned}$$

(30)

As, learning rate $\eta _{1}~>0$, therefore we have

$$\begin{aligned} 0<\eta _{1}<\frac{2}{H_{q}^2}~~~~\forall q \in \mathscr {Q} \end{aligned}$$

(31)

When $\eta _{1}$ satisfies Eq. (31), $\varDelta V_{k}<0$, the proposed deep neural network is stable and the given algorithm is convergent to its global minimum; hence, the theorem is proved completely.

Appendix B

Define a Lyapunov function:

$$\begin{aligned}&V_{k}=\frac{1}{2}E_{k}^2\Rightarrow \varDelta V_{k}=\frac{1}{2}E_{k+1}^2-\frac{1}{2}E_{k}^2 \end{aligned}$$

(32)

$$\begin{aligned}&Now,~~ E_{k+1}=E_{k}+\varDelta E_{k}=E_{k}+\frac{d E_{k}}{d W_{pq}}\varDelta W_{pq}~,~where~\varDelta W_{pq}=-\eta _{2}\frac{\partial \mathscr {J}}{\partial W_{pq}}=\eta _{2}H_{p}~\nonumber \\&\sum _{\forall i \in \mathscr {Q}}\delta _{i}W_{ir}H_{i}(1-H_{i});\frac{d E_{k}}{d W_{pq}}=-\sum _{\forall i \in \mathscr {Q}}W_{ir}H_{i}(1-H_{i})H_{p}\nonumber \\&Therefore,~E_{k+1}=E_{k}-\eta _{2}H_{p}^2~\sum _{\forall i \in \mathscr {Q}} \delta _{i}W_{ir}^2(H_{i}(1-H_{i}))^2~~~~ \forall q \in \mathscr {Q}~,~\forall p \in \mathscr {P} \end{aligned}$$

(33)

According to Eqs. (32) and (33), we can write

$$\begin{aligned}&\varDelta V_{k}=\frac{1}{2}\Bigg [E_{k}-\eta _{2}H_{p}^2~\sum _{\forall i \in \mathscr {Q}} \delta _{i}W_{ir}^2(H_{i}(1-H_{i}))^2\Bigg ]^2-\frac{1}{2}E_{k}^2=\Bigg [-E_{k}\eta _{2}H_{p}^2~\nonumber \\&\sum _{\forall i \in \mathscr {Q}} \delta _{i}W_{ir}^2(H_{i}(1-H_{i}))^2+\frac{1}{2}\eta _{2}^2H_{p}^4~\sum _{\forall i \in \mathscr {Q}} \delta _{i}^2W_{ir}^4(H_{i}(1-H_{i}))^4\Bigg ]\nonumber \\&\quad =H_{p}^2\sum _{\forall i \in \mathscr {Q}}\delta _{i}^2W_{ir}^2(H_{i}(1-H_{i}))^2\Bigg \{-\eta _{2}+\frac{\eta _{2}^2}{2}H_{p}^2~\sum _{\forall i \in \mathscr {Q}}W_{ir}^2(H_{i}(1-H_{i}))^2\Bigg \}~~~~\nonumber \\&\qquad \qquad \qquad \forall q \in \mathscr {Q}~,~\forall p \in \mathscr {P}\nonumber \\ \end{aligned}$$

(34)

In order to let the algorithm be convergent, $\varDelta V_{k}<0$ (i.e. $H_{p}^2\sum _{\forall i \in \mathscr {Q}}\delta _{i}^2W_{ir}^2(H_{i}(1-H_{i}))^2>0$). Therefore, we have

$$\begin{aligned} -\eta _{2}+\frac{\eta _{2}^2}{2}H_{p}^2~\sum _{\forall i \in \mathscr {Q}}W_{ir}^2(H_{i}(1-H_{i}))^2<0~~\forall q \in \mathscr {Q}~,~\forall p \in \mathscr {P} \end{aligned}$$

(35)

As, learning rate $\eta _{2}~>0$, therefore we get

$$\begin{aligned} 0<\eta _{2}<\frac{2}{H_{p}^2~\sum _{\forall i \in \mathscr {Q}}W_{ir}^2(H_{i}(1-H_{i}))^2}~~\forall q \in \mathscr {Q}~,~\forall p \in \mathscr {P} \end{aligned}$$

(36)

When $\eta _{2}$ satisfies Eq. (36), $\varDelta V_{k}<0$, the proposed deep neural network is stable and the algorithm converges towards its global minimum.

Appendix C

Define a Lyapunov function:

$$\begin{aligned}&V_{k}=\frac{1}{2}E_{k}^2\Rightarrow \varDelta V_{k}=\frac{1}{2}E_{k+1}^2-\frac{1}{2}E_{k}^2 \end{aligned}$$

(37)

$$\begin{aligned}&Now,~~ E_{k+1}=E_{k}+\varDelta E_{k}=E_{k}+\frac{d E_{k}}{d W_{np}}\varDelta W_{np}~,~where~\varDelta W_{np}=-\eta _{3}\frac{\partial \mathscr {J}}{\partial W_{np}}=\eta _{3}H_{n}~\nonumber \\&\sum _{\forall i \in \mathscr {Q}}\delta _{i}W_{ir}H_{i}(1-H_{i})\sum _{\forall j \in \mathscr {P}}W_{jq}H_{j}(1-H_{j})~;\nonumber \\&\frac{d E_{k}}{d W_{np}}=-\sum _{\forall i \in \mathscr {Q}}~\sum _{\forall j \in \mathscr {P}}W_{jq}W_{ir}H_{i}(1-H_{i})H_{j}(1-H_{j})H_{n}\nonumber \\&Therefore,~E_{k+1}=E_{k}-\eta _{3}H_{n}^2~\sum _{\forall i \in \mathscr {Q}} \delta _{i}W_{ir}^2(H_{i}(1-H_{i}))^2\sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2~~~~\nonumber \\&\forall q \in \mathscr {Q}~,~\forall n \in \mathscr {N} \end{aligned}$$

(38)

According to Eqs. (37) and (38), we can write

$$\begin{aligned}&\varDelta V_{k}=\frac{1}{2}\Bigg [E_{k}-\eta _{3}H_{n}^2~\sum _{\forall i \in \mathscr {Q}} \delta _{i}W_{ir}^2(H_{i}(1-H_{i}))^2\sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2\Bigg ]^2-\frac{1}{2}E_{k}^2\nonumber \\&=\Bigg [-E_{k}\eta _{3}H_{n}^2~\sum _{\forall i \in \mathscr {Q}} \delta _{i}W_{ir}^2(H_{i}(1-H_{i}))^2\sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2+\frac{1}{2}\eta _{3}^2H_{n}^4~\nonumber \\&\sum _{\forall i \in \mathscr {Q}} \delta _{i}^2W_{ir}^4(H_{i}(1-H_{i}))^4\sum _{\forall j \in \mathscr {P}}W_{jq}^4(H_{j}(1-H_{j}))^4\Bigg ]\nonumber \\&=H_{n}^2~\sum _{\forall i \in \mathscr {Q}}\delta _{i}^2W_{ir}^2(H_{i}(1-H_{i}))^2\sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2\Bigg \{-\eta _{3}+\frac{\eta _{3}^2}{2}H_{n}^2~\nonumber \\&\sum _{\forall i \in \mathscr {Q}}W_{ir}^2(H_{i}(1-H_{i}))^2\sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2\Bigg \}~~~~\forall q \in \mathscr {Q}~,~\forall n \in \mathscr {N}\nonumber \\ \end{aligned}$$

(39)

In order to let the algorithm be convergent, $\varDelta V_{k}<0$ (i.e. $H_{n}^2\sum _{\forall i \in \mathscr {Q}}\delta _{i}^2W_{ir}^2(H_{i}(1-H_{i}))^2\sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2>0$). Therefore, we get

$$\begin{aligned}&-\eta _{3}+\frac{\eta _{3}^2}{2}H_{n}^2~\sum _{\forall i \in \mathscr {Q}}W_{ir}^2(H_{i}(1-H_{i}))^2\nonumber \\&\qquad \sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2<0 \forall q \in \mathscr {Q}~,~\forall n \in \mathscr {N} \end{aligned}$$

(40)

As, learning rate $\eta _{3}~>0$, therefore we get

$$\begin{aligned} 0<\eta _{3}&<\frac{2}{H_{n}^2\sum _{\forall i \in \mathscr {Q}}W_{ir}^2(H_{i}(1-H_{i}))^2\sum _{\forall j \in \mathscr {P}}W_{jq}^2(H_{j}(1-H_{j}))^2}\nonumber \\&\forall q \in \mathscr {Q}~,~\forall n \in \mathscr {N} \end{aligned}$$

(41)

When $\eta _{3}$ satisfies Eq. (41), $\varDelta V_{k}<0$, the proposed deep neural network is stable and the algorithm is convergent to its global minimum.