Appendix 1
The average SER for an M-ary PSK modulation scheme is given by
$$P_{e} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)p_{\beta } \left( \beta \right)d\beta }$$
(15)
where Q(.) is the Gaussian Q-function which is given by
$$Q(x) = \frac{1}{\pi }\int\limits_{0}^{{\left( {M - 1} \right)\pi /M}} {e^{{\left( {\frac{{ - x^{2} }}{{2\sin^{2} \theta }}} \right)}} } d\theta$$
(16)
Substituting (16) in (15),
$$\begin{aligned} P_{e} & = \int\limits_{0}^{\infty } {\frac{1}{\pi }\int\limits_{0}^{{\left( {M - 1} \right)\pi /M}} {e^{{\left( {\frac{{ - b^{2} \beta }}{{2\sin^{2} \theta }}} \right)}} } } d\theta \,p_{\beta } \left( \beta \right)\,d\beta \\ P_{e} & = \frac{1}{\pi }\int\limits_{0}^{{\left( {M - 1} \right)\pi /M}} {\left[ {\int\limits_{0}^{\infty } {e^{{^{{\left( {\frac{{ - b^{2} \beta }}{{2\sin^{2} \theta }}} \right)}} }} } p_{\beta } \left( \beta \right)d\beta } \right]} d\theta \\ \end{aligned}$$
(17)
where b is a modulation constant which depends on the specific modulation and detection combination. For M-ary PSK, b is given by,
$$b^{2} = 2\sin^{2} \frac{\pi }{M}$$
(18)
For Rayleigh fading channel, the probability density function (pdf) of instantaneous SNR is of the form [45]
$$p_{\beta } \left( \beta \right) = \frac{1}{{\bar{\beta }}}e^{{\left( { - \frac{\beta }{{\bar{\beta }}}} \right)}} ,\beta \ge 0$$
(19)
where \(\bar{\beta }\) is the average SNR which is given by
$$\bar{\beta } = \frac{{E_{s} }}{{N_{0} }}$$
(20)
Substituting (19) in (17) and simplifying gives the average SER of M-ary PSK modulation over a Rayleigh fading channel which is given by [45]
$$P_{e} = \left[ {\frac{M - 1}{M}\left\{ {1 - \sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}{{1 + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}} \frac{M}{{\left( {M - 1} \right)\pi }}\left[ {\frac{\pi }{2} + \tan^{ - 1} \left( {\sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}{{1 + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}\cot \frac{\pi }{M}} } \right)} \right]} \right\}} \right]$$
(21)
For Nakagami-m fading channel, the pdf of instantaneous SNR is of the form [45]
$$p_{\beta } \left( \beta \right) = \frac{{m^{m} \beta^{m - 1} }}{{\bar{\beta }^{m} \varGamma (m)}}e^{{\left( { - \frac{m\beta }{{\bar{\beta }}}} \right)}} ,\beta \ge 0$$
(22)
where m is the Nakagami-m fading factor which ranges from ½ to \(\infty\) and Γ(•) is the Gamma function. Substituting (22) in (17) gives the average SER of M-ary PSK modulation over a Nakagami-m fading channel which is given by [45]
$$P_{e} = \left[ {\frac{M - 1}{M} - \frac{1}{\pi }\sqrt {\frac{{b^{2} \bar{\beta }/2m}}{{1 + (b^{2} \bar{\beta }/2m)}}} \left\{ \begin{aligned} \left( {\frac{\pi }{2} + \tan^{ - 1} \alpha } \right)\sum\limits_{g = 0}^{m - 1} {\left( \begin{aligned} 2g \hfill \\ \,g \hfill \\ \end{aligned} \right)\frac{1}{{\left[ {4\left( {1 + b^{2} \bar{\beta }/2m} \right)} \right]^{g} }}} \hfill \\ \quad +\sin (\tan^{ - 1} \alpha )\sum\limits_{g = 1}^{m - 1} {\sum\limits_{i = 1}^{g} {\frac{{T_{ig} }}{{\left( {1 + b^{2} \bar{\beta }/2m} \right)^{g} }}} \left[ {\cos \left( {\tan^{ - 1} \alpha } \right)} \right]^{2(g - i) + 1} } \hfill \\ \end{aligned} \right\}} \right]$$
(23)
where
$$\alpha\,\triangleq\,\sqrt {\frac{{b^{2} \bar{\beta }/2m}}{{1 + b^{2} \bar{\beta }/2m}}} \cot \frac{\pi }{M}$$
(24)
$$T_{ig} \,\triangleq\, \frac{{\left( \begin{aligned} 2g \hfill \\ \,g \hfill \\ \end{aligned} \right)}}{{\left( \begin{aligned} 2(g - i) \hfill \\ \,\,\,\,g - i \hfill \\ \end{aligned} \right)4^{i} \left[ {2\left( {g - i} \right) + 1} \right]}}$$
(25)
Appendix 2
2.1 Scheme 1
The pdf of instantaneous SNR for the antenna selection scheme 1 is given by [40]
$$p_{\beta } \left( \beta \right) = \frac{{N\left( {N - 1} \right)}}{{\frac{{2\;E_{s} }}{{N_{0} }}}}\left[ \begin{aligned} 2\;\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ \;\;\;g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}\left\{ {e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} - \frac{2}{{2 + g - g\rho^{2} }}\;e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}\;\frac{{\left( {g + 2} \right)}}{{(g + 2 - g\rho^{2} )}}}} } \right\} \hfill \\ \quad +\left( {1 - \rho^{2} + \frac{{\rho^{2} \beta }}{{\frac{{E_{s} }}{{N_{0} }}}}} \right)\;e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} \hfill \\ \end{aligned} \right]$$
(26)
Substituting \(p_{\beta } \left( \beta \right)\) in (15)
$$P_{e} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N\left( {N - 1} \right)}}{{\frac{{2\;E_{s} }}{{N_{0} }}}}\left[ \begin{aligned} 2\;\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ \;\;\;g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}\left\{ {e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} - \frac{2}{{2 + g - g\rho^{2} }}\;e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}\;\frac{{\left( {g + 2} \right)}}{{(g + 2 - g\rho^{2} )}}}} } \right\} \hfill \\ \quad +\left( {1 - \rho^{2} + \frac{{\rho^{2} \beta }}{{\frac{{E_{s} }}{{N_{0} }}}}} \right)\;e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} \hfill \\ \end{aligned} \right]} d\beta$$
(27)
On splitting the integral in (27) into 4 parts,
$$P_{e} = P_{1} - P_{2} + P_{3} + P_{4}$$
(28)
where
$$P_{1} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\left( {\frac{{N\left( {N - 1} \right)}}{{\frac{{2\;E_{s} }}{{N_{0} }}}}2\;\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} } \right)} d\beta$$
(29)
$$P_{2} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\left( {\frac{{N\left( {N - 1} \right)}}{{\frac{{2\;E_{s} }}{{N_{0} }}}}2\;\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}\frac{2}{{2 + g - g\rho^{2} }}\;e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}\;\frac{{\left( {g + 2} \right)}}{{g + 2 - g\rho^{2} }}}} } \right)} d\beta$$
(30)
$$P_{3} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N\left( {N - 1} \right)}}{{\frac{{2\;E_{s} }}{{N_{0} }}}}\left( {1 - \rho^{2} } \right)e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} } d\beta$$
(31)
$$P_{4} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N\left( {N - 1} \right)}}{{\frac{{2\;E_{s} }}{{N_{0} }}}}\frac{{\rho^{2} \beta }}{{\frac{{E_{s} }}{{N_{0} }}}}e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} } d\beta$$
(32)
Considering P1 in (29),
$$P_{1} = N\left( {N - 1} \right)\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}\int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{1}{{\frac{{E_{s} }}{{N_{0} }}}}e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} } d\beta$$
(33)
Rewriting (33) similar to (19) and comparing with (21) gives
$$P_{1} = N\left( {N - 1} \right)\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}\left[ {\frac{M - 1}{M}\left\{ \begin{aligned} 1 - \sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}{{1 + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}} \frac{M}{{\left( {M - 1} \right)\pi }} \hfill \\ \left[ {\frac{\pi }{2} + \tan^{ - 1} \left( {\sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}{{1 + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)/2}}\cot \frac{\pi }{M}} } \right)} \right] \hfill \\ \end{aligned} \right\}} \right]$$
(34)
Considering P2 in (30),
$$P_{2} = N\left( {N - 1} \right)\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} 2}}{g(g + 2)}\int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{\left( {g + 2} \right)}}{{\left( {g + 2 - g\rho^{2} } \right)\frac{{E_{s} }}{{N_{0} }}}}} \;e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}\;\frac{{\left( {g + 2} \right)}}{{(g + 2 - g\rho^{2} )}}}} d\beta$$
(35)
Rewriting (35) similar to (19) and comparing with (21) gives
$$P_{2} = N\left( {N - 1} \right)\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} 2}}{g(g + 2)}\left[ {\frac{M - 1}{M}\left\{ \begin{aligned} 1 - \sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)\left( {g + 2 - g\rho^{2} } \right)}}{{2(g + 2) + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)(g + 2 - g\rho^{2} )}}} \frac{M}{{\left( {M - 1} \right)\pi }} \hfill \\ \left[ {\frac{\pi }{2} + \tan^{ - 1} \left( {\sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)\left( {g + 2 - g\rho^{2} } \right)}}{{2(g + 2) + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)(g + 2 - g\rho^{2} )}}} \cot \frac{\pi }{M}} \right)} \right] \hfill \\ \end{aligned} \right\}} \right]$$
(36)
Considering P3 in (31),
$$P_{3} = \frac{N(N - 1)}{2}\left( {1 - \rho^{2} } \right)\int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{1}{{\frac{{E_{s} }}{{N_{0} }}}}} \;e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} d\beta$$
(37)
Rewriting (37) similar to (19) and comparing with (21) gives
$$P_{3} = \frac{N(N - 1)}{2}\left( {1 - \rho^{2} } \right)\left\{ {\frac{M - 1}{M}\left[ \begin{aligned} 1 - \sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}{{2 + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}} \frac{M}{{\left( {M - 1} \right)\pi }} \hfill \\ \left\{ {\frac{\pi }{2} + \tan^{ - 1} \left( {\sqrt {\frac{{b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}{{1 + b^{2} \left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}} \cot \frac{\pi }{M}} \right)} \right\} \hfill \\ \end{aligned} \right]} \right\}$$
(38)
Considering P4 in (32),
$$P_{4} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N\left( {N - 1} \right)}}{{\frac{{2\;E_{s} }}{{N_{0} }}}}\frac{{\rho^{2} \beta }}{{\frac{{E_{s} }}{{N_{0} }}}}e^{{\frac{ - \beta }{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}}} } d\beta$$
(39)
Rewriting (39) similar to (22) and comparing with (23) gives
$$P_{4} = \frac{N(N - 1)}{2}\rho^{2} \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{4\beta }{{\left[ {\frac{{2E_{s} }}{{N_{0} }}} \right]^{2} \varGamma (2)}}e^{{\frac{ - 2\beta }{{\left( {\frac{{2E_{s} }}{{N_{0} }}} \right)}}}} d\beta } ,\quad m = 2$$
(40)
Substituting (34), (36), (38) and (40) in (28) gives average SER expression of M-ary PSK modulation [i.e. (4)] for the antenna selection scheme 1.
2.2 Scheme 2
The pdf of instantaneous SNR for the antenna selection scheme 2 is given by [40]
$$\begin{aligned} p_{\beta } \left( \beta \right) &= \frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{array}{c} N_{1} - 1 \hfill \\ r \hfill \\ \end{array} \right)\left( \begin{array}{c} N_{2} - 1 \hfill \\ r \hfill \\ \end{array} \right)}}{{\left( {1 + r - r\rho^{2} } \right)^{2} }}} \left\{ {1 - \rho^{2} + \frac{{\beta \rho^{2} }}{{\left( {1 + r - r\rho^{2} } \right)\frac{{E_{s} }}{{N_{0} }}}}} \right\}e^{{\frac{ - \beta (1 + r)}{{\frac{{E_{s} }}{{N_{0} }}\left( {1 + r - r\rho^{2} } \right)^{2} }}}} \hfill \\ &\quad +\frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{array}{c} N_{1} - 1 \hfill \\ n \hfill \\ \end{array} \right)\left( \begin{array}{c} N_{2} - 1 \hfill \\ m \hfill \\ \end{array} \right)\left( { - 1} \right)^{m + n} }}{n - m}} } \hfill \\ &\quad \left\{ {\frac{{e^{{\frac{ - \beta (1 + m)}{{\frac{{E_{s} }}{{N_{0} }}\left( {1 + m - m\rho^{2} } \right)}}}} }}{{1 + m - m\rho^{2} }} - \frac{{e^{{\frac{ - \beta (1 + n)}{{\frac{{E_{s} }}{{N_{0} }}\left( {1 + n - n\rho^{2} } \right)}}}} }}{{1 + n - n\rho^{2} }}} \right\},\beta \ge 0 \hfill \\ \end{aligned}$$
(41)
$$P_{e} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\left[ \begin{aligned} \frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )^{2} }}} \left\{ {1 - \rho^{2} + \frac{{\beta \rho^{2} }}{{(1 + r - r\rho^{2} )\frac{{E_{s} }}{{N_{0} }}}}} \right\}e^{{\frac{ - \beta (1 + r)}{{\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )}}}} + \hfill \\ \frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{n - m}} } \left\{ {\frac{{e^{{\frac{ - \beta (1 + m)}{{\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )}}}} }}{{(1 + m - m\rho^{2} )}} - \frac{{e^{{\frac{ - \beta (1 + n)}{{\frac{{E_{s} }}{{N_{0} }}(1 + n - n\rho^{2} )}}}} }}{{(1 + n - n\rho^{2} )}}} \right\} \hfill \\ \end{aligned} \right]} d\beta$$
(42)
(42) is obtained by substituting (41) in (15). On splitting the integral in (42) into 4 parts
$$P_{e} = P_{1} + P_{2} + P_{3} - P_{4}$$
(43)
where
$$P_{1} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )^{2} }}} \left( {1 - \rho^{2} } \right)} e^{{\frac{ - \beta (1 + r)}{{\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )}}}} d\beta$$
(44)
$$P_{2} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )^{2} }}} \frac{{\beta \rho^{2} }}{{(1 + r - r\rho^{2} )\frac{{E_{s} }}{{N_{0} }}}}} e^{{\frac{ - \beta (1 + r)}{{\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )}}}} d\beta$$
(45)
$$P_{3} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{n - m}} } } \frac{{e^{{\frac{ - \beta (1 + m)}{{\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )}}}} }}{{(1 + m - m\rho^{2} )}}d\beta$$
(46)
$$P_{4} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{{N_{1} N_{2} }}{{\frac{{E_{s} }}{{N_{0} }}}}\sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{n - m}} } } \frac{{e^{{\frac{ - \beta (1 + n)}{{\frac{{E_{s} }}{{N_{0} }}(1 + n - n\rho^{2} )}}}} }}{{(1 + n - n\rho^{2} )}}d\beta$$
(47)
Considering P3 in (46)
$$P_{3} = N_{1} N_{2} \sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{{\left( {n - m} \right)\left( {1 + m} \right)}}} } \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)} \left( {1 + m} \right)\frac{{e^{{\frac{ - \beta (1 + m)}{{\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )}}}} }}{{\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )}}d\beta$$
(48)
Rewriting (48) similar to (19)
$$P_{3} = N_{1} N_{2} \sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{{\left( {n - m} \right)\left( {1 + m} \right)}}} } \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)} \frac{1}{{\overline{\beta } }}e^{{ - \frac{\beta }{{\overline{\beta } }}}} d\beta$$
(49)
where
$$\overline{\beta } = \frac{{\left( {\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )} \right)}}{{\left( {1 + m} \right)}}$$
(50)
Comparing (49) with (21) gives
$$P_{3} = N_{1} N_{2} \sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{{\left( {n - m} \right)\left( {1 + m} \right)}}} } \frac{M - 1}{M}\left\{ \begin{aligned} 1 - \sqrt {\frac{{b^{2} \frac{{\overline{\beta } }}{2}}}{{1 + b^{2} \frac{{\overline{\beta } }}{2}}}} \frac{M}{{\left( {M - 1} \right)\pi }} \hfill \\ \left[ {\frac{\pi }{2} + \tan^{ - 1} \left( {\sqrt {\frac{{b^{2} \frac{{\overline{\beta } }}{2}}}{{1 + b^{2} \frac{{\overline{\beta } }}{2}}}} \cot \frac{\pi }{M}} \right)} \right] \hfill \\ \end{aligned} \right\}$$
(51)
Similarly P4 in (47) can be simplified as
$$P_{4} = N_{1} N_{2} \sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{{\left( {n - m} \right)\left( {1 + n} \right)}}} } \frac{M - 1}{M}\left\{ \begin{aligned} 1 - \sqrt {\frac{{b^{2} \frac{{\overline{\beta } }}{2}}}{{1 + b^{2} \frac{{\overline{\beta } }}{2}}}} \frac{M}{{\left( {M - 1} \right)\pi }} \hfill \\ \left[ {\frac{\pi }{2} + \tan^{ - 1} \left( {\sqrt {\frac{{b^{2} \frac{{\overline{\beta } }}{2}}}{{1 + b^{2} \frac{{\overline{\beta } }}{2}}}} \cot \frac{\pi }{M}} \right)} \right] \hfill \\ \end{aligned} \right\}$$
(52)
where
$$\overline{\beta } = \frac{{\left( {\frac{{E_{s} }}{{N_{0} }}(1 + n - n\rho^{2} )} \right)}}{{\left( {1 + n} \right)}}$$
(53)
Considering P1 in (44)
$$P_{1} = N_{1} N_{2} \sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )}}\frac{{\left( {1 - \rho^{2} } \right)}}{{\left( {1 + r} \right)}}} \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)} \frac{{\left( {1 + r} \right)}}{{\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )}}e^{{\frac{ - \beta (1 + r)}{{\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )}}}} d\beta$$
(54)
Rewriting (54) similar to (19) and comparing with (21) gives
$$P_{1} = N_{1} N_{2} \sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )}}\frac{{\left( {1 - \rho^{2} } \right)}}{{\left( {1 + r} \right)}}} \left[ {\frac{M - 1}{M}\left\{ \begin{aligned} 1 - \sqrt {\frac{{b^{2} \frac{{\overline{\beta } }}{2}}}{{1 + b^{2} \frac{{\overline{\beta } }}{2}}}} \frac{M}{{\left( {M - 1} \right)\pi }} \hfill \\ \left[ {\frac{\pi }{2} + \tan^{ - 1} \left( {\sqrt {\frac{{b^{2} \frac{{\overline{\beta } }}{2}}}{{1 + b^{2} \frac{{\overline{\beta } }}{2}}}} \cot \frac{\pi }{M}} \right)} \right] \hfill \\ \end{aligned} \right\}} \right]$$
(55)
Rewriting (45) similar to (22) and comparing with (23) gives
$$\begin{aligned} P_{2} = N_{1} N_{2} \sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )^{2} }}\frac{{\rho^{2} }}{{\left( {1 + r} \right)^{2} }}} \hfill \\ \quad \quad \left[ {\frac{M - 1}{M} - \frac{1}{\pi }\sqrt {\frac{{b^{2} \frac{{\overline{\beta } }}{2m}}}{{1 + b^{2} \frac{{\overline{\beta } }}{2m}}}} \left\{ {\left( {\frac{\pi }{2} + \tan^{ - 1} \alpha } \right)\sum\limits_{k = 0}^{m - 1} {\left( \begin{aligned} 2k \hfill \\ \;k \hfill \\ \end{aligned} \right)\frac{1}{{\left[ {4\left( {1 + b^{2} \frac{{\overline{\beta } }}{2m}} \right)} \right]^{k} }}} + \frac{{\sin (\tan^{ - 1} \alpha )\cos \left( {\tan^{ - 1} \alpha } \right)}}{{2\left( {1 + b^{2} \frac{{\overline{\beta } }}{4}} \right)}}} \right\}} \right] \hfill \\ \end{aligned}$$
(56)
where
$$\overline{\beta } = \frac{{\left( {2\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )} \right)}}{{\left( {1 + r} \right)}}$$
(57)
Substituting (55), (56), (51) and (52) in (43) gives average SER expression of M-ary PSK modulation [i.e. (8)] for the antenna selection scheme 2.
2.3 Scheme 3
The pdf of instantaneous SNR for the antenna selection scheme 3 is given by [40]
$$p_{\beta } \left( \beta \right) = \frac{N}{2}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{{n!\left( {\frac{{E_{s} }}{{N{}_{0}}}} \right)^{k - n + 2} }}\left\{ {\frac{{\left( {k + 1} \right)!}}{{\left( {k - n + 1} \right)!}}} \right\}^{2} \frac{{\left( {\rho^{2} } \right)^{k + 1 - n} \left( {1 - \rho^{2} } \right)^{n} }}{{\left( {1 + m - m\rho^{2} } \right)^{2m - n + 3} }}\beta^{{\left( {k - n + 1} \right)}} e^{{\frac{{ - \beta \left( {1 + m} \right)}}{{\frac{{E_{s} }}{{N_{0} }}\left( {1 + m - m\rho^{2} } \right)}}}} d\beta } } }$$
(58)
Substituting (58) in (15) gives
$$\begin{aligned} P_{e} = \int\limits_{0}^{\infty } {Q\left( {b\sqrt \beta } \right)\frac{N}{2}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{{n!\left( {\frac{{E_{s} }}{{N_{0} }}} \right)^{k - n + 2} }}} } } } \left\{ {\frac{(k + 1)!}{(k - n + 1)!}} \right\}^{2} \hfill \\ \quad \quad \frac{{\left( {\rho^{2} } \right)^{k + 1 - n} (1 - \rho^{2} )^{n} }}{{(1 + m - m\rho^{2} )^{2k - n + 3} }}\beta^{{\left( {k - n + 1} \right)}} e^{{\frac{ - \beta (1 + m)}{{\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )}}}} d\beta \hfill \\ \end{aligned}$$
(59)
$$\begin{aligned} P_{e} = \frac{N}{2}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{n!}} } } \frac{{((k + 1)!)^{2} }}{(k - n + 1)!}\left( {\rho^{2} } \right)^{k + 1 - n} (1 - \rho^{2} )^{n} \hfill \\ \quad \quad \int\limits_{0}^{\infty } {\frac{{Q\left( {b\sqrt \beta } \right)\beta^{k - n + 1} }}{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)^{k - n + 2} (k - n + 1)!}}\frac{1}{{(1 + m - m\rho^{2} )^{2k - n + 3} }}e^{{\frac{ - \beta (1 + m)}{{\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )}}}} d\beta } \hfill \\ \end{aligned}$$
(60)
Rewriting (60) similar to (22)
$$\begin{aligned} P_{e} = \frac{N}{2}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{{n!(1 + m)^{k - n + 2} }}} } } \frac{{((k + 1)!)^{2} }}{(k - n + 1)!}\frac{{\left( {\rho^{2} } \right)^{k + 1 - n} (1 - \rho^{2} )^{n} }}{{(1 + m - m\rho^{2} )^{k + 1} }} \hfill \\ \quad \quad \int\limits_{0}^{\infty } {\frac{{Q\left( {b\sqrt \beta } \right)\beta^{k - n + 1} }}{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)^{k - n + 2} (k - n + 1)!}}\frac{{(1 + m)^{k - n + 2} }}{{(1 + m - m\rho^{2} )^{k - n + 2} }}e^{{\frac{ - (k - n + 2)\beta (1 + m)}{{(k - n + 2)\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )}}}} d\beta } \hfill \\ \end{aligned}$$
(61)
Comparing (61) with (23) gives the average SER expression of M-ary PSK modulation for antenna selection [i.e. (12)] scheme 3.
Appendix 3
To verify the obtained closed form average SER expressions of M-ary PSK modulation scheme, substitute \(b^{2} = 2\) and \(M = 2\) in (4), (8), (12). This leads to average SER expressions of BPSK modulation scheme. The obtained results matches with the results in [40].
3.1 Scheme 1
On substituting \(b^{2} = 2\) and \(M = 2\) in (34), (36), (38) and (40), we can obtain the following reduced expressions.
$$P_{1}^{\prime } = N\left( {N - 1} \right)\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}\left[ {\frac{1}{2}\left\{ {1 - \sqrt {\frac{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}{{1 + \left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}} } \right\}} \right]$$
(62)
$$P_{2}^{\prime } = N\left( {N - 1} \right)\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g(g + 2)}\left\{ {1 - \sqrt {\frac{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)\left( {g + 2 - g\rho^{2} } \right)}}{{(g + 2) + \left( {\frac{{E_{s} }}{{N_{0} }}} \right)(g + 2 - g\rho^{2} )}}} } \right\}$$
(63)
$$P_{3}^{\prime } = \frac{N(N - 1)}{2}\left( {1 - \rho^{2} } \right)\left\{ {\frac{1}{2}\left[ {1 - \sqrt {\frac{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}{{1 + \left( {\frac{{E_{s} }}{{N_{0} }}} \right)}}} } \right]} \right\}$$
(64)
$$P_{4}^{\prime } = \frac{N(N - 1)}{2}\rho^{2} \left[ {\frac{1}{2} - \frac{1}{2}\sqrt {\frac{{E_{s} /N_{0} }}{{1 + \left( {E_{s} /N_{0} } \right)}}} \left( {1 + \frac{2}{{4\left( {1 + \left( {E_{s} /N_{0} } \right)} \right)}}} \right)} \right]$$
(65)
Substituting (62), (63), (64) and (65) in (28) gives
$$P_{e} = N\left( {N - 1} \right)\left[ {\sum\limits_{g = 1}^{N - 2} {\left( \begin{aligned} N - 2 \hfill \\ g \hfill \\ \end{aligned} \right)} \;\frac{{\left( { - 1} \right)^{g} }}{g}\left\{ {\frac{{1 - \sigma_{0} }}{2} - \frac{{\left( {1 - \sigma_{g} } \right)}}{g + 2}} \right\} + \frac{{1 - \sigma_{0} }}{4} - \frac{{\rho^{2} \sigma_{0}^{3} }}{{8\frac{{E_{s} }}{{N_{0} }}}}} \right]$$
(66)
where
$$\sigma_{g} = \sqrt {\frac{{\left( {\frac{{E_{s} }}{{N_{0} }}} \right)\left( {g + 2 - g\rho^{2} } \right)}}{{(g + 2) + \left( {\frac{{E_{s} }}{{N_{0} }}} \right)(g + 2 - g\rho^{2} )}}} ,\quad g = 0,1 \ldots$$
(67)
3.2 Scheme 2
On substituting \(b^{2} = 2\) and \(M = 2\) in (51), (52), (55) and (56), we can obtain the following reduced expressions.
$$P_{3}^{\prime } = N_{1} N_{2} \sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{{\left( {n - m} \right)}}} } \left\{ {\frac{{1 - \sigma_{m} }}{1 + m}} \right\}$$
(68)
where
$$\sigma_{m}^{2} = \frac{{\left( {\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )} \right)}}{{1 + m + \left( {\frac{{E_{s} }}{{N_{0} }}(1 + m - m\rho^{2} )} \right)}};\quad m = 0,1,2, \ldots .$$
(69)
$$P_{4}^{\prime } = \frac{{N_{1} N_{2} }}{2}\sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ n \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)( - 1)^{m + n} }}{{\left( {n - m} \right)}}\left\{ {\frac{{1 - \sigma_{n} }}{1 + n}} \right\}} }$$
(70)
where
$$\sigma_{n}^{2} = \frac{{\left( {\frac{{E_{s} }}{{N_{0} }}(1 + n - n\rho^{2} )} \right)}}{{1 + n + \left( {\frac{{E_{s} }}{{N_{0} }}(1 + n - n\rho^{2} )} \right)}};\quad n = 0,1,2, \ldots$$
(71)
$$P_{1}^{\prime } = \frac{{N_{1} N_{2} }}{2}\sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )}}\left( {1 - \rho^{2} } \right)\left\{ {\frac{{1 - \sigma_{r} }}{1 + r}} \right\}}$$
(72)
$$P_{2}^{\prime } = \frac{{N_{1} N_{2} }}{2}\sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{aligned} N_{1} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)\left( \begin{aligned} N_{2} - 1 \hfill \\ r \hfill \\ \end{aligned} \right)}}{{(1 + r - r\rho^{2} )(1 + r)^{2} }}\rho^{2} \left[ {1 - \sigma_{r} \left( {1 + \frac{{1 - \sigma_{r}^{2} }}{2}} \right)} \right]}$$
(73)
Substituting (68), (70), (72) and (73) in (43) gives
$$\begin{aligned} P_{e} &= \frac{{N_{1} N_{2} }}{2}\sum\limits_{r = 0}^{R - 1} {\frac{{\left( \begin{array}{c} N_{1} - 1 \hfill \\ r \hfill \\ \end{array} \right)\left( \begin{array}{c} N_{2} - 1 \hfill \\ r \hfill \\ \end{array} \right)}}{1 + r}\left\{ {\frac{{1 - \sigma_{r} }}{1 + r} - \frac{{\sigma_{r} \rho^{2} }}{{2\left( {1 + r - r\rho^{2} } \right)\left( {1 + r + \left( {E_{s} /N_{0} \left( {1 + r - r\rho^{2} } \right)} \right)} \right)}}} \right\}} \hfill \\ &\quad + \frac{{N_{1} N_{2} }}{2}\sum\limits_{n = 0}^{{N_{1} - 1}} {\sum\limits_{\begin{subarray}{l} m = 0 \\ m \ne n \end{subarray} }^{{N_{2} - 1}} {\frac{{\left( \begin{array}{c} N_{1} - 1 \hfill \\ n \hfill \\ \end{array} \right)\left( \begin{array}{c} N_{2} - 1 \hfill \\ m \hfill \\ \end{array} \right)( - 1)^{m + n} }}{{\left( {n - m} \right)}}\left\{ {\frac{{1 - \sigma_{m} }}{1 + m} - \frac{{1 - \sigma_{n} }}{1 + n}} \right\}} } \hfill \\ \end{aligned}$$
(74)
where
$$\sigma_{r}^{2} = \frac{{\left( {\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )} \right)}}{{1 + r + \left( {\frac{{E_{s} }}{{N_{0} }}(1 + r - r\rho^{2} )} \right)}};r = 0,1,2, \ldots$$
(75)
3.3 Scheme 3
On substituting \(b^{2} = 2\) and \(M = 2\) in (12), the average SER expression reduces to
$$\begin{aligned} P_{e} = \frac{N}{2}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{{n!(1 + m)^{k - n + 2} }}} } } \frac{{((k + 1)!)^{2} }}{(k - n + 1)!}\frac{{\left( {\rho^{2} } \right)^{k + 1 - n} (1 - \rho^{2} )^{n} }}{{(1 + m - m\rho^{2} )^{k + 1} }} \hfill \\ \quad \quad \left[ {\frac{1}{2} - \frac{1}{\pi }\sigma_{m} \left\{ {\left( {\frac{\pi }{2} + \tan^{ - 1} (0)} \right)\sum\limits_{t = 0}^{k - n + 1} {\left( \begin{aligned} 2t \hfill \\ \,t \hfill \\ \end{aligned} \right)\left( {\frac{{1 - \sigma_{m}^{2} }}{4}} \right)^{t} } } \right\}} \right] \hfill \\ \end{aligned}$$
(76)
$$\begin{aligned} P_{e} = \frac{N}{4}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{{n!(1 + m)^{k - n + 2} }}} } } \frac{{((k + 1)!)^{2} }}{(k - n + 1)!}\frac{{\left( {\rho^{2} } \right)^{k - n + 1} (1 - \rho^{2} )^{n} }}{{(1 + m - m\rho^{2} )^{k + 1} }} \hfill \\ \quad \quad \left[ {1 - \sigma_{m} \left\{ {1 + \sum\limits_{t = 1}^{k - n + 1} {\frac{{\left( {2t} \right)!}}{t!t!}} \;\sigma_{m}^{2t} \left( {\frac{{\sigma_{m}^{ - 2} - 1}}{4}} \right)^{t} } \right\}} \right] \hfill \\ \end{aligned}$$
(77)
$$\begin{aligned} P_{e} = \frac{N}{4}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{{n!(1 + m)^{k - n + 2} }}} } } \frac{{((k + 1)!)^{2} }}{(k - n + 1)!}\frac{{\left( {\rho^{2} } \right)^{k - n + 1} (1 - \rho^{2} )^{n} }}{{(1 + m - m\rho^{2} )^{k + 1} }} \hfill \\ \quad \quad \left[ {1 - \sigma_{m} - \sum\limits_{t = 1}^{k - n + 1} {\frac{{\left( {2t} \right)!}}{{t!2^{t} }}} \frac{{\sigma_{m}^{2t + 1} }}{t!}\left( {\frac{{\sigma_{m}^{ - 2} - 1}}{2}} \right)^{t} } \right] \hfill \\ \end{aligned}$$
(78)
$$\begin{aligned} P_{e} = \frac{N}{4}\sum\limits_{m = 0}^{{\frac{N}{2} - 1}} {\sum\limits_{k = 0}^{m} {\sum\limits_{n = 0}^{k + 1} {\frac{{\left( \begin{aligned} \frac{N}{2} - 1 \hfill \\ m \hfill \\ \end{aligned} \right)\left( \begin{aligned} m \hfill \\ k \hfill \\ \end{aligned} \right)\left( { - 1} \right)^{m} }}{{n!(1 + m)^{k - n + 2} }}} } } \frac{{((k + 1)!)^{2} }}{(k - n + 1)!}\frac{{\left( {\rho^{2} } \right)^{k - n + 1} (1 - \rho^{2} )^{n} }}{{(1 + m - m\rho^{2} )^{k + 1} }} \hfill \\ \quad \quad \left[ {1 - \sigma_{m} - \sum\limits_{t = 1}^{k - n + 1} {\left\{ {1.3..\left( {2t - 1} \right)} \right\}\left\{ {\frac{1 + m}{{2\left( {1 + m - m\rho^{2} } \right)\frac{{E_{s} }}{{N_{0} }}}}} \right\}^{t} } \frac{{\sigma_{m}^{2t + 1} }}{t!}} \right] \hfill \\ \end{aligned}$$
(79)