1 Introduction

Increasing channel potential remains the most important quest in modern communication systems [1, 2]. An important strategy to achieve this lies in rigorously estimating the Channel State Information (CSI). This process, known as channel estimation, holds the key to unlocking optimal performance by providing accurate information about the dynamic characteristics of the communication channel [3,4,5]. The communication systems can effectively adapt to diverse the challenging environments, optimize resource allocation and provide strong connectivity [6]. Channel estimation plays a crucial role in various fields within communication systems. Its applications cover important functions such as precoding, signal detection, resource optimization, indoor, and outdoor positioning, and strengthening physical layer security [7]. These diverse applications underscore the importance of precise channel estimation in ensuring efficient and reliable communications in modern systems [8]. The channel estimation process can be carried out using two basic methods: blind estimation and pilot-based methods. Blind prediction applies a statistical approach to estimate channel characteristics. The pilot-based method uses a known pilot signal in transmission to reveal the characteristics of the channel. Both methods are known to be application-based [9,10,11]. The statistical approach in time division duplex (TDD) systems avoids the need for a pilot-based channel estimation that is more complex to implement but leads to poor estimation accuracy [11,12,13]. Pilot channel estimation is less accurate than statistical channel estimation but has the advantage of being included in the transmission so that the channel can be characterized based on the received signal [14]. The evaluation of the pilot signal normally relies on LS or MMSE methods [15,16,17,18]. Both methods use basic interpolation techniques for processing. The MMSE algorithm provides improved prediction accuracy compared to LS. The improved accuracy is due to full channel statistical data and noise variance consideration but normally requires prior statistical information during the evaluation process [19,20,21].

The large number of subcarriers in the 5G network requires an accurate representation of the channel during the transmission [22,23,24]. To reveal the fading channel characteristics, it is of great importance to estimate many parameter values ​​accurately. The parameter estimation involves highly sensitive algorithms one of which is known as compressed sensing to estimate the channel coefficients. The success of the algorithm contributes to building the recovery matrix efficiently. Some of the other optimization algorithms use assumptions and model approximation processes that lead to limitations in handling complex type scenarios [25].

The latest research to improve the accuracy of the channel coefficients uses a deep learning process that requires a large number of data and a long-duration training process. The process has a real-time data update facility that makes it possible to extract accurate channel coefficients in dynamic wireless environments. The success of the process is due to the accuracy of feature extraction and nonlinear mapping process. The important point in such an application is to balance the computational complexity during the training and testing operating under the online conditions [4, 26].

Research on fading channels remains a hot topic in minimizing the intercarrier interference (ICI) effect. It is due to this reason that a newly proposed model has been introduced to represent the channel characteristic for the fading channel for 5G and future communication systems using various levels of QAM schemes. The model does not use any assumptions and also there is no need for channel knowledge before the application [26, 27].

1.1 Motivation

The motivation arises from the need to accurately determine the channel coefficients for dynamic scenarios where the channel variations occur instantaneously. The accurate representation facilitates improved performance across diverse modulation schemes and the channel coefficients at the transmitter and provides a higher signal-to-noise ratio (SNR) at the receiver. The proposed mathematical model includes an exponential term with a second-order polynomial to capture linear and nonlinear variations in the channel. The reason for selecting an exponential function is its ability to more effectively and accurately represent dynamic processes characterized by rapid changes. The fading severity is defined using a gamma function with scaling factor k as in Nakagami-m where m controls the fading depts. The variations are parameterized to enable dynamic representation of the channel characteristics. The proposed method successfully achieves a level of representation that classical approaches cannot provide, thereby introducing flexibility and adaptability that traditional methods are unable to offer.

The paper is organized as follows. The related work is detailed in Sect. 2. The OFDM channel models are explained in Sect. 3. The multipath channel is introduced in Sect. 4. The channel estimation process for the proposed model has been described in Sect. 5. Section 6 gives the experimental setup and measurements. The results are discussed in Sect. 7. The work is concluded in Sect. 8.

2 Related Works

The channel estimation using pilot-based is sensitive to the channel variations [28]. The other channel estimation method uses the statistical approach as in [29]. The extension research work in [30] introduces a blind source estimation process to overcome some problems related to multiple-input-multiple-output (MIMO) systems. The blind source estimation has a drawback of difficulty to apply. The difficulty problem was resolved in [31] with a reduction of spectral efficiency.

The channel parameter extraction uses different types of algorithms for the evaluation. The common algorithms are known as MMSE and Least Square (LS) which adds extra computational complexity during the execution. The authors in [32] proposed a new algorithm to overcome such computational complexity.

The latest research leads in the application of the deep learning algorithm which can be used in real-time applications and reduces the computational complexity [33]. The main aim of the algorithm is to enable tracking the Intercarrier Interference (ICI) in a fading communication channel. The author in [34] proposed additional work to minimize the additional pilot density while maintaining the ICI tracking in the process.

The authors in [35, 36] introduced an alternative approach to enhance the performance using temporary changes in time-varying channels. The papers in [37, 38] introduce a time-varying window that leads to research introducing an adaptive window for precision [39].

Network and system management for 5G and beyond has increasingly addressed challenges such as dynamic resource allocation, ultra-reliable low latency communication (URLLC), and efficient system optimization. The paper [40] introduced Deep Reinforcement Learning (DRL) classified as an advanced technique method to optimize the management of resources and ensure stable communication and balance energy efficiency with URLLC constraints in swarm robotic systems. The critical tasks in the paper optimize the necessary network parameters such as channel estimation using classical models (e.g., Rayleigh and Rician channel). Performance metrics such as BER or MSE are rarely addressed for system management needs for scalability, adaptability, and real-time decision-making operations. Our paper proposed a statistical model that contributes to the field by integrating robust channel estimation into a framework designed for advanced requirements. The proposed model demonstrates its potential to enhance system management efficiency and reliability.

The latest research focuses on important aspects such as optimizing communication performance for scalability and security by introducing intelligent frameworks. The paper [41] stated the complexity of Physical Cell Identity (PCI) and introduced a hybrid approach to overcome PCI collisions and confusion. The overall system management was improved by novel algorithms such as Symmetrical Comparison (SC) and Symmetrical Triangular Cycling (STC), emphasising the importance of proactive planning and optimization to increase system management. The paper [42] proposed intelligent software-defined networking (SDN) for dynamic resource allocation, improves security, and provides real-time decision-making. Our proposed model contributes by addressing the critical challenge of channel estimation in 5G and beyond networks. Our proposed model focuses on minimizing MSE and BER while optimizing system performance. The proposed model is comparable with the management model introduced as PCI, which centers on resource allocation and collision prevention. The proposed model in the research article directly tackles communication link reliability using advanced statistical estimation techniques. The robustness efficiency of communication channels under varying modulation schemes was enhanced using LS and MSE optimization algorithms for security and real-time adaptability. The contribution of the proposed model, together with the stated improvements, is considered to advance both the technical foundation of network management and the operational efficiency of future communication systems.

Researchers in [43] use complex frameworks and Network methods to improve communication performance in different scenarios. Channel estimations are of great importance in these applications. A masked autoencoder (MAE) based channel estimator and a hierarchical reinforcement learning (HRL) based pilot are used, specifically designed for predominantly AI-focused applications. This method reduces the communication load and exhibits high performance in image reconstruction processing. In the presented network, the Framework’s orthogonal frequency division multiplexing (OFDM) layers and artificial intelligence integration optimize resource allocation for highly dynamic network environments.

Our proposed work presents a fundamentally different approach by deriving a new probability distribution function (PDF) to model randomly varying noisy data for channel coefficient extraction. Unlike traditional Rayleigh and Rician distributions, the proposed PDF function offers significantly higher accuracy in representing channel features, directly improving the channel estimation reliability. The proposed PDF is parameterized to apply to any randomly varying data dynamically. Instead of applying task-specific optimization methods as is the case in AI applications, our work succeeds in minimizing the bit error rate (BER) and mean square error (MSE) across various modulation schemes, including 64, 256, and 1024-QAM, thus providing a broader network performance improvement. The generalized approach provided by the parameterized PDF function provides robustness and adaptability in various communication scenarios, making a valuable contribution to the evolution of network management in 5G and beyond. Comparing the two studies, both highlight complementary advances in channel prediction; one focuses on the direction of AI-enabled semantic communication, while the other addresses fundamental improvements in modelling and prediction accuracy.

3 OFDM Channel Models

The OFDM system facilitates to combination of multiple input bits to generate a symbol. The generated symbols denoted as X(0) to X(K-1) are modulated using one of the digital modulation techniques. The data symbols are converted back to parallel representation using Inverse Fourier Transformation methods (IFFT). The advantage of the IFFT process is to ensure the orthogonality of each subcarrier so that K data symbols are mapped onto K subcarriers. The Process of IFFT can be expressed mathematically as

$$\:x\left(k\right)=\frac{1}{\sqrt{K}}\sum_{n=0}^{K-1}\:X\left(n\right){e}^{j2\pi\:nk/K}$$
(1)

The selection of K-point IFFT samples provides OFDM symbol ‘\(\:k\)’ changing from zero (0) to \(\:\left(K-1\right)\) where \(\:x\left(k\right)\) represents the \(\:k\)th sample. The fading channel affects the generated symbols such that they become one of the symbols generated from OFDM samples. This action causes inter-block interference (IBI). The effect of the IBI can be minimised by introducing a cyclic prefix the longer than the number of channel tabs (L). The samples at the receiver can be stated as

$$\:y\left(k\right)=x\left(k\right)\circledast h\left(l\right)+v\left(k\right)$$
(2)

The variable \(\:y\left(k\right)\) represents the \(\:k\)th received sample, \(\:h\left(l\right)\) represents the \(\:l\)th channel tab, and \(\:v\left(k\right)\) represents the noise samples.

Eq. (2) gives rise to expression represented in Fast Fourier transformation (FFT) of the received signal as.

$$\:Y\left(n\right)=X\left(n\right)H\left(n\right)+V\left(n\right)$$
(3)

The \(\:X\left(n\right)\), \(\:H\left(n\right)\), \(\:Y\left(n\right)\) and \(\:V\left(n\right)\) represents K-point FFT samples equivalent to variables in Eq. (2). The mathematical representation can be stated as,

$$\:\begin{aligned}Y\left(n\right)&=\frac{1}{\sqrt{K}}\sum_{k=0}^{K-1}\:y\left(k\right){e}^{-\frac{j2\pi\:nk}{K}},\\H\left(n\right)&=\frac{1}{\sqrt{K}}\sum_{l=0}^{K-1}\:h\left(l\right){e}^{-\frac{j2\pi\:nk}{K}}.\\ X\left(n\right)&=\frac{1}{\sqrt{N}}\sum_{n=0}^{N-1}\:x\left(k\right){e}^{-\frac{j2\pi\:nk}{K}},\\V\left(n\right)&=\frac{1}{\sqrt{N}}\sum_{n=0}^{N-1}\:v\left(k\right){e}^{-\frac{j2\pi\:nk}{K}}.\end{aligned}$$
(4)

4 Channel with Multipath Fading

The wireless channel can be analyzed using a general representation of small and large fading models. The faded signal heavily relies on the existence of the line-of-sight (LOS). The signal non-line-of-sight (N-LOS) was analyzed using several fading signals. The faded signals are processed using the Rayleigh distribution or Rician distribution.

4.1 Rayleigh Fading Channel

The Rayleigh distribution is mostly used for applications where LOS paths do not exist between the transmitter and the receiver. The Rayleigh fading model applies to the fast-fading channel model. This model is most suitable for channels that have many numbers of uncorrelated multipath reflections. The transmitted signal normally undergoes some distortion. The cause of the distortion using the Rayleigh fading channel can be best represented by the Rayleigh distribution. The reason for multipath reception is due to Rayleigh fading phenomena. The antenna at the receiver normally receives several reflected and scattered waves that lead to interference in such a process. The probability distribution function (PDF) of the Rayleigh random variable can be stated as

$$\:p\left(x\right)=\frac{x}{{\sigma}^{2}}{e}^{\left(-{x}^{2}/2{\sigma}^{2}\right)},\text{\:for\:x}>0$$
(5)

The variable \(\:{\sigma}^{2}\) denotes the time average power of the received signal.

4.2 Rician Fading Channel

The Rician distribution is used where LOS and N-LOS exist between the transmitter and receiver. The Rician fading is more suitable to apply when a strong direct signal component exists between the transmitter and receiver. The amplitude gain of a signal normally follows Rician distribution if the channel is described as a Rician fading channel. The Rician fading channel exhibits a LOS in transmission. This fading model emerges when one of the received signals, usually the LOS component, exhibits a notably higher strength compared to the others. The Rician distribution is expressed as.

$$\:p\left(x\right)=\frac{x}{{\sigma}^{2}}{e}^{\left(-\left({x}^{2}-{A}^{2}\right)/\mid\:2{\sigma}^{2}\right)}{I}_{0}\left(\frac{Ar}{{\sigma}^{2}}\right)\text{\:for}(A\ge\:0,x\ge\:0)\text{,}$$
(6)

The variable A denotes the amplitude of a signal. \(\:{I}_{0}\) is the result of the Bessel function.

The channel selection is important when designing a network system. Therefore, a special consideration must be taken while analyzing a multipath fading channel.

The general channel representation of the multipath fading signal follows as in Eqs. 7–9.

$$\:\text{h}=\sum_{i=0}^{L-1}\:{\text{a}}_{i}{e}^{-j2\pi\:{F}_{c}{\tau}_{i}}$$
(7)
$$\begin{array}{c}{h=\stackrel{{h}_{r}}{\overbrace{\left.\sum_{i=0}^{L-1}\:{a}_{i}\text{c}\text{o}\text{s}\left(2\pi\:{F}_{c}{\tau}_{i}\right)\right)}}-j\stackrel{{h}_{i}}{\overbrace{\left(\sum_{i=0}^{L-1}\:{a}_{i}\text{s}\text{i}\text{n}\left(2\pi\:{F}_{c}{\tau}_{i}\right)\right)}}}\\{h={h}_{r}-j{h}_{i}}\end{array}$$
(8)

where x represents the real part and y represents the imaginary part of the fading signal.

$$\:{h}_{r}=\sum_{i=0}^{L-1}\:{\text{a}}_{i}\text{c}\text{o}\text{s}\left(2\pi\:{F}_{c}{\tau}_{i}\right){h}_{i}=-j\sum_{i=0}^{L-1}\:{\text{a}}_{i}\text{s}\text{i}\text{n}\left(2\pi\:{F}_{c}{\tau}_{i}\right)$$
(9)

The variables \(\:{\text{a}}_{i}\) denotes the attenuations and \(\:{\tau}_{i}\) represents delay, \(\:L\) indicates the number of multipath fading signals in consideration.

The random variation of \(\:{h}_{r}\) and \(\:{h}_{i}\) require statistical analysis of the data that will be described in the next section.

5 Channel Estimation

5.1 Statistical Analysis

The random variations in the channel require further statistical analysis to determine the channel characteristics. The variations in multipath fading signal are the amplitude and phase (delay) components. The Root mean squared (RMS) delay spread can be stated as in Eq. 10.

$$\:{\tau}_{rms}=\sqrt{\overline{{\tau}^{2}}-{\bar{\tau}}^{2}}$$
(10)

where

$$\:\overline{\tau}=\frac{{\sum}_{l=1}^{L}{P}_{l}{t}_{l}}{{\sum}_{l=1}^{L}{P}_{l}}\:\text{a}\text{n}\text{d}\:\overline{{\tau}^{2}}=\frac{{\sum}_{l=1}^{L}{P}_{l}{t}_{l}^{2}}{{\sum}_{l=1}^{L}{P}_{l}}.$$

The variable \(\:L\) denotes the number of received rays. \(\:t\) is the ray arrival time and \(\:P\) denotes the ray strength.

5.2 Proposed Statistical Modelling

The random variation of x and y is normally assumed to follow the Gaussian distribution. This work proposes a most general PDF that includes all classical distributions stated in Eq. 11.

$$\:f\left(x;\:a,\:b,\:k\right)=\:\frac{1}{\varGamma\left(k+1\right)}(2ax+b)(a{x}^{2}+{bx)}^{k}.\:{e}^{-\left(a{x}^{2}+bx\right)}.$$
(11)

where x > 0, \(\:a\), \(\:b\) ≥ 0, Re(\(\:k\)) >-1

The proof of Eq. 11 follows:

$$\:f\left(x;a,\:b,\:k\right)=1-\frac{\varGamma\:\left(\text{k}+1,\text{a}{x}^{2}+\text{b}x\right)}{\varGamma\:\left(k+1\right)}\:\:\:\:\:\:\:x>0,\text{a},\text{b}\ge\:0,\:Re\left(k\right)>-1,$$
(12)

The function at the numerator \(\:\varGamma\:\left(\text{k}+1,\text{a}{x}^{2}+\text{b}x\right)\) can be denoted as,

$$\:\varGamma\:\left(\text{k}+1,\text{a}{x}^{2}+\text{b}x\right)={\int}_{a{x}^{2}+\text{b}x}^{{\infty}}{q}^{k}{e}^{-\text{q}}dq.$$
(13)

Eq 11 using boundaries as \(\:0<x<\infty\:\) yields:

$$\:\frac{1}{\varGamma\left(\text{k}+1\right)}{\int}_{0}^{{\infty}}(2\text{a}x+\text{b})(\text{a}{x}^{2}+{bx)}^{k}.\:{e}^{-(\text{a}{x}^{2}+\text{b}x)}dx=\frac{1}{\varGamma\left(\text{k}+1\right)}{\int}_{0}^{{\infty}}{u}^{k}{e}^{-u}du.$$
(14)

The variable u equals to \(\:(\text{a}{x}^{2}+\text{b}x)\).

The gamma function can be stated as

$$\:\varGamma\left(\text{k}+1\right)={\int}_{0}^{{\infty}}{u}^{k}{e}^{-u}du.$$
(15)

Combining Eqs. 14 and 15 gives a value of 1 which is sufficient proof to make Eq. 11 a valid PDF.

Eq. 11 is generalized and reduced to well-known classical distributions listed in Table 1.

Table 1 Special parameter values to represent classical distributions
Full size table

The parameters in Eq. 11 are extracted from the given random data. The extraction process follows the classical method known as maximum likelihood estimation (MLE).

The process uses the cumulative distribution function (CDF), log-likelihood optimization and partial derivative equations that produce three equations with three unknowns (a, b, k) to be solved.

CDF has already been defined in Eq. 12. The log-likelihood equation can be stated as,

$$\begin{aligned}\text{L}\left(a,\:b,\:k\right)&=-n\:ln\left(\varGamma\:\left(\text{k}+1\right)\right)+\sum_{i=1}^{n}ln\left(2\text{a}{x}_{i}+\text{b}\right)\\&+\sum_{i=1}^{n}{ln\left(a{x}_{i}^{2}b{x}_{i}\right)}^{q}-\sum_{i=1}^{n}ln\left(a{x}_{i}^{2}b{x}_{i}\right).\end{aligned}$$
(16)

The derivative of Eq. 16 concerning a, b, k produces

$$\:\frac{\partial\:L}{\partial\:a}=\sum_{i=1}^{n}\frac{2{x}_{i}}{2a{x}_{i}+b}+\sum_{i=1}^{n}\frac{k{x}_{i}}{a{x}_{i}+b}-\sum_{i=1}^{n}{x}_{i}^{2},$$
(17)
$$\:\frac{\partial\:L}{\partial\:b}=\sum_{i=1}^{n}\frac{1}{2a{x}_{i}+b}+\sum_{i=1}^{n}\frac{k}{a{x}_{i}+\text{b}}-\sum_{i=1}^{n}{x}_{i},$$
(18)
$$\:\frac{\partial\:L}{\partial\:k}=-n\frac{{\varGamma}^{{\prime}}\left(\text{k}+1\right)}{\varGamma}(\text{k}+1)+\sum_{i=1}^{n}ln\left(a{x}_{i}^{2}b{x}_{i}\right)=0.$$
(19)

The identified parameter values of a, b, and k are substituted in Eq. 11. The defined PDF is complete at this stage. The next step is to identify the channel coefficients of the fading channel. Normally, the fading channel model needs to be identified before determining the channel coefficients. This work introduces a general PDF to cover all fading channel models. Therefore, the coefficients will be extracted without naming the channel model.

The fading coefficients are complex and can be represented as in Eq. 8. This can be expressed as

$$\:f\left({h}_{r}\mid\:{x}_{r}\right)=\frac{f\left({x}_{r},\:{h}_{r}\right)}{f\left({x}_{r}\right)}$$
(20)

where,

  • \(\:f\left({x}_{r},\:{h}_{r}\right)\) represents the joint PDF of the real part.

  • \(\:f\left({x}_{r}\right)\) represent the marginal PDF of the real part.

Rearranging Eq. 20 as

$$\begin{array}{c}{f\left({x}_{r},\:{h}_{r}\right)=f\left({h}_{r}\mid\:{x}_{r}\right).}\\{f\left({x}_{r}\right)f\left({x}_{r},{h}_{r}\right)=\frac{1}{{\Gamma}(q+1)}\cdot\:\left(2\alpha\:{x}_{r}+\beta\:\right)\cdot\:{\left(\alpha\:{x}_{r}^{2}+\beta\:{x}_{r}\right)}^{q}\cdot\:{e}^{-\left(\alpha\:{x}_{r}^{2}+\beta\:{x}_{r}\right)}\cdot\:{e}^{-{h}_{r}^{2}}}\end{array}$$
(21)

Then, \(\:f\left({x}_{r}\right)\) can be represented as

$$\:\begin{array}{l}f\left({x}_{r}\right)={\int}_{-{\infty}}^{{\infty}}\:f\left({x}_{r},{h}_{r}\right)d{h}_{r}\\\:f\left({x}_{r}\right)=\frac{1}{{\Gamma}(q+1)}\cdot\:\left(2\alpha\:{x}_{r}+\beta\:\right)\cdot\:{\left(\alpha\:{x}_{r}^{2}+\beta\:{x}_{r}\right)}^{q}\cdot\:{e}^{-\left(\alpha\:{x}_{r}^{2}+\beta\:{x}_{r}\right)}\cdot\:{\int}_{-{\infty}}^{{\infty}}\:{e}^{-{h}_{r}^{2}}d{h}_{r}\\\:f\left({x}_{r}\right)=\frac{1}{{\Gamma}(q+1)}\cdot\:\left(2\alpha\:{x}_{r}+\beta\:\right)\cdot\:{\left(\alpha\:{x}_{r}^{2}+\beta\:{x}_{r}\right)}^{q}\cdot\:{e}^{-\left(\alpha\:{x}_{r}^{2}+\beta\:{x}_{r}\right)}\cdot\:\sqrt{\pi}\end{array}$$
(22)

Then, \(\:f\left({h}_{r}\mid\:{x}_{r}\right)\) can be computed as

$$\:\begin{array}{l}f\left({h}_{r}\mid\:{x}_{r}\right)=\frac{f\left({x}_{r},{h}_{r}\right)}{f\left({x}_{r}\right)}\\\:f\left({h}_{r}\mid\:{x}_{r}\right)=\frac{\left(2\alpha\:{x}_{r}+\beta\:\right)\cdot\:{\left(\alpha\:{x}_{r}^{2}+\beta\:{x}_{r}\right)}^{q}\cdot\:{e}^{-{h}_{r}^{2}}}{\sqrt{\pi}}\end{array}$$
(23)

Equation 23 represents the real part of the channel coefficients provided the real components of the received signal are provided.

The same process can be applied to the imaginary part.

The expression results as

$$\:\begin{array}{l}f\left({h}_{i}\mid\:{x}_{i}\right)=\frac{f\left({x}_{i},{h}_{i}\right)}{f\left({x}_{i}\right)}\\\:f\left({h}_{i}\mid\:{x}_{i}\right)=\frac{\left(2\alpha\:{x}_{i}+\beta\:\right)\cdot\:{\left(\alpha\:{x}_{i}^{2}+\beta\:{x}_{i}\right)}^{q}\cdot\:{e}^{-{h}_{i}^{2}}}{\sqrt{\pi}}\end{array}$$
(24)

The next step is to express the joint distribution of the real and imaginary parts of the channel coefficients. The process requires the utilization of the joint PDF of the real and imaginary components of incoming signal x.

The real and imaginary parts of x can be represented as \(\:f\left({x}_{r},{x}_{i}\right)\), and transformed to obtain the joint distribution of the real and imaginary parts of h.

The transformation from x to h in Eq. 8 yields the joint PDF \(\:f\left({h}_{r},{h}_{i}\right)\) expressed as,

$$\:f\left({h}_{r},{h}_{i}\right)=f\left({x}_{r},{x}_{i}\right)\left|\frac{\partial\:\left({x}_{r},{x}_{i}\right)}{\partial\:\left({h}_{r},{h}_{i}\right)}\right|$$
(25)

where \(\:\left|\frac{\partial\:\left({x}_{r},{x}_{i}\right)}{\partial\:\left({h}_{r},{h}_{i}\right)}\right|\) denotes the Jacobian determinant of the transformation.

The partial derivatives are in the form as

$$\:\frac{\partial\:\left({x}_{r},{x}_{i}\right)}{\partial\:\left({h}_{r},{h}_{i}\right)}=\left|\begin{array}{c}\frac{\partial\:{x}_{r}}{\partial\:{h}_{r}}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\frac{\partial\:{x}_{r}}{\partial\:{h}_{i}}\\\:\frac{\partial\:{x}_{i}}{\partial\:{h}_{r}}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\frac{\partial\:{x}_{i}}{\partial\:{h}_{i}}\end{array}\right|$$
(26)

The resulting value of Eq. 26 substitutes for Eq. 25 to provide a joint PDF of \(\:f\left({h}_{r},{h}_{i}\right)\).

Employing MLE on the derived joint PDF yields the coefficients characterizing the fading channel.

6 Experimental Setup and Measurements

The experiment in this research analyses the BER for different channel models (Rayleigh, Rician etc., fading channel) and compares it with the proposed model using different modulation schemes. Furthermore, the effect of the number of pilot signals in the application has been analyzed and the comparable results are recorded for further analysis.

The experimental location is indicated in Fig. 1.

Fig. 1
figure 1

Experimental field

Full size image

The experimental area is situated at the Near East University campus. The transmitter is located on top of the building shown as Tx. The receiver is purposely selected at a location where LOS path and multipath fading signals are available. The weather condition was sunny at a temperature of 25 Centigrade during the experiment.

The location of the receiver enables to get the LOS path as well as multipath faded signals. The purpose of such an arrangement is to facilitate data for Rayleigh and Rician fading channels. Accordingly, the relevant equipment was carefully selected and used for this application.

The equipment setup in the experiment is shown in Fig. 2.

Fig. 2
figure 2

Block representation of the setup equipment

Full size image

The Keysight E7515B as a base station was set as a transmitter and tuned to a centre frequency of 2.5 GHz with a total bandwidth of 100 MHz. The receiver and the transmitter were installed with an omnidirectional antenna having a gain of 60 dBi. The measurements were repeated at 20 different locations for a link measurement. Both the transmitter and the receiver were static during the measurements. The experiments were conducted at a location to capture LOS and NLOS signals.

There are four omnidirectional antennas at the receiving location connected to Keysight N5106A. One of the antennas is adjusted to receive LOS and the other three antennas are used to capture the faded signals.

The Keysight is connected to the signal simulator to measure and evaluate a wide range of the received signal frequency, amplitude, and phase related to the two faded signals for the transmitted channel under test. It also enables to evaluation of the effect of faded signals in the process.

The Rohde & Schwarz FSWP is sensitive equipment to produce the phase noise of the faded signals. The recorded data was used to analyze the analyze to identify the PDF of the random variation.

The Microsemi-Spectra Time SA.45 was connected to Rohde & Schwarz FSWP. The precision of timing in faded signals is very important so that stability can be maintained in the recorded data.

The setup parameters in the experiment are listed in Table 2.

Table 2 Setup parameters
Full size table

The experiment was conducted at six different locations with varying distances, LOS, NLOS, and fading type. The important parameters such as the SNR, RSS, phase noise, and timing precision were measured and listed in Table 3.

Table 3 Experimental SNR, RSS, phase noise, and timing data
Full size table

6.1 Practical Details

The practical details are described below,

  • Signal Measurement: The channel characteristics under fading conditions were measured using the Keysight equipment. The measured parameters are amplitude, frequency, and phase data.

  • Phase Noise: The signal degradation in a multipath environment is a critical task that was measured using Rohde & Schwarz FSWP which analyzes the impact of phase noise.

  • Timing Stability: The synchronization and accurate model fitting require accurate timing precision in recorded data that was measured using Microsemi-Spectra Time SA.45.

  • Signal-to-Noise Ratio (SNR): The SNR values at the receiver are recorded using Keysight N5106A for 64-QAM, 256-QAM, and 1024-QAM.

  • BER: The Keysight N5106A is used to evaluate BER for different modulation schemes. The process uses the received data stream to compare against the transmitted using the formulation:

$$\:BER=Number\:of\:Bit\:Errors/Total\:Number\:of\:Transmitted\:Bits$$

  • MSE: The MSE defines the errors between the transmitted signal and the estimated signal using channel coefficients. The MSE calculates the SNR levels using the estimated channel coefficients and the true coefficients as:

$$\:\text{M}\text{S}\text{E}=\frac{1}{N}\sum_{i=1}^{N}\:{\left|{h}_{\text{true},i}-{h}_{\text{estimated},i}\right|}^{2}$$

where htrue denotes the true channel coefficients and hestimated denotes the estimated coefficients.

  • Modulation Scheme: 64-QAM, 256-QAM, and 1024-QAM modulation levels were configured at the transmitter for the specific QAM levels and measured repeatedly.

6.2 Practical Details Using Pilot Signal

The known pilot signal is transmitted through the wireless channel to identify the channel’s behaviour. The purpose of the measurement is to determine the Bit Error Rate (BER) which is the ratio of incorrectly decoded pilot bits to the total number of transmitted pilot bits. The processing steps are listed below.

  • Transmit the Pilot Signal: Generate a known pilot sequence (e.g., QAM-64, QAM-256, QAM-1024 symbols). Transmit this pilot signal through the channel under test using the Keysight E7515B.

  • Introduce Noise: Simulate or measure noise added to the pilot signal. Vary the noise power to achieve different SNR levels.

  • Receive and Demodulate: At the Keysight N5106A, capture the received pilot signal. Demodulate the signal and recover the pilot sequence.

  • Compare with the Transmitted Signal: Compare the received pilot bits with the known transmitted pilot bits. Count the number of bit errors for each SNR level.

Performance analysis of the pilot signal characterization is detailed as follows,

  • BER Calculation:

$$\:BER=\frac{Number\:of\:Erroneous\:Bits}{Total\:Number\:of\:Transmitted\:Bits}$$

  • BER vs. SNR Plot: Use different SNR values to determine the BER in transmision.

  • Channel Estimation: Use the pilot signal to estimate the channel (e.g., channel impulse response). Compute the estimated channel coefficients, hest signal.

  • Measure Received Symbols: Record the received pilot symbols after applying the channel estimation.

  • Calculate Symbol Errors: For each received pilot symbol, compute the error and square the error as:

$$\begin{aligned}Error&={x}_{transmitted}-{x}_{received}\\Squared\:Error&={\mid\:{x}_{transmitted}-{x}_{received}\mid}^{2}\end{aligned}$$

  • Calculate MSE: Average the squared errors over all pilot symbols:

$$\:\text{M}\text{S}\text{E}=\frac{1}{N}\sum_{i=1}^{N}\:{\left|{x}_{i}-{\widehat{x}}_{i}\right|}^{2}$$

where \(\:{x}_{i}\) is the transmitted symbol, and \(\:{\widehat{x}}_{i}\) is the received symbol.

7 Results

The research aims to analyze the effect of the newly proposed model on classical models such as Rayleigh and Rician distribution channel characteristics.

The experiment was conducted to analyze the BER for different channel models in the MIMO-OFDM system. The channel models were examined using channel characteristic coefficients. The data-driven parameters are extracted and substituted in Eq. 11 which describes the probability distribution function of the faded channel. The PDF was processed as indicated by Gaussian distribution to yield the fading channel coefficients. The five-tab channel coefficients are optimized using the LS method.

The faded signals experience distortion, which leads to intersymbol interference (ISI), which increases the error level at the receiver. The effect of such phenomena was minimized using the equalization method.

The performance of the system was compared using various levels of Quantized Amplitude Modulation (QAM) such as 64, 256, and 1024.

The plots include the proposed, Rayleigh and Rician models for comparison. The faded channel results using 5 tap coefficients with 64-QAM plotted in Fig. 3.

Fig. 3
figure 3

Performance plot for 64-QAM using LS estimators

Full size image

The performance of the proposed model has minimum BER for all SNR values compared with the classical channel models under test.

The BER performance results illustrated in Fig. 3 highlight the significant advantages of the proposed model when applied to 64-QAM modulation using the LS estimation technique, across varying SNR values. The results can be categorized into three regions:

At an SNR of 15 dB, the proposed model achieves a BER of 0.05, while the Rayleigh and Rician models result in BERs of 0.12 and 0.14, respectively. This translates to a reduction of approximately 58.3% and 64.3% demonstrating the superior performance of the proposed approach.

This improvement can be attributed to the model’s dynamic parameterization, which adjusts to instantaneous variations in channel conditions. In particular, the use of the gamma function and the scaling factor k enhances its adaptability, making it well-suited for dynamic environments.

Figure 4 shows the performance of the different channel models using the MSE method with 64-QAM.

Fig. 4
figure 4

Performance plot for 64-QAM using MSE estimators

Full size image

The MSE performance analysis for 64-QAM modulation illustrated in Fig. 4 using the MSE estimation method underscores the efficacy of the proposed model in accurately estimating channel coefficients across various SNR levels. The results are summarized as follows:

At an SNR of 7 dB, the proposed model achieves an MSE value close to 0.08, while the Rayleigh and Rician models exhibit MSEs of approximately 0.12 and 0.16, respectively. This corresponds to reductions of 34% and 43% compared to the Rayleigh and Rician models.

The notable improvement in the low SNR range highlights the proposed model’s resilience under challenging noise and fading conditions. This enhancement arises from the model’s adaptive design, which incorporates a second-order polynomial and gamma-based parameterization to effectively characterize variations in the channel.

This enhanced accuracy in moderate SNR conditions can be attributed to the model’s dynamic adaptability to real-time channel fluctuations. Traditional Rayleigh and Rician models lack this level of flexibility, which limits their ability to estimate coefficients with equivalent precision. Although the performance gap between the models narrows at higher SNR levels, the proposed model retains its advantage due to its precise mathematical representation of fading effects. This results in consistently lower estimation errors, even under near-ideal conditions.

The results of the experiment with 256-QAM are plotted in Fig. 5.

Fig. 5
figure 5

Performance plot for 256-QAM using LS estimators

Full size image

The BER performance of the proposed model for 256-QAM modulation using LS estimators demonstrates significant improvement compared to the Rayleigh and Rician channel models:

The proposed model outperforms both the Rayleigh and Rician channel models, demonstrating a significant reduction in BER values, particularly at a low SNR of 15 dB. At this point, the BER decreases notably from an initial value of 0.37 to nearly a value of 0.08, indicating the model’s enhanced precision in channel coefficient estimation. The proposed model reaches to 0 BER at an SNR of 27 dB and the other two models reduce to 0 at an SNR of 30 and 35 dB. These findings consistently demonstrate the proposed model’s robustness and effectiveness in minimizing bit error rates, particularly for 256-QAM signals across various SNR levels.

Figure 6 shows the performance of the different channel models using the MSE method with 256-QAM.

Fig. 6
figure 6

Performance plot for 256-QAM using MSE estimators

Full size image

The performance of the proposed model is better than Rayleigh and Rician channel models and the MSE values reduce to a minimum level at low SNR value (15dB). The MSE value reduces from 0.7 to 0.

The MSE performance of the proposed model with the MSE estimator for 256-QAM reveals notable advancements over the Rayleigh and Rician models, especially at lower SNR levels. For instance, at 5 dB SNR, the proposed model achieves an MSE of 0.21, which is significantly lower than the 0.31 and 0.42 observed for the Rayleigh and Rician models, respectively. This indicates the model’s enhanced capability to accurately estimate channel coefficients under noisy and highly faded conditions.

As the SNR increases into the mid-range (10–20 dB), the MSE for the proposed model decreases sharply, reaching 0.02 at 15 dB, while the Rayleigh and Rician models plateau at approximately 0.05 and 0.06. This improvement, estimated to be around 60–66%, underscores the accuracy and reliability of the proposed method in minimizing estimation errors.

At higher SNR values (20–35 dB), the MSE values for all three models trend toward zero. However, the proposed model retains a noticeable edge, achieving an MSE value as low as 0.001 at 21 dB. These findings collectively affirm the robustness and efficiency of the proposed model in reducing mean square errors across diverse SNR conditions for 256-QAM signals.

The results of the experiment with 1024-QAM are plotted in Fig. 7.

Fig. 7
figure 7

Performance plot for 1024-QAM using LS estimators

Full size image

The BER performance of the proposed model employing the LS estimator for 1024-QAM demonstrates clear improvements compared to the Rayleigh and Rician channel models over the entire SNR range. At 0 dB SNR, the proposed model achieves a BER of 0.42, while both the Rayleigh and Rician models yield higher error rates, indicating their reduced ability to counteract noise and channel fading in low-SNR scenarios.

With increasing SNR values, the proposed model consistently delivers lower BER values than its counterparts. For instance, at 15 dB SNR, the BER decreases to approximately 0.17 for the proposed approach, whereas the Rayleigh and Rician models remain elevated at around 0.21. This translates to highlighting the superior channel estimation accuracy of the proposed model.

In the high-SNR region (above 25 dB), the BER values for all models begin to converge as noise becomes negligible. Nevertheless, the proposed model continues to show the best performance, achieving a near-zero BER at 35 dB SNR, outperforming the Rayleigh and Rician approaches. These findings underscore the effectiveness and robustness of the proposed LS estimator in enhancing performance and reducing bit errors for 1024-QAM signals.

Figure 8 shows the performance of the different channel models using the MSE method with 1024-QAM.

Fig. 8
figure 8

Performance plot for 1024-QAM using MSE estimators

Full size image

The MSE performance of the proposed model utilizing the MSE estimator for 1024-QAM demonstrates its clear advantage over both Rayleigh and Rician models. At the initial low SNR values, the proposed method starts with an MSE of 0.67, which significantly decreases as the SNR rises. Meanwhile, the Rayleigh and Rician models exhibit higher MSE values, reflecting the proposed model’s enhanced capacity to accurately estimate channel coefficients under challenging conditions.

As SNR increases, the MSE for the proposed model steadily drops to nearly zero at 35 dB, indicating superior performance. In comparison, the Rayleigh and Rician models show more gradual reductions in MSE, remaining higher even at 35 dB, further demonstrating the efficiency of the proposed model in providing precise channel estimation.

The proposed model’s consistent reduction in MSE values, particularly at low SNR levels, showcases its effectiveness in improving channel estimation and enhancing the quality of communication signals, making it a reliable approach for 1024-QAM in diverse environments.

The proposed statistical model compared with the classical pilot signal characterization method and plotted in Fig. 9.

Fig. 9
figure 9

Performance comparison between the proposed statistical model and the pilot-based approach for 64-QAM using LS estimators

Full size image

The statistical model developed in this study was evaluated under a low-order modulation scheme, specifically 64-QAM. Figure 9 presents a performance comparison between the proposed model and the pilot-based signal for 64-QAM using LS estimators. The results indicate that the proposed model provides a more accurate representation of the channel dynamics, especially in high-SNR conditions. At an SNR of 20 dB, the pilot-based method exhibits a notable limitation, yielding a BER of 0.1, suggesting suboptimal channel characterization compared to the proposed approach.

This disparity highlights a potential drawback of pilot-based methods. Their limited ability to adapt to rapidly changing conditions, such as those caused by dynamic environments or varying weather conditions, reduces their effectiveness. Statistical models, on the other hand, are inherently more robust, as they excel in capturing channel variations and updating parameters in real-time. These observations underscore the advantages of the proposed approach, particularly in scenarios where accurate and responsive channel estimation is crucial.

The performance of the pilot-based scheme was further evaluated and plotted in Fig. 10. under a high-order modulation scenario (1024-QAM) utilizing an MSE estimator to analyze its effectiveness in capturing channel characteristics.

Fig. 10
figure 10

Performance comparison between the proposed statistical model and the pilot-based approach for 1024-QAM using MSE estimators

Full size image

The plot highlights the superior MSE performance of the proposed statistical model compared to the pilot-based approach, as shown in the figure. The enhanced performance can be attributed to the proposed model’s ability to adaptively capture channel variations with higher precision, particularly in high-order modulation schemes like 1024-QAM. In contrast, the pilot-based approach suffers from limitations in accurately estimating channel parameters under complex signal configurations, resulting in higher estimation errors.

The proposed statistical model was further extended to compare its performance with additional statistical models, including Nakagami-m, Weibull, and Suzuki, as illustrated in Fig. 11.

Fig. 11
figure 11

Performance comparison of the proposed model with Nakagami-m, Weibull, Suzuki, Rayleigh, and Rician models under 256-QAM using LS estimators

Full size image

To maintain consistency with prior analyses, the evaluation utilized a 256-QAM modulation scheme in combination with LS estimators. Figure 11 presents the comparative performance of the proposed model alongside Nakagami-m, Weibull, and Suzuki statistical models, as well as the classical Rayleigh and Rician models.

The findings demonstrate that the proposed model offers a more precise and reliable depiction of the channel, especially under high-SNR conditions. For example, at an SNR of 20 dB, both the Nakagami and Weibull models show a higher BER, nearing 0.02 and 0.1, respectively, while the Suzuki model exhibits moderate results. In contrast, the proposed model achieves a substantially lower BER, showcasing its adaptability to varied channel conditions. This detailed comparison underscores the shortcomings of conventional models, which often face challenges in accurately reflecting dynamic environmental changes. On the other hand, the proposed statistical approach is better equipped to capture and respond to subtle variations in channel behaviour. These results highlight its advantages, particularly in situations where accurate and timely channel estimation is critical.

8 Conclusions

This research study investigates the multipath fading channel using the proposed model and compares it with the Rayleigh and Rician channel models. The proposed model is further used to verify the applicability of different modulation schemes such as 64, 256, and 1024 QAM. The MSE and BER performance measures were used for evaluation. The experimental setup provided real-world conditions, including the precise measurement of channel coefficients, which significantly enhanced the accuracy of the model. The BER performance for the Rician fading channel was better than the Rayleigh channel due to the presence of the LOS signal. The analysis of the proposed model highlighted more robustness by including the impact of pilot signals and the evaluation of various fading scenarios.

Pilot-based estimation is a commonly used method for real-time channel assessment; however, it has certain drawbacks, including interpolation inaccuracies, sensitivity to noise, and a limited capacity to represent the complete statistical properties of the channel. In contrast, the statistical model we propose, grounded in extensive empirical data, overcomes these challenges by offering a more thorough and reliable representation of the channel. By eliminating the dependency on pilot signals, our approach minimizes estimation errors and optimizes the use of available resources. As a result, it demonstrates enhanced accuracy and dependability within the framework of our study. The results of the experimental work using the proposed model yielded BER values that align with established standards. Specifically, for 64 QAM, the BER is 2.6 × 10− 5; for 256 QAM, it is 1.9 × 10− 5; and for 1024 QAM, the BER is 1.1 × 10− 5. The analysis of the experimental data yielded MSE values that are within established standards: 3.6 × 10− 3 for 64 QAM, 2.3 × 10− 4 for 256 QAM, and 1.6 × 10− 5 for 1024 QAM.

The results obtained from plotting the BER against SNR reveal that, when using 256 QAM for testing, the proposed statistical model outperforms all the traditional models, including Rayleigh, Rician, Suzuki, Weibull, and Nakagami-m. The proposed model shows a distinct advantage by achieving significantly lower BER at all SNR values and approaches almost zero around 35 dB, demonstrating its robustness and accuracy in capturing the channel’s behaviour.

The proposed model provides the ability to dynamically represent the fading channel characteristics based on the real-time calculated parameters. The overall performance at different amplitude modulation schemes shows a decrease in MSE and BER performance measures as the SNR increases. These improvements in experimental validation and system modelling underline the potential for applying the proposed model in future wireless communication systems, particularly in the context of 5G and IoT networks.