Skip to main content
SpringerLink
Log in

Overfitting effect of artificial neural network based nonlinear equalizer: from mathematical origin to transmission evolution

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Overfitting effect of artificial neural network (ANN) based nonlinear equalizer (NLE) leads to a trap of bit error ratio (BER) overestimation in optical fiber communication system, especially when the performance is evaluated by the commonly-used pseudo-random binary sequence (PRBS). First, we mathematically investigate the PRBS generation and Gray code mapping rules, in comparison with the use of Mersenne Twister random sequence (MTRS). Under the condition of a symbol erasure channel, we identify that ANN can recognize both the PRBS generation and symbol mapping rules, by increasing the weights of NLE at specific positions, whereas the MTRS is currently safe owing to the limited input length of current ANN based NLE. Then, we design four channel models of fiber optical transmission to experimentally examine various impairments on the evolution of overfitting effect. When both the additive white Gaussian noise (AWGN) channel and the bandwidth limited channel are considered, the mitigation of overfitting becomes possible by the use of pruned PRBS (P-PRBS) training set with removing the generation and mapping rules determined input symbols. However, as for both the chromatic dispersion (CD) uncompensated channel and the CD managed channel, the overfitting effect becomes serious, because both CD and fiber nonlinearity induced inter-symbol interference (ISI) is beneficial for ANN to identify the PRBS symbol rules. Finally, possible solutions to mitigate the overfitting effect are summarized.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks. In: Proceedings of Advances in neural information processing systems (NIPS), 2012. 1097–1105

  2. Hinton G, Deng L, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Magaz, 2012, 28: 82–97

    Article  Google Scholar 

  3. Sagiroglu S, Yavanoglu U, Guven E N. Web based machine learning for language identification and translation. In: Proceedings of the 6th International Conference on Machine Learning and Applications, Cincinnati, 2007. 280–285

  4. Jarajreh M A, Giacoumidis E, Aldaya I, et al. Artificial neural network nonlinear equalizer for coherent optical OFDM. IEEE Photon Technol Lett, 2015, 27: 387–390

    Article  Google Scholar 

  5. Giacoumidis E, Le S T, Ghanbarisabagh M, et al. Fiber nonlinearity-induced penalty reduction in CO-OFDM by ANN-based nonlinear equalization. Opt Lett, 2015, 40: 5113–5116

    Article  Google Scholar 

  6. Luo M, Gao F, Li X, et al. Transmission of 4×50-Gb/s PAM-4 signal over 80-km single mode fiber using neural network. In: Proceedings of Optical Fiber Communication Conference, 2018. M2F.2

  7. Yang Z, Gao F, Fu S, et al. Radial basis function neural network enabled C-band 4×50-Gb/s PAM-4 transmission over 80 km SSMF. Opt Lett, 2018, 43: 3542–3545

    Article  Google Scholar 

  8. Chuang C, Liu L, Wei C, et al. Convolutional neural network based nonlinear classifier for 112-Gbps high speed optical link. In: Proceedings of Optical Fiber Communication Conference, 2018. W2A.43

  9. Ye C, Zhang D, Hu X, et al. Recurrent neural network (RNN) based end-to-end nonlinear management for symmetrical 50 Gbps NRZ PON with 29 dB+ loss budget. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  10. Karanov B, Chagnon M, Thouin F, et al. End-to-end deep learning of optical fiber communications. J Lightw Technol, 2018, 36: 4843–4855

    Article  Google Scholar 

  11. Karanov B, Lavery B, Bayvel P, et al. End-to-end optimized transmission over dispersive intensity-modulated channels using bidirectional recurrent neural networks. Opt Express, 2019, 27: 19650–19663

    Article  Google Scholar 

  12. Wang D, Zhang M, Li Z, et al. Modulation format recognition and OSNR estimation using CNN-based deep learning. IEEE Photon Technol Lett, 2017, 29: 1667–1670

    Article  Google Scholar 

  13. Dong Z, Khan F N, Sui Q, et al. Optical performance monitoring: a review of current and future technologies. J Lightw Technol, 2016, 34: 525–543

    Article  Google Scholar 

  14. Chen X, Li B, Shamsabardeh M, et al. On real-time and self-taught anomaly detection in optical networks using hybrid unsupervised/supervised learning. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  15. Charalabopoulos G, Stavroulakis P, Aghvami A H. A frequency-domain neural network equalizer for OFDM. In: Proceedings of IEEE Global Telecommunications Conference, 2003. 571–575

  16. Rajbhandari S, Ghassemlooy Z, Angelova M. Effective denoising and adaptive equalization of indoor optical wireless channel with artificial light using the discrete wavelet transform and artificial neural network. J Lightw Technol, 2009, 27: 4493–4500

    Article  Google Scholar 

  17. ITU-T. Digital test patterns for performance measurements on digital transmission equipment. CCITT Recommendation O.150. https://www.itu.int/rec/T-REC-O.150-199210-S/en

  18. IEEE Standards Association. IEEE Standard for Ethernet Amendment 10: Media Access Control Parameters, Physical Layers, and Management Parameters for 200 Gb/s and 400 Gb/s Operation. IEEE Std 802.3bs. https://standards.ieee.org/standard/802_3bs-2017.html

  19. Eriksson T A, Bülow H, Leven A. Applying neural networks in optical communication systems: possible pitfalls. IEEE Photon Technol Lett, 2017, 29: 2091–2094

    Article  Google Scholar 

  20. Shu L, Li J, Wan Z, et al. Overestimation trap of artificial neural network: learning the rule of PRBS. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  21. Chuang C, Liu L, Wei C, et al. Study of training patterns for employing deep neural networks in optical communication systems. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  22. Yi L, Liao T, Huang L, et al. Machine learning for 100 Gb/s/A passive optical network. J Lightw Technol, 2019, 37: 1621–1630

    Article  Google Scholar 

  23. Matsumoto M, Nishimura T. Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans Model Comput Simul, 1998, 8: 330

    Article  MATH  Google Scholar 

  24. Doran R W. The Gray code. J Univ Comput Sci, 2007, 13: 1573–1597

    MathSciNet  Google Scholar 

  25. Agrawal G P. Nonlinear Fiber Optics. 4th ed. San Diego: Academic Press, 2001

    MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by National Key R&D Program of China (Grant No. 2018YFB1801301) National Natural Science Foundation of China (Grant No. 61875061), and Key Project of R&D Program of Hubei Province (Grant No. 2018AAA041).

Author information

Authors and Affiliations

  1. Wuhan National Laboratory for Optoelectronics, School of Optical and Electronic Information, Huazhong University of Science and Technology, Wuhan, 430074, China

    Zheng Yang, Songnian Fu, Ming Tang & Deming Liu

  2. Alibaba Infrastructure Service, Alibaba Group, Hangzhou, 311121, China

    Fan Gao

Authors
  1. Zheng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Songnian Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Deming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Songnian Fu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Z., Gao, F., Fu, S. et al. Overfitting effect of artificial neural network based nonlinear equalizer: from mathematical origin to transmission evolution. Sci. China Inf. Sci. 63, 160305 (2020). https://doi.org/10.1007/s11432-020-2873-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-020-2873-x

Keywords

Search

Navigation