Skip to main content
SpringerLink
Log in

Attention-based neural joint source-channel coding of text for point to point and broadcast channel

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

In this work, we consider the transmissions of structured data such as text over a noisy channel and correlated texts over a broadcast channel. As the separate source-channel coding principle no longer holds in such scenarios, we propose a joint source-channel coding scheme which is based on deep learning architecture. In order to enhance the convergence speed, we adopt the bidirectional gated recurrent unit at the encoder. For the decoder, to improve the recovery quality, we propose the following two types of strategies: (1) After a unidirectional neural network based decoder is used, a generative adversarial network is applied to train the whole joint source-channel coding framework and pointwise mutual information is added to the objective function of beam search process; (2) Rather than using a unidirectional neural network-based decoder, we develop a bidirectional neural network based and bidirectional attention mechanism integrated decoder to utilize past and future information. Experiments under different types of channels show that our schemes are superior to the existing deep learning joint source-channel coding method and in the case of low bit budget, long sentence length and small channel signal to noise ratio, our models are significantly superior to those of separate source-channel coding. In addition, we extend the proposed unidirectional and bidirectional decoders to the broadcast channel. Additionally, to improve the performance of unidirectional decoding, we utilize not only the correlation between adjacent words in the same text but also the correlation between words in different languages with the same meaning in the beam search process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. CoRR. arXiv:abs/1409.0473

  • Balsa J, Fresnedo Ó, Domínguez-Bolaño T, García-Naya JA, Castedo L (2019) Experimental evaluation of analog encoding for the wireless transmission of still images. In: 2019 IEEE 20th international workshop on signal processing advances in wireless communications (SPAWC), pp 1–5

  • Bennatan A, Burshtein D, Caire G, Shamai S (2006) Superposition coding for side-information channels. IEEE Trans Inf Theory 52(5):1872–1889

    Article  MathSciNet  Google Scholar 

  • Bourtsoulatze E, Kurka D, Gündüz D (2019) Deep joint source-channel coding for wireless image transmission. In: ICASSP 2019—2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 4774–4778

  • Cheok MJ, Omar Z, Jaward MH (2019) A review of hand gesture and sign language recognition techniques. Int J Mach Learn Cybern 10:131–153

    Article  Google Scholar 

  • Choi K, Tatwawadi K, Grover A, Weissman T, Ermon S (2019) Neural joint source-channel coding. In: ICML

  • Cover T (1972) Broadcast channels. IEEE Trans Inf Theory 18:214

    Article  MathSciNet  Google Scholar 

  • Farsad N, Rao M, Goldsmith A (2018) Deep learning for joint source-channel coding of text. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 2326–2330

  • Gao Y, Tuncel E (2011) Wyner-ziv coding over broadcast channels: hybrid digital/analog schemes. IEEE Trans Inf Theory 57(9):5660–5672

    Article  MathSciNet  Google Scholar 

  • Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: NIPS

  • Graves A (2012) Sequence transduction with recurrent neural networks. arXiv:abs/1211.3711

  • Guzmán F, Chen P, Ott M, Pino J, Lample G, Koehn P, Chaudhary V, Ranzato M (2019) Two new evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. In: EMNLP/IJCNLP

  • Hershey J, Roux JL, Weninger F (2014) Deep unfolding: model-based inspiration of novel deep architectures. arXiv:abs/1409.2574

  • Huffman D (1952) A method for the construction of minimum-redundancy codes. Resonance 11:91–99

    Article  Google Scholar 

  • Jain A, Jain A, Chauhan N, Singh V, Thakur N (2017) Information retrieval using cosine and Jaccard similarity measures in vector space model. Int J Comput Appl 164:28–30

    Google Scholar 

  • Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. CoRR. arXiv:abs/1412.6980

  • Li J, Galley M, Brockett C, Gao J, Dolan W (2016) A diversity-promoting objective function for neural conversation models. arXiv:abs/1510.03055

  • Li J, Monroe W, Shi T, Jean S, Ritter A, Jurafsky D (2017) Adversarial learning for neural dialogue generation. In: The 2017 conference on empirical methods in natural language processing. arXiv:abs/1701.06547

  • Liu T, Chen X (2020) Deep learning-based belief propagation algorithm over non-binary finite fields. In: 2020 International conference on wireless communications and signal processing (WCSP), pp 164–169. https://doi.org/10.1109/WCSP49889.2020.9299875

  • Luong T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv:abs/1508.04025

  • Navarro G (2001) A guided tour to approximate string matching. ACM Comput Surv 33:31–88

    Article  Google Scholar 

  • Nor MIAM, Izhar M, Norulhusna Ahmad, Kaidi H (2018) Joint source channel decoding exploiting 2 d source correlation with parameter estimation for image transmission over Rayleigh fading channels. vol 8, pp 2633–2642

  • Pejoski S, Kafedziski V (2013) Joint source channel coding framework for real time h.264/avc video transmission over wireless mimo channels using outage probability. In: 2013 11th international conference on telecommunications in modern satellite, cable and broadcasting services (TELSIKS) 01, pp 221–224

  • Rao M, Farsad N, Goldsmith A (2018) Variable length joint source-channel coding of text using deep neural networks. In: 2018 IEEE 19th international workshop on signal processing advances in wireless communications (SPAWC), pp 1–5

  • Reed I, Solomon G (1960) Polynomial codes over certain finite fields. J Soc Ind Appl Math 8:300–304

    Article  MathSciNet  Google Scholar 

  • Shannon C (1948) A mathematical theory of communication. Bell Syst Tech J 27:379–423

    Article  MathSciNet  Google Scholar 

  • Shemona JS, Chellappan AK (2020) Segmentation techniques for early cancer detection in red blood cells with deep learning-based classifier—a comparative approach. IET Image Process 14:1726–1732

    Article  Google Scholar 

  • Shewalkar A, Nyavanandi D, Ludwig SA (2019) Performance evaluation of deep neural networks applied to speech recognition: RNN, LSTM and GRU. J Artif Intell Soft Comput Res 9:235–245

    Article  Google Scholar 

  • Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    MathSciNet  MATH  Google Scholar 

  • Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: NIPS

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: NIPS

  • Wu C, Wang B (2017) Extracting topics based on word2vec and improved jaccard similarity coefficient. In: 2017 IEEE second international conference on data science in Cyberspace (DSC), pp 389–397

  • Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J, Shah A, Johnson M, Liu X, Kaiser L, Gouws S, Kato Y, Kudo T, Kazawa H, Stevens K, Kurian G, Patil N, Wang W, Young C, Smith J, Riesa J, Rudnick A, Vinyals O, Corrado GS, Hughes M, Dean J (2016) Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv:abs/1609.08144

  • Wu L, Xia Y, Zhao L, Tian F, Qin T, Lai J, Liu T (2018) Adversarial neural machine translation. arXiv:abs/1704.06933

  • Xu L, Zhang X, Dong Q (2020) Cluecorpus2020: a large-scale chinese corpus for pre-training language model. arXiv:abs/2003.01355

  • Yang Z, Chen W, Wang F, Xu B (2018) Improving neural machine translation with conditional sequence generative adversarial nets. arXiv:abs/1703.04887

  • Zhang J, Cao Y, Han G, Fu X (2019) Deep neural network-based underwater OFDM receiver. IET Commun 13:1998–2002

    Article  Google Scholar 

  • Zhang X, LeCun Y (2015) Text understanding from scratch. arXiv:abs/1502.01710

  • Zhang X, Su J, Qin Y, Liu Y, Ji R, Wang H (2018) Asynchronous bidirectional decoding for neural machine translation. In: The thirty-second AAAI conference on artificial intelligence. arXiv:abs/1801.05122

  • Zhou L, Zhang J, Zong C (2019) Synchronous bidirectional neural machine translation. Trans Assoc Comput Linguist 7:91–105

    Article  Google Scholar 

  • Ziv J, Lempel A (1977) A universal algorithm for sequential data compression. IEEE Trans Inf Theory 23:337–343

    Article  MathSciNet  Google Scholar 

Download references

Funding

National Natural foundation of China (61301181, U1530120) and the Scientific Research Foundation of Central South University.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Central South University, Changsha, China

    Ting Liu & Xuechen Chen

  2. School of Electronics and Information Technology, Sun Yat-Sen University, Guangzhou, China

    Ting Liu

Authors
  1. Ting Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuechen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuechen Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by NSF of China under Grant No. 61301181, Fundamental Research Funds for Central Universities of the Central South University, Scientific Research Starting Foundation of Central South University.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, T., Chen, X. Attention-based neural joint source-channel coding of text for point to point and broadcast channel. Artif Intell Rev 55, 2379–2407 (2022). https://doi.org/10.1007/s10462-021-10067-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-021-10067-3

Keywords

Search

Navigation