Speech communication human-machine interfaces exploit automatic speech recognition to implement speech-to-text conversion. Unfortunately, in the past, not much effort has been devoted to add punctuation marks to the recognized word chain after speech recognition. This affects human readability and makes interpretation hard. This paper presents an effort to restore punctuation marks by keeping low the latency resulting from this post-processing step. The approach exploits the prosodic structure and proposes a sequential modelling paradigm based on recurrent neural networks. Results show satisfying punctuation restoration abilities, especially taking into account that sentence boundaries are reliably detected. Even if the predicted punctuation sequence is not error free w.r.t. writing standards, human perception is expected to “repair” these errors more easily compared to the case when no punctuation is given at all and the reader is left in confusion regarding the basic segmentation of the word chain.