Neural network-based models have been implemented to predict various health indicators of asphalt pavement using pavement historical detection data. Unfortunately, their accuracy and reliability are not acceptable owing to their shallow architecture. To solve the issue, this study proposed an improved Transformer network to predict asphalt pavement health, called the Transformer with forward and reversed time series (Transformer FRTS). In terms of the input data, Transformer FRTS uses a new data form, so-called the random difference time series, to reduce the time dependency of the network prediction. In terms of the network architecture, the proposed network uses its encoder and decoder to obtain the data association from the forward and reverse time series. In addition, Transformer FRTS uses a post-processing decision criterion to improve the accuracy and reliability of prediction. The numerical experiment using the detection data from RIOHTrack full-scale track demonstrates that the proposed network has state-of-the-practice performance in asphalt pavement health prediction.