The fast and accurate recognition of the container code and type is very important to speed up the passage of trucks through the gate and improve the efficiency of port. In this article, a practical unified network (PUN) is proposed for the first time to recognize arbitrary-oriented container code and type. The model is based on the encoder-decoder network, which integrates the detection of the second part of container code, the semantic segmentation of the three parts of container code and type, and the classification of single character features of container code and type. The ablation experiments verified the performance of each part of PUN and obtained a stable and reliable model. The comparison experiments compare PUN with the state-of-the-art text detection and end-to-end text recognition algorithms, and the results show that PUN achieves almost the best performance in both container code and type detection and end-to-end recognition. The proposed algorithm has only 15.53 M parameters, with a frame per second (FPS) of 4.86 with an input size of $768\times 768$ , which meets the company’s needs. Our model has been deployed and running in multismart gates of Shanghai Port for nearly a year, showed certain practical application capabilities.