Automatic modulation classification (AMC) aims to identify the modulation format of the received signals corrupted by the noise, which plays a major role in radio monitoring. In this paper, we propose a novel cascaded convolutional neural network (CasCNN)-based hierarchical digital modulation classification scheme, where M-ary phase shift keying (PSK) and M-ary quadrature amplitude modulation (QAM) modulation formats are considered to be classified. In CasCNN, two-block convolutional neural networks are cascaded. The first block network is utilized to classify the different classes of modulation formats, namely PSK and QAM. The second block is designed to identify the indexes of the modulations in the same PSK or QAM class. Moreover, it is noted that the gird constellation diagram extracted from the received signal is utilized as the inputs to the CasCNN. Extensive simulations demonstrate that CasCNN yields performance gain and performs stronger robustness to frequency offset compared with other recent methods. Specifically, CasCNN achieves 90% classification accuracy at 4 dB signal-to-noise ratio when the symbol length is set as 256.