Nuclear cataract is one of the most common types of cataract. In the recent, ophthalmologists are increasingly using anterior segment optical coherence tomography (AS-OCT) images to diagnose many ocular diseases including cataract. The relationship between cataract and the lens opacity based on AS-OCT images has been being studied in clinical pioneer research. However, using AS-OCT images to classify cataract automatically based on computer-aided diagnosis (CAD) technique has not been seriously studied. This paper proposes a novel Convolutional Neural Network (CNN) model named GraNet for nuclear cataract classification based on AS-OCT images. In the GraNet, we introduce a grading block to learn high-level feature representations based on the pointwise convolution method. To further improve the classification performance, we propose a simple and efficient cross-training method is comprised of focal loss and cross-entropy loss. Extensive experiments are conducted on the AS-OCT image dataset, the results demonstrate that the proposed methods achieve better nuclear cataract classification results than baselines.