Subjective evaluation is a commonly used method in the real recognition process. Generally, two fuzziness can be found in evaluation information, namely what values should be given to fully describe the information and how to distinguish different values. The recently developed probabilistic hesitant fuzzy set could perfectly address these issues. In this article, we propose a dual-fuzzy convolutional neural network (DF-CNN) by fusing the hot neural network algorithm into the probabilistic hesitant fuzzy environment and then using it in a practical handwritten image recognition process. For this new DF-CNN, we provide the whole calculation process including the forward propagation, backward propagation, and parameter updating calculations. Also, the optimization algorithm of the DF-CNN is given to derive its optimal results. Finally, we apply the DF-CNN and its optimization algorithm to deal with a real issue, namely the handwritten numeral image recognition. The calculation process and the comparison fully demonstrate the feasibility and effectiveness of the proposed new model and algorithm.