This research is based on the framework of a fully automated smart fruit factory that builds a small and simple fruit grading device, and spots defects in the three fruit models of apples, lemons and oranges which were used as test target, and the entire process of detection is performed in a dark box. There is a ring-shaped LED light to regulate the light source inside the dark box. The fruits to be identified are moved into the dark box by a conveyor belt, an infrared sensor is used to judge whether the fruits are within the shooting range of the image capture area, and then the photos are sent to the SSD (Single Shot Multi Box Detector) neural network model to identify defects. This system screens the surface of apples, lemons and oranges for defects like damage, pest damage, dryness, bruises, etc. and removes them to preserve the fruits that are good in quality. It has been verified by several experiments that the identification accuracy rate can reach upto more than 95%.