Abstract
Agriculture presents challenges in automation, especially so in vision systems. Varying lighting conditions, sporadic diversity, and large amounts of noise create difficulty in detecting target objects. Our Mummy Nuts datasets present these challenges in tiny scale, camouflaged, dark, or even hidden target objects. However, the most recent advancements in Convolutional Neural Networks (CNN) in the object detection task have become increasingly accurate and robust. As there are many different CNNs, selecting which CNN will perform the best may become challenging. This paper proposes a two-dimensional benchmarking methodology to evaluate five popular CNN models (YOLOv3, YOLOv5, CenterNet, Faster R-CNN, and MobileNet SSD) on two NVIDIA GPUs (Tesla T4 and A100). Our benchmarking methodology evaluates accuracy across all models and performance among models on each GPU. Our results show the benefits of selecting models using our Augmented dataset over the Original dataset. CNN Models overall see an increase in recall values during inference by an average of 2.77X (with the highest increase as YOLOv3 by 6.5X). For performance, over both Original and Augmented datasets, the model training time reduces by an average of 4.45X when using A100 over Tesla T4.
This work was supported in part by the NSF research grants CCF #2132049, EEC #1941529, and a COR grant from University of California, Merced.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Almonds, C.: Navel Orangeworm (2022). https://www.almonds.com/almond-industry/industry-news/mummy-nut-removal-ready-set
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)
Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J.M., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88, 303–338 (2009)
Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: SUN database: large-scale scene recognition from abbey to zoo. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485–3492 (2010)
Dollár, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: an evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 34, 743–761 (2012)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Girshick, R.: Fast R-CNN. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440–1448 (2015)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: Centernet: keypoint triplets for object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6569–6578 (2019)
Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Hui, Y., Lien, J., Lu, X.: Early experience in benchmarking edge AI processors with object detection workloads. In: Gao, W., Zhan, J., Fox, G., Lu, X., Stanzione, D. (eds.) Bench 2019. LNCS, vol. 12093, pp. 32–48. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49556-5_3
Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 483–499. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_29
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Canziani, A., Paszke, A., Culurciello, E.: An analysis of deep neural network models for practical applications. arXiv preprint arXiv:1605.07678 (2016)
Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2
Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
Jocher, G., et al.: Ultralytics/YOLOv5: v6.1 - TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference (2022)
Redmon, J.: Darknet: Open Source Neural Networks in C (2013/2016). http://pjreddie.com/darknet/
Hui, J.: mAP (mean Average Precision) for Object Detection (2022). https://medium.com/p/45c121a31173
NVIDIA. NVIDIA A100 Tensor Core GPU Datasheet (2022). https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf
Intel: Intel Xeon Gold 6336Y Processor Datasheet (2022). https://www.intel.com/content/www/us/en/products/sku/215280/intel-xeon-gold-6336y-processor-36m-cache-2-40-ghz/specifications.html
NVIDIA: NVIDIA Tesla T4 Tensor Core GPU Datasheet (2022). https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-t4/t4-tensor-core-datasheet-951643.pdf
Wang, J., Xu, C., Yang, W., Yu, L.: A normalized Gaussian Wasserstein distance for tiny object detection. arXiv preprint arXiv:2110.13389 (2021)
Lu, Y., Young, S.: A survey of public datasets for computer vision tasks in precision agriculture. Comput. Electron. Agric. 178, 105760 (2020)
Choi, D., Lee, W., Ehsani, R., Schueller, J., Roka, F.: Detection of dropped citrus fruit on the ground and evaluation of decay stages in varying illumination conditions. Comput. Electron. Agric. 127, 109–119 (2016)
Gan, H., Lee, W., Alchanatis, V., Ehsani, R., Schueller, J.: Immature green citrus fruit detection using color and thermal images. Comput. Electron. Agric. 152, 117–125 (2018)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Qin, R., Liu, Q., Gao, G., Huang, D., Wang, Y.: MRDET: a multi-head network for accurate oriented object detection in aerial images. arXiv preprint arXiv:2012.13135 (2020)
Yi, J., Wu, P., Liu, B., Huang, Q., Qu, H., Metaxas, D.: Oriented object detection in aerial images with box boundary-aware vectors. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2150–2159 (2021)
Zand, M., Etemad, A., Greenspan, M.: Oriented bounding boxes for small and freely rotated objects. IEEE Trans. Geosci. Remote Sens. 60, 1–15 (2022)
Han, J., Ding, J., Xue, N., Xia, G.-S.: ReDet: a rotation-equivariant detector for aerial object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2786–2795 (2021)
Hui, Y., Lien, J., Lu, X.: Characterizing and accelerating end-to-end EdgeAI inference systems for object detection applications. In: 2021 IEEE/ACM Symposium on Edge Computing (SEC), pp. 01–12 (2021)
Gao, C., Gutierrez, A., Rajan, M., Dreslinski, R.G., Mudge, T., Wu, C.-J.: A study of mobile device utilization. In: 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 225–234 (2015)
Allan, A.: Benchmarking Edge Computing (2022). https://aallan.medium.com/benchmarking-edge-computing-ce3f13942245
Zhang, Q., et al.: A survey on deep learning benchmarks: do we still need new ones? In: Zheng, C., Zhan, J. (eds.) Bench 2018. LNCS, vol. 11459, pp. 36–49. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32813-9_5
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ng, D. et al. (2023). Benchmarking Object Detection Models with Mummy Nuts Datasets. In: Gainaru, A., Zhang, C., Luo, C. (eds) Benchmarking, Measuring, and Optimizing. Bench 2022. Lecture Notes in Computer Science, vol 13852. Springer, Cham. https://doi.org/10.1007/978-3-031-31180-2_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-31180-2_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31179-6
Online ISBN: 978-3-031-31180-2
eBook Packages: Computer ScienceComputer Science (R0)