Yield and maturity estimation of apples in orchards using a 3-step deep learning–based method
Main Article Content
Keywords
deep learning, image analysis, maturity estimation, yield estimation
Abstract
Yield and maturity estimation of apples in orchards before harvest is essential for labor resource management. Current yield and maturity estimation are usually manually conducted, which is neither accurate nor efficient. In this paper, a 3-step deep learning–based approach for yield estimation and maturity classification is presented to address these issues. The proposed framework included an optimized detection network to count the visible fruits from both sides of a tree, a classification network to filter out mis-detected objects and perform maturity estimation, and a fruit load estimation algorithm to obtain the total fruit count of a tree. Images from three different apple orchards were collected to evaluate the performance of the proposed method. According to a series of comparative experiments, the proposed method outperformed several detection networks regarding the counting accuracy, indicating that an optimized architecture of the detection network combined with a fine classification network is necessary for enhanced precision of yield estimation. The presented workflow can be readily extended to other fruit crops for automation yield and maturity estimation featuring high efficiency and accuracy.
References
Apolo-Apolo, O.E., Pérez-Ruiz, M., Martínez-Guanter, J. and Valente, J., 2020. A cloud-based environment for generating yield estimation maps from apple orchards using uav imagery and a deep learning technique. Frontiers in Plant Science 11. 10.3389/fpls.2020.01086
Badrinarayanan, V., Kendall, A. and Cipolla, R., 2017. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence 39: 2481–2495. 10.1109/TPAMI.2016.2644615
Bandi, S.R., Varadharajan, A. and Chinnasamy, A., 2013. Performance evaluation of various statistical classifiers in detecting the diseased citrus leaves. International Journal of Engineering Science & Technology 5: 298–307.
Bargoti, S. and Underwood, J., 2017. Deep fruit detection in orchards. Proceedings of the IEEE 3626–3633. 10.1109/ICRA.2017.7989417
Bochkovskiy, A., Wang, C.Y. and Liao, H., 2020. Yolov4: optimal speed and accuracy of object detection. 2004: 10934.
Chen, S.W., Shivakumar, S.S., D Cunha, S., Das, J., Okon, E., Chao, Q., Taylor, C.J. and Kumar, V., 2017. Counting apples and oranges with deep learning: a data-driven approach. IEEE Robotics & Automation Letters 2: 781–788. 10.1109/LRA.2017.2651944
Chen, Y., An, X., Gao, S., Li, S. and Kang, H., 2021. A deep learning-based vision system combining detection and tracking for fast on-line citrus sorting. Frontiers in Plant Science 12. 10.3389/fpls.2021.622062
Chen, Y., Lee, W.S., Gan, H., Peres, N. and He, Y., 2019. Strawberry yield prediction based on a deep neural network using high-resolution aerial orthoimages. Remote Sensing 11. 10.3390/rs11131584
Costa, L., Ampatzidis, Y., Rohla, C., Maness, N., Cheary, B. and Zhang, L., 2021. Measuring pecan nut growth utilizing machine vision and deep learning for the better understanding of the fruit growth curve. Computers and Electronics in Agriculture 181: 105964. 10.1016/j.compag.2020.105964
Das, J., Cross, G., Chao, Q., Makineni, A. and Kumar, V., 2015. Devices, systems, and methods for automated monitoring enabling precision agriculture. In: Proceedings of the IEEE International Conference on Automation Science & Engineering, October 2015, pp. 462–469. 10.1109/CoASE.2015.7294123
Faisal, M., Albogamy, F., Elgibreen, H., Algabri, M. and Alqershi, F.A., 2020. Deep learning and computer vision for estimating date fruits type, maturity level, and weight. IEEE Access 8: 206770–206782. 10.1109/ACCESS.2020.3037948
Fu, L., 2020. Real-time kiwifruit detection in orchard using deep learning on android smartphones for yield estimation. Computers and Electronics in Agriculture 179: 105856. 10.1016/j.compag.2020.105856
Garillos-Manliguez, C.A. and Chiang, J.Y., 2021. Multimodal deep learning and visible-light and hyperspectral imaging for fruit maturity estimation. Sensors (Basel, Switzerland) 21: 1288. 10.3390/s21041288
He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. IEEE 770–778. 10.1109/CVPR.2016.90
Hni, N., Roy, P. and Isler, V., 2020. A comparative study of fruit detection and counting methods for yield mapping in apple orchards. Journal of Field Robotics 37: 263–282. 10.1002/rob.21902
Kang, H. and Chen, C., 2019. Fruit detection and segmentation for apple harvesting using visual sensor in orchards. Sensors (Basel, Switzerland) 19. 10.3390/s19204599
Kang, H. and Chen, C., 2020. Fast implementation of real-time fruit detection in apple orchards using deep learning–ScienceDirect. Computers and Electronics in Agriculture 168. 10.1016/j.compag.2019.105108
Liu, S., Qi, L., Qin, H., Shi, J. and Jia, J., 2018a. Path aggregation network for instance segmentation. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8759–8768. 10.1109/CVPR.2018.00913
Liu, Z., Yong, H., Cen, H. and Lu, R., 2018b. Deep feature representation with stacked sparse auto-encoder and convolutional neural network for hyperspectral imaging-based detection of cucumber defects. Transactions of the Asabe 61: 425–436. 10.13031/trans.12214
Long, H. and Schupp, J., 2018. Sensing and automation in pruning of apple trees: a review. Agronomy 8: 211. 10.3390/agronomy8100211
Mesa, A.R. and Chiang, J.Y., 2021. Multi-input deep learning model with rgb and hyperspectral imaging for banana grading. Agriculture 11. 10.3390/agriculture11080687
Nasiri, A., Taheri-Garavan, D.A. and Zhang, Y.D., 2019. Image-based deep learning automated sorting of date fruit. Postharvest Biology and Technology 153: 133–141. 10.1016/j.postharvbio.2019.04.003
Otsu, N., 2007. A threshold selection method from gray-level histograms. IEEE Transactions on Systems Man & Cybernetics 9: 62–66. 10.1109/TSMC.1979.4310076
Pan, S.Q., Qiao, J.F., Wang, R, Yu, H.L., Wang, C., Taylor, K., Pan, H. 2021. Intelligent diagnosis of northern corn leaf blight with deep learning model. Journal of Integrative Agriculture 2022: 21(4): 1094–1. 10.1016/S2095-3119(21)63707
Ren, S., He, K., Girshick, R. and Sun, J., 2017. Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis & Machine Intelligence 39: 1137–1149. 10.1109/TPAMI.2016.2577031
Roy, P., Kislay, A., Plonski, P.A., Luby, J. and Isler, V., 2019. Vision-based preharvest yield mapping for apple orchards. Computers and Electronics in Agriculture 164: 104879. 10.1016/j.compag.2019.104897
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.C., 2018. Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510–4520. 10.1109/CVPR.2018.00474
Tan, M. and Le, Q.V., 2019. Efficientnet: rethinking model scaling for convolutional neural networks. 2019: 10691–10700.
Tu, S., Pang, J., Liu, H., Zhuang, N. and Xue, Y., 2020. Passion fruit detection and counting based on multiple scale faster r-cnn using rgb-d images. Precision Agriculture 21: 1072–1091. 10.1007/s11119-020-09709-3
Wan, S. and Goudos, S., 2019. Faster r-cnn for multi-class fruit detection using a robotic vision system. Computer Networks 168: 107036. 10.1016/j.comnet.2019.107036
Wang, Q.I., Nuske, S., Bergerman, M. and Singh, S., 2013. Automated crop yield estimation for apple orchards. In: Proceedings of the Proceedings of International Symposium of Experimental Robotics, pp. 745–758. 10.1007/978-3-319-00065-7_50
Zemmour, E., Kurtser, P. and Edan, Y., 2019. Automatic parameter tuning for adaptive thresholding in fruit detection. Sensors (Basel, Switzerland) 19: 2130. 10.3390/s19092130
Zhao, Z.Q., Zheng, P., Xu, S.T. and Wu, X., 2019. Object detection with deep learning: a review. arXiv e-prints 30: 3212–3232. 10.1109/TNNLS.2018.2876865
Zhu, Y. and Newsam, S., 2018. Densenet for dense flow. In: Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), September 2017, pp. 790–794. 10.1109/ICIP.2017.8296389
Badrinarayanan, V., Kendall, A. and Cipolla, R., 2017. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence 39: 2481–2495. 10.1109/TPAMI.2016.2644615
Bandi, S.R., Varadharajan, A. and Chinnasamy, A., 2013. Performance evaluation of various statistical classifiers in detecting the diseased citrus leaves. International Journal of Engineering Science & Technology 5: 298–307.
Bargoti, S. and Underwood, J., 2017. Deep fruit detection in orchards. Proceedings of the IEEE 3626–3633. 10.1109/ICRA.2017.7989417
Bochkovskiy, A., Wang, C.Y. and Liao, H., 2020. Yolov4: optimal speed and accuracy of object detection. 2004: 10934.
Chen, S.W., Shivakumar, S.S., D Cunha, S., Das, J., Okon, E., Chao, Q., Taylor, C.J. and Kumar, V., 2017. Counting apples and oranges with deep learning: a data-driven approach. IEEE Robotics & Automation Letters 2: 781–788. 10.1109/LRA.2017.2651944
Chen, Y., An, X., Gao, S., Li, S. and Kang, H., 2021. A deep learning-based vision system combining detection and tracking for fast on-line citrus sorting. Frontiers in Plant Science 12. 10.3389/fpls.2021.622062
Chen, Y., Lee, W.S., Gan, H., Peres, N. and He, Y., 2019. Strawberry yield prediction based on a deep neural network using high-resolution aerial orthoimages. Remote Sensing 11. 10.3390/rs11131584
Costa, L., Ampatzidis, Y., Rohla, C., Maness, N., Cheary, B. and Zhang, L., 2021. Measuring pecan nut growth utilizing machine vision and deep learning for the better understanding of the fruit growth curve. Computers and Electronics in Agriculture 181: 105964. 10.1016/j.compag.2020.105964
Das, J., Cross, G., Chao, Q., Makineni, A. and Kumar, V., 2015. Devices, systems, and methods for automated monitoring enabling precision agriculture. In: Proceedings of the IEEE International Conference on Automation Science & Engineering, October 2015, pp. 462–469. 10.1109/CoASE.2015.7294123
Faisal, M., Albogamy, F., Elgibreen, H., Algabri, M. and Alqershi, F.A., 2020. Deep learning and computer vision for estimating date fruits type, maturity level, and weight. IEEE Access 8: 206770–206782. 10.1109/ACCESS.2020.3037948
Fu, L., 2020. Real-time kiwifruit detection in orchard using deep learning on android smartphones for yield estimation. Computers and Electronics in Agriculture 179: 105856. 10.1016/j.compag.2020.105856
Garillos-Manliguez, C.A. and Chiang, J.Y., 2021. Multimodal deep learning and visible-light and hyperspectral imaging for fruit maturity estimation. Sensors (Basel, Switzerland) 21: 1288. 10.3390/s21041288
He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. IEEE 770–778. 10.1109/CVPR.2016.90
Hni, N., Roy, P. and Isler, V., 2020. A comparative study of fruit detection and counting methods for yield mapping in apple orchards. Journal of Field Robotics 37: 263–282. 10.1002/rob.21902
Kang, H. and Chen, C., 2019. Fruit detection and segmentation for apple harvesting using visual sensor in orchards. Sensors (Basel, Switzerland) 19. 10.3390/s19204599
Kang, H. and Chen, C., 2020. Fast implementation of real-time fruit detection in apple orchards using deep learning–ScienceDirect. Computers and Electronics in Agriculture 168. 10.1016/j.compag.2019.105108
Liu, S., Qi, L., Qin, H., Shi, J. and Jia, J., 2018a. Path aggregation network for instance segmentation. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8759–8768. 10.1109/CVPR.2018.00913
Liu, Z., Yong, H., Cen, H. and Lu, R., 2018b. Deep feature representation with stacked sparse auto-encoder and convolutional neural network for hyperspectral imaging-based detection of cucumber defects. Transactions of the Asabe 61: 425–436. 10.13031/trans.12214
Long, H. and Schupp, J., 2018. Sensing and automation in pruning of apple trees: a review. Agronomy 8: 211. 10.3390/agronomy8100211
Mesa, A.R. and Chiang, J.Y., 2021. Multi-input deep learning model with rgb and hyperspectral imaging for banana grading. Agriculture 11. 10.3390/agriculture11080687
Nasiri, A., Taheri-Garavan, D.A. and Zhang, Y.D., 2019. Image-based deep learning automated sorting of date fruit. Postharvest Biology and Technology 153: 133–141. 10.1016/j.postharvbio.2019.04.003
Otsu, N., 2007. A threshold selection method from gray-level histograms. IEEE Transactions on Systems Man & Cybernetics 9: 62–66. 10.1109/TSMC.1979.4310076
Pan, S.Q., Qiao, J.F., Wang, R, Yu, H.L., Wang, C., Taylor, K., Pan, H. 2021. Intelligent diagnosis of northern corn leaf blight with deep learning model. Journal of Integrative Agriculture 2022: 21(4): 1094–1. 10.1016/S2095-3119(21)63707
Ren, S., He, K., Girshick, R. and Sun, J., 2017. Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis & Machine Intelligence 39: 1137–1149. 10.1109/TPAMI.2016.2577031
Roy, P., Kislay, A., Plonski, P.A., Luby, J. and Isler, V., 2019. Vision-based preharvest yield mapping for apple orchards. Computers and Electronics in Agriculture 164: 104879. 10.1016/j.compag.2019.104897
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.C., 2018. Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510–4520. 10.1109/CVPR.2018.00474
Tan, M. and Le, Q.V., 2019. Efficientnet: rethinking model scaling for convolutional neural networks. 2019: 10691–10700.
Tu, S., Pang, J., Liu, H., Zhuang, N. and Xue, Y., 2020. Passion fruit detection and counting based on multiple scale faster r-cnn using rgb-d images. Precision Agriculture 21: 1072–1091. 10.1007/s11119-020-09709-3
Wan, S. and Goudos, S., 2019. Faster r-cnn for multi-class fruit detection using a robotic vision system. Computer Networks 168: 107036. 10.1016/j.comnet.2019.107036
Wang, Q.I., Nuske, S., Bergerman, M. and Singh, S., 2013. Automated crop yield estimation for apple orchards. In: Proceedings of the Proceedings of International Symposium of Experimental Robotics, pp. 745–758. 10.1007/978-3-319-00065-7_50
Zemmour, E., Kurtser, P. and Edan, Y., 2019. Automatic parameter tuning for adaptive thresholding in fruit detection. Sensors (Basel, Switzerland) 19: 2130. 10.3390/s19092130
Zhao, Z.Q., Zheng, P., Xu, S.T. and Wu, X., 2019. Object detection with deep learning: a review. arXiv e-prints 30: 3212–3232. 10.1109/TNNLS.2018.2876865
Zhu, Y. and Newsam, S., 2018. Densenet for dense flow. In: Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), September 2017, pp. 790–794. 10.1109/ICIP.2017.8296389
Article Sidebar
Published
Apr 29, 2022
Article Details
Section
Research Article
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
