Introduction

Being one of the most widely consumed fruits with a global annual production of 86 million tons (Vasylieva and James, 2021), apple produce is vulnerable to many diseases responsible for the degradation of its quality and economic value (Belete and Boyraz, 2017). The traditional method of fruit disease diagnosis by human experts is time-consuming, expensive, delayed, error-prone (Dubey and Jalal, 2013), and often not approachable to small-time and remote orchardists. Timely detection of diseases has shown to have a significant impact on the output of the crop (Roberts, 2006). This has led to several technological interventions for intelligent disease diagnosis from imagery data. Literature survey (Table 1) reveals promising results in the detection of plant diseases using deep learning, which has inspired us to carry out research models for apple disease diagnosis.

Machine learning has been widely used to perform complex tasks of image recognition, speech recognition, and text comprehension which are almost impossible for conventional algorithms (Das et al., 2015; Liakos et al., 2018; Roberts, 2006; Shruthi et al., 2019). These tasks have been further specialized using deep learning algorithms, which automatically extract millions of features from data not apparent to the human eye enabling them to recognize complex patterns in data (Kamilaris and Prenafeta-Boldu, 2018; LeCun et al., 2015). The inherent potential of deep learning to understand complex patterns in the data has enabled its widespread usage in the recognition, indexing, and segmentation of imagery data. This makes it very feasible to do tasks like disease diagnosis from images both in humans and plants (Amara et al., 2017). Deep learning has been successfully applied in various fields of biotechnology including agriculture (Bayer and Edwards, 2021), horticulture (Yang and Xu, 2021) and food technology (Mishra et al., 2022), etc. Artificial neural networks are replacing classical mathematical and statistical models in biotechnology (Kujawa and Niedbała, 2021) due to their versatility and ability to represent complex functions. Deep learning has been extensively used in disease diagnostics and yield prediction (Zhang et al., 2022) from imaging modalities in humans as well as plants (Bakator and Radosav, 2018; Ferentinos, 2018; Kamilaris and Prenafeta-Boldu, 2018). Taking inspiration from it, we applied deep learning to detect and classify apple diseases like Scab, Rust, Black Rot, and Alternaria, which afflict the apple tree.

We propose a deep learning model AppleNet to instantly detect diseases in apple trees using visual imagery without the physical intervention of human experts. The apple disease detection from images using AppleNet is illustrated in Figure 1. To train and test our model, we collected a dataset of 2897 images and targeted four apple diseases, viz., Black Rot, Scab, Alternaria, and Rust. These are some of the prevalent diseases responsible for Apple yield reduction (Belete and Boyraz, 2017). The visual symptoms of these diseases inflicting apple trees are identified by our model to predict the relevant class of disease. To improve the performance of AppleNet and develop it in an efficient way, we took advantage of transfer learning. Transfer learning uses the knowledge from one context and in another context (Kaya et al., 2019). To implement transfer learning, our model is built on the top of RestNet50 (Residual Neural Network Architecture) pretrained model. RestNet50 is trained to extract the features in images. These features are then used by our designed classifiers to predict the appropriate diseases. The results of experimentation on a collected dataset with data augmentation demonstrated that AppleNet can be efficiently used for apple disease detection with a classification accuracy of 96.00%. To examine the effectiveness of our proposed approach, we compare our model with other pretrained models and with a model created from scratch. Results of the experiment demonstrate that transfer learning improves the performance of deep learning models and using pretrained models based on residual neural network architectures gives remarkable results as compared to other pretrained models.

Figure 1. Classification of apple disease using deep learning.

A comparative analysis of various machine learning and deep learning techniques for plant leaf disease detection was carried out by Sujatha et al. (2021). They analyzed Support Vector Machine (SVM), Random Forest (RF), and Stochastic Gradient descent (SGD) among the machine learning models, and Inception-v3, VGG-16, and VGG-19, among the deep models. Their study revealed that deep learning models had a better performance rate than machine learning algorithms with VGG16 performing the best. Ouhami et al. (2020) have used transfer learning in their study for tomato disease detection. In their study, they used various flavors of pretrained convolutional neural networks with DenseNet161 (Huang et al., 2017) outperforming other flavors. INC-VGGN is a deep learning architecture for plant disease diagnosis proposed by Chen et al. (2020). It is the modified version of the pretrained VGG model trained and tested on images of rice and maize in order to find the appropriate class of diseases in rice and maize. Their proposed model performed with an accuracy of 91.83. Based on the pretrained model DenseNet-121, Zhong and Zhao (2020) proposed three methods for apple disease detection, viz., regression, multi-label classification, and focus loss function. For experimentation, they used a dataset of 2462 images, which include six apple diseases, namely, Healthy Apple, General Apple Scab, Serious Apple Scab, Apple Gray Spot, General Cedar Apple, Rust and Serious Cedar Apple Rust. The paper reported accuracies of 93.51% for regression, 93.31% for multi-label classification, and 93.31% for focus loss function. A low-cost method for apple disease classification is proposed by Bi et al. (2020). In their study, they used MobileNet-based transfer learning architecture and conducted an experiment on a dataset of 2004 images, which comprise two diseases, viz., Alternaria Leaf Blotch, and Rust. On the dataset having two classes of diseases, their proposed approach got an accuracy of 73.50%. The study proposed by Alsayed et al. (2021) compared pretrained models VGG16, InceptionV3, MobileNetV2, and ResNetV2 for classification of images of apple diseases. They conducted the experimentation of four class datasets containing Scab, Rust, Multiple, and Healthy images. Their study showed that the Adam optimizer is effective in transfer learning and got a classification accuracy of 94% (Ozden, 2021) combined PlantVillage dataset and Plant Pathology dataset to train and test their transfer learning–based model for apple disease detection. In their study, they fine-tuned their model with various optimizers and concluded that Adagrad optimizer performs well for the apple disease classification task with a classification accuracy of 91%.

Paper contribution

1. We specifically designed a deep learning–based multi-class classification model AppleNet to include extra apple plant diseases, which has not been the case with other previously designed models.

2. The hyper-parameters were uniquely fine-tuned to maximize the model efficiency.

3. We created our own dataset from the images taken directly from the trees, which, unlike the publicly available datasets created in controlled settings with smooth (white) background, has been created in a real-world environment and includes background noise as well. This helped us train our model in a more realistic way.

The past work supports the idea of using deep learning reinforced with transfer learning to develop a disease detection algorithm for diseases in apple trees. This inspired us to use deep learning approaches with improved architectures for better performance in disease diagnosis of apple plants. The remaining part of the paper is structured as follows. Section 2 includes data collection and pre-processing. Section 3 gives a detailed description of the proposed methodology, model architecture, and hyper-parameter setup. Experimental results and discussions are presented in Section 4. The conclusions are included in Section 5.

Dataset

To train AppleNet, we collected data directly from orchards by conducting multiple field surveys and reinforced the dataset with publicly available data from the PlantVillage dataset. PlantVillage (Ali, 2019) contains around 50 k images of infected and healthy plant leaves. This dataset is categorized into 38 classes of plant diseases for 14 plant species. All the images in the plant village are annotated by experts at the Plant Village platform (PlantVillage, 2022). Plant village dataset contains four categories related to apple, viz., Black Rot, Scab, Rust, and Healthy. The images were captured on the same background and under the same illumination conditions, which are very different from images captured in a real environment having diverse illumination effects and complex backgrounds. To get more variety in data, we also collected images directly from orchards under realistic environmental conditions and combined them with the plant village dataset for more generalization of the model. Canon EOS700 D with autofocus technology was used to collect images of apple leaves, which were collected at various locations for a more comprehensive study. These images were annotated by horticulture experts. Another class of disease in captured data is Alternaria.

Description of data collected

A total of 2897 images were collected, corresponding to five classes: Black Rot, Scab, Rust, Alternaria, and Healthy. A brief description of the dataset is given in Table 2.

Data preprocessing

Since Convolutional Neural Networks take input data with constant dimensionality and are sensitive to noise in the data, collected data were preprocessed, via the removal of unwanted backgrounds and bad-quality images. The images were scaled to a fixed resolution of 224 × 224 pixels to maintain consistency during training. Data augmentation is used to artificially expand the size of the training dataset by creating modified versions of images in the dataset in order to improve the accuracy of the model. Standard transformations like shifts, flips, zooms. etc., were used. Finally, the dataset was split into training, testing, and validation sets in the ratio of 70:15:15.

Methodology

As shown in Figure 2, the classification of images is done in two phases: feature extraction and classification. The feature extraction part constitutes the input layer, convolution layers with ReLu activation function, and pooling layers. The input layer takes in three channel 224 × 224 pixel images, which are then passed on to the convolutional layer. The convolutional layer performs filtering to create a feature map. At a particular layer, convolution operation is formally defined as equation 1:

where Y_i is the output feature-map, X_j is the input from previous layer, * is the convolution operation, F_ij is kernal(s), and h(.) is the activation function.

Figure 2. Methodology of the proposed work.

Rectified Linear Unit (ReLu) activation function h(.) is used at each node of the convolutional layer to handle nonlinearity of features and is given in equation 2.

After convolution operations, the pooling layer (also called the down-sampling layer) is used to reduce dimensionality to prevent over-fitting (Toda and Okura, 2019). The pooling is performed by computing the maximum value in the neighborhood called max-pooling to down-sample the feature-map and is given in equation 3.

The extracted features are then flattened into a one-dimensional array, which is used as input to our classifier. We used a feed forward neural network given by equation 4 having a softmax activation function at the output layer.

where Y is the vector of all outputs, W is the matrix of weights, B is the vector of biases, and X is the vector of inputs. The softmax function in equation 5 gives us a probability distribution of classified diseases and the node with the highest probability is a predicted class of disease.

where y_i is the value for a particular class at the output layer and k is the total number of classes.

In order to check how well our model performs, we used categorical cross-entropy as loss function, which measures the difference between two probability distributions in order to compute the cost of our model as is given in equation 6.

E(P,L) is the error, the distance between prediction vector P and label vector L. The error E is then back propagated to adjust the weights of the model.

Transfer learning can reuse existing models to new problems by transferring knowledge learned in one context into anther context, therefore, saving precious resources and time. Transfer learning can improve the performance of the model in those situations where data are not sufficient to train the model (Torrey and Shavlik, 2010). Instead of rebuilding AppleNet from scratch, we took advantage of transfer learning and developed our model on top of a pretrained model.

AppleNet architecture

We used ResNet50 (He et al., 2016) as a feature extractor in our model and build a classifier on top of it. The classifier was built by adding a global average pooling layer, two dense layers with ReLu activations, and one output layer with softmax activation. The complete architecture of the AppleNet is given in the Figure 3.

Figure 3. AppleNet model architecture.

Residual Neural Network (ResNet) architecture was developed to have a depth of up to 1000 layers. In deep learning models like CNN, more layers will supposedly extract more features but increasing the number of layers poses the problem of vanishing or exploding gradient, i.e., the loss function’s gradient tends to approach zero making computation hard. The ResNet architectures resolve these problems by using the skip connections technique. Skipping jumps over some layers in the neural network and makes training stable by eliminating the effect of vanishing/exploding gradient descent problem.

The RestNet50 model is trained on a huge ImageNet dataset, which contains more than 1 million images in 1000 different categories. Since AppleNet is built on RestNet50, which has a large number of layers and is trained on a very large dataset, it is capable of extracting intricate details of apple diseases from images.

Training and model hyperparameters

In the developed model, various parameters were set to get the desired output. An initial learning rate of 0.001 is used, which is decayed by a factor of 20 each time the validation loss plateaus after an epoch. RMSprop optimizer is used with categorical cross entropy as a loss function with a batch size of 36 over 100 epochs.

Results and Discussion

We tested our model on data, which have never been used during training to get unbiased predictions. The size of the test set was 434 images, which are 15% of the entire dataset and all the samples were taken randomly. To measure the performance of our model, we used four evaluation metrics; classification accuracy, precision, recall, and F1-Score. These metrics of the model are computed on the basis of possible outcomes of a model, viz., True Positive (TP), False Positive (FP), True Negative (TN), and False Negative (FN). The visual summarization of the predictions of our model is given in a confusion matrix 4.1 (Figure 4).

Figure 4. Confusion matrix of Applenet.

The confusion matrix allows us a more detailed analysis of predictions over the entire dataset as well as over individual classes. We can compute classification accuracy given by the ratio of correct predictions to the total number of predictions. Our models give a classification accuracy of 96.99%. Formally, classification accuracy is computed as equation 7.

From classification accuracy, we can also compute classification error, which is the ratio of incorrect predictions to the total number of predictions. In the case of our model AppleNet, the classification error is 3.01%.

To evaluate our model on individual classes of disease precision and recall was computed as given in Equations 8 & 9 and is given in Table 3.

Precision is the ability of the model to find the relevant predictions and recall is the ability of the model to find all predictions in a relevant class. Precision and recall for individual classes are given in Table 3. From Table 3, it can be seen that Block Rot disease has both precision and recall of 100% because all the images of Black Rot disease in our dataset have the same background, and constant illumination conditions, and every image consists of a single disease, which can be visually distinguished from other classes. Some samples of Black Rot and other diseases are shown in Figure 5. In the rest of the classes, we added images taken directly from orchards under realistic environmental conditions for more generalization of the model; therefore, we can see a slight decrease in precision and recall for these classes. We also measured the model performance in terms of F-score since the distribution of samples in our dataset is slightly unbalanced. F-score is the harmonic mean of precision and recall given in equation 10 for our model, and it was computed to be 97.00%.

Figure 5. Sample images from dataset.

Comparison of models

In order to examine the effectiveness of AppleNet by using transfer learning, we compare its performance with four other models trained on the same dataset. For comparison we also built a baseline model from scratch (without transfer learning). The results are compared with other pretrained models including the model built from scratch. The results are summarized in Table 4 and Figure 6. We find that the model based on ResNet performs significantly better with an accuracy of 96.99% and F1 score of 97%. From the comparisons specified in the table, we can clearly observe that transfer learning improved the performance of our model as the F-Score and classification accuracy of transfer learning models are higher than the base-line model created from scratch. We can also see that the model based on Residual Neural Network architecture gives remarkable results as compared to other pretrained models. The mean difference in classification accuracy between AppleNet and other experimental models was 21.54%.

Figure 6. Comparison of predictions.

The performance of all the models during training and testing is collectively summarized in Figure 7. From this figure, we can clearly observe that the confusion matrix of AppleNet based on ResNet 50 architecture has less number of misclassifications as we have less number of FP and FN. From confusion matrices of models other than AppleNet, we found that models are most confusing in case of Rust disease because it is complex to distinguish from other classes visually and needs more number of filters to accurately identify it, which is resolved in ResNet architecture (AppleNet). In Figure 7, we can also observe that AppleNet learns quicker and gets an accuracy of more than 90% in less than 20 epochs. In other experimental models, we can see that there is more fluctuation in training and testing accuracies depicting more misclassifications both in training and testing as compared to AppleNet.

Figure 7. Comparison of models.

Compared to state-of-the-art studies in apple disease detection, our model covered more diseases while maintaining better accuracy. This is due to unique hyper-parameter tuning and adaption of ResNet architecture in which skip connections are used to avoid gradient vanishing problems thus enabling us to add more layers for better representation of abstract features in our data. The comparison between our proposed model and state-of-the-art results is summarized in Table 5. In comparison to the disc.

Conclusion

In this study, we proposed a deep learning model with the transfer learning approach for apple disease detection and classification. To train and evaluate the model’s performance in more realistic settings, we collected a dataset of 2897 images of infected apple leaves. The collected dataset was captured directly from trees and included noise in the shape of background data. The results of our model performance for disease classification specified in terms of classification accuracy and F1 score were 96.99% and 97.00%, respectively. The results suggest that our proposed model can be applied effectively in the field of horticulture as it can eliminate the downsides of using human experts for apple disease classification.

To improve efficiency, Generative Adversarial Networks can be used for image augmentation to increase the size of the training set. Attention Networks have been used to focus the model on a particular portion of the data. This can be beneficial in images with enough background noise to enhance the effect of the diseased portion of the image during training. Explainability is a new hot topic in the field of deep learning. Explainable models are being used to explain the inner working of the deep model. Visualization frameworks like Grad-Cam can be used to make the model explainable. After diagnosing the disease in apple trees, there is a need for sprays that can protect apple trees from pests and diseases. But it is important to know the appropriate spray for the detected disease under particular conditions. Besides disease detection, we can also develop a spray recommender system based on disease detected, location, date–time, and weather conditions. The recommender can be used without the physical intervention of horticultural experts, which will make it available remotely to farmers at the proper time.