Introduction

Hyperspectral imaging (HSI) is a non-destructive and non-polluting imaging technique that combines spectroscopic technique and imaging technique to be collectively called ‘Imaging Spectroscopy’ (Jia et al., 2020). In a traditional colour image, each pixel is categorised into three colour channels (Red, Green and Blue). However, each pixel in HSI is categorised by many continuous bands; the number of bands depends on the spectral resolution of the hyperspectral camera. Traditional RGB cameras mimic the recognition capability of the human eye on the basis of the shape and colour of the imaged object. Sun is the ultimate source of all electromagnetic radiation reaching the earth. Out of the total radiation illuminating the scene, human eyes and traditional cameras are sensitive to the visible (VIS) bands of the electromagnetic spectrum. To visualise scenes outside the VIS band, the technique of spectroscopy has been used and has proved useful in eliminating the limitation of the human eye and traditional photography. Imaging spectroscopy captures intrinsic information in the form of images by revealing information about the target in the whole electromagnetic spectrum. The advantage of HSI lies in reliable and precise identification, classification, detection, characterisation, differentiation, and quantification. HSI spans every field of study ranging from biomedical imaging, molecular biology, astronomy, mineralogy, geology, cultural heritage, physics and surveillance to food processing and agriculture (Niedermaier et al., 2019).

HSI is a growing research field in the area of food engineering and has become a valuable tool for food quality analysis and control. Nowadays, HSI is regarded as a pioneer tool for quality control in agri-food products. The combination of spectroscopic and deep learning technologies is the strong driving force behind the development of HSI systems in the evaluation of food quality and to find out the hidden information non-destructively. In addition, direct identification of different components and their spatial distribution in food systems can be carried out (Liu et al., 2014). The objective of this review is to conduct a comprehensive exploration and extension of HSI applications in food analysis, emphasising the integration of advanced deep learning techniques. It seeks to address current challenges in the application of HSI and deep learning in the food domain, identify opportunities for overcoming limitations, and propose avenues for future interdisciplinary research. This paper is organised into the following sections. Section 2 deals with materials and methods that have been consulted for the comprehensive review. Section 3 deals with the fundamentals of HSI. Section 4 provides an overarching introduction about the state of art in deep learning with an extension of HSI application of each technique for food analysis. Section 5 provides extensive application of HSI for food analysis creating an intuition for future collaborations using HSI and deep learning for enhanced accuracy and precision. Section 6 lays out a practical framework for undertaking the task of applying deep learning using hyperspectral images for food analysis. Section 7 discusses the conclusion of the study.

Novelty

This review paper represents a pioneering endeavour in the domain of food quality and safety assessment. While HSI and deep learning have individually garnered attention for their applications in various fields, their convergence in the context of food evaluation is a distinctive feature of this review. A comprehensive analysis of the cutting-edge synergy between deep learning algorithms and HSI techniques is presented. The focus on diverse food categories, spanning fruits, vegetables, cereals, milk and meat, showcases the breadth of this review. A step-by-step guide to transforming theory into actionable practice and facilitating the adoption of deep learning-based HSI for food quality assessment is discussed. In essence, the novelty of this work lies in its role as a trailblazer, bridging the gap between two transformative technologies and offering a transformative approach to ensure the safety and quality of our food supply.

Research gap

One notable aspect of our review is the identification of specific research gaps within the field of deep learning HSI for food quality and safety assessment. While our analysis provides valuable insights into the current state of research, it also illuminates areas where further investigation is warranted. One such research gap is the need for more extensive exploration of HIS’s applications beyond food quality and safety. While our review primarily focuses on this aspect, there are numerous uncharted territories where HSI could offer innovative solutions. These include real-time monitoring of food processing, early detection of emerging contaminants and the assessment of the long-term effects of storage conditions on food quality. Additionally, the potential for cross-disciplinary collaboration remains underexplored. The integration of HSI with emerging technologies like IoT (Internet of Things) and blockchain could revolutionise food supply chain management.

Materials and Methods

The methodology adopted for this review is anchored in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework, as per Page et al., 2021. Our systematic literature search, conducted up to 27 November 2023, aimed to rigorously select articles for review. We utilised keywords such as ‘HIS’, ‘deep learning’, ‘machine learning’, ‘food technology’, ‘image processing’ and ‘computer vision’ in our search strategy. Primary databases included Web of Science, Scopus and Google Scholar, ensuring a comprehensive and accurate yield of relevant studies. All sourced articles underwent an initial screening, focusing on relevance to HSI, deep learning and food technology. Of the 854 research articles initially identified (depicted in Figure 1), we applied Boolean AND filters to refine the selection. Consequently, 305 articles, spanning from 2016 to 2023, were earmarked for in-depth analysis. To enhance the meta-analysis aspect, each selected article was subjected to a systematic evaluation based on specific criteria: relevance to the subject, methodological soundness and contribution to the field. The statistical analysis involved synthesising data points such as study outcomes, methodologies and results to quantitatively assess trends and common findings among the studies. This meta-analysis not only compared the results of the authors but also involved a statistical synthesis of the data, seeking patterns, discrepancies and insights across the literature. The final phase of our methodology involved reaching a consensus on the findings, which was achieved through a rigorous, iterative discussion among the research team.

Figure 1. Trend of research papers published in the area of Hyperspectral imaging.

Hyperspectral Imaging System

HSI is a passive imaging technique and requires the scene to be illuminated by either the sun or some other source of light. Once the scene is properly illuminated, it has to be captured by an optical unit, which involves lenses or a combination of different lenses to allow for the transport of light and its convergence. The light energy carries information with it and needs to be captured by a sensory unit. Earlier, a film was used to capture light, but this approach has been replaced by digital light sensors that capture/store light energy. The sensory unit deals with extracting the different colour channels from the VIS white light. Raw data transforms into useful information after going through a processing unit which imparts additional information to the raw data (images) to be revealed on to the display unit which can take the form of a computer screen or can be printed (print medium). Figure 2 represents various components of image generation.

Figure 2. Schematic diagram of hyperspectral imaging system.

The point of distinction between hyperspectral images and traditional images lies in the spectroscopic unit, which is absent in traditional image processing. The spectroscopic unit consists of a spectrograph that measures energy produced by matter on interaction with electromagnetic radiation in different wavelength bands. A colour image can be thought of as a cube of pixels with red, green and blue pixel planes stacked together. A hyperspectral data cube is also a 3D cube but not limited to three channels; the number of channels are in hundreds. Any picture element (pixel) of the hyperspectral data cube is represented by three units (x, y, z) called a voxel. (x, y) determines the spatial location of a voxel and z determines the band/channel location. A spatial location, for example, [7, 8], would refer to the seventh row and eighth column of the 2D data matrix (Figure 3). A series of values such as [7, 8, 0], [7, 8, 1] and [7, 8, 2] … [7, 8, 99] would determine values of pixel location [7, 8] at bands (0, 1, 2 … 99) of a hyperspectral data cube with 100 bands. Such consecutive values form the spectrum or spectral signature from a series of voxels (ElMasry and Nakauchi, 2016). There are three conventional specifications of the imaging systems to generate a hyperspectral data cube. First specification includes whisk broom imaging which captures a single pixel in all specified spectral channels at an instance (Figure 4A). The second specification is push broom imaging which scans an entire line in all specified channels at an instance (Figure 4B). Area scanners which capture one entire spectral wave band (scene) at a time are included under third specification (Figure 4C). There is a surge of hyperspectral cameras in the market. However, only a specific type of industrial/laboratory-based hyperspectral camera can be used for a particular application, depending on the material to be analysed. As such, the various types of cameras along with their operational range in nanometre are RedEye 1.7 (950–1700) nm, RedEye 2.2 (1200–2200) nm, BlackEye (2900–4200) nm, BlueEye (220–380) nm, GreenEye (400–1000) nm and OrangeEye (580–1000) nm (Niedermaier et al., 2019).

Figure 3. A hyperspectral data cube.

Figure 4. Acquisition modes of hyperspectral images.

Hyperspectral image analysis operations

In order to maximise the retrieval of information from a high-dimensional hyperspectral data cube, there are certain procedures that should be undertaken to generate images of high spatial and spectral quality. The theoretical background of image analysis phases is beyond the scope of this article; thus only the listing of the procedures carried out at each phase is tabulated in Table 1.

Deep Learning Models for Hyperspectral Imaging

Deep learning belongs to the category of neural networks, a biologically inspired programming paradigm which imparts learning capability to computers that is naturally present in human beings. Deep learning techniques have multiple layers of neural networks to learn hidden features from raw data without human intervention. In training, large deep neural networks performance continues to increase with more and more data. It can be said that deep learning models are data hungry. Subsequently, the hyperspectral data cube has high dimensionality and the interaction of incident radiation with chemical molecules imparts chemical characteristics to the absorption bands of food samples. Deep learning has shown great potential in extracting this hidden information that is impossible to obtain by traditional imaging. The necessity of using spectral data instead of traditional imaging arises from the rich chemical information embedded within the molecular structure of the food (Bureau et al., 2019). To establish a relationship between spectral information and the chemical concentration of food, a variety of linear and non-linear chemical chemometric methods such as Partial Least Squares (PLS), Artificial Neural Networks (ANN) and Support Vector Machines (SVM) have been proposed. These models are robust and accurate but with low sensitivity. Spectral data presents a challenge of strong correlations between neighbouring wavelengths and overtones and among absorption peaks and noise. Linear models fail to extract complex non-linear features from spectral data. To overcome this limitation, non-linear machine learning models such as SVM, ANN and random forest have been utilised. However, machine learning models are prone to the risk of over-fitting. Another challenge working with spectral data is its complex constitution. A minute change in the geographical origin, harvesting time and variety of the food affect the physical properties of shape, size and surface texture, leading to a change in vibration absorption. As for chemical properties, water content variation affects spectral data. The extent of hydrogen bonding also leads to changes in spectra and prediction errors. Ageing spectrometers cause response shifts, hence a distinct spectral shape. All these challenges call for the use of deep learning techniques that are robust to over-fitting and can extract hidden and sophisticated representations (linear and non-linear) from raw data without the need to perform feature engineering (Zhang et al., 2021). The trend to use deep learning-based spectral analysis in the field of food science is increasing each day. This paper has attempted to describe some of the basic architectures of deep learning models that have been utilised in the field of HSI of food and agro products to enable the reader to have a basic knowledge of these models.

Autoencoder

An autoencoder is a classical neural network with a VIS layer of the inputs, one hidden layer of k units, which compresses the input data to representative features, and one reconstruction layer of d units (Figure 5). The process of training with an autoencoder has the encoding part followed by the decoding part. The objective of an autoencoder is to make the output as similar as possible to the input. Autoencoders can be stacked together by attaching the output of one layer to the input of another layer to be called stacked autoencoders (SAEs). SAE can be employed for spectral classification of HSI where each pixel vector can be considered as an input. Autoencoders are used for feature extraction to learn the internal pattern of non-labelled data (Windrim et al., 2019).

Figure 5. Diagrammatic representation of an autoencoders.

For the first time, autoencoders were put to use by Chen et al. (2014) for extracting features. As the spectral signature (vector) of each pixel is fed to the encoder input, the decoder reconstructs it, imparting an ability to the encoder to extract spectral features. As HSI image is characterised by spectral features as well as spatial features, to extract spatial features principal component analysis (PCA) is used which allows for reducing the dimensionality of the data (hyperspectral image). A vector is derived by flattening the image patch. Another autoencoder is used to memorise the spatial features. The joint spectral and spatial information obtained is used and classified. The deep stacked sparse autoencoder has been used by Abdi et al. (2017) for spectral-spatial feature learning. A stacked denoise autoencoder has been used for feature extraction and classification of HSI by Xing et al. (2016). The methods adopted by Abdi et al. (2017), Xing et al. (2016) and Chen et al. (2014) take advantage of training the encoder fully in an unsupervised paradigm followed by a supervised paradigm for fine-tuning the classifier. A variation in the encoder type or pre-processing method is done for HSI classification in the scenario of the sample size being small. A case of such variation was employed by Xing et al. (2016), where the authors have stacked together multiple denoising units of encoders for performing feature extraction, and the result of such variation is robustness towards feature extraction in a noisy scenario. In spectral classification scenarios, two typical cases arise; one is that two similar ground objects reveal distinct spectral signatures and the other is that two distinct ground objects reveal the same spectral signature leading to poor classification performance. The spatial classification acts as a remedial major for overcoming spectral classification problems. To go for joint spectral-spatial classification, a technique that combines autoencoders with convolution, neural networks have been developed by Yue et al. (2016) and Hao et al. (2017). Spectral features are extracted by autoencoders and spatial features are extracted by CNN; the product is fused leading to final spatial-spectral features. To examine the joint spectral-spatial features efficiently, Li et al. (2015) have used the 3D Gabor operator in the pre-processing phase that allows for joint spatial-spectral feature extraction; this fused product is given to the autoencoder to mine better abstract features. Another variation to autoencoder was done by Mei et al. (2019) using a 3D convolution operator for autoencoder construction, where the model extracts spatial-spectral features directly. Spectral feature extraction from an HSI data cube using a SAE has been performed by Yu et al. (2018) in determining the freshness grades (fresh and stale) of shrimp. In another study, Yu et al. (2019) used SAEs to extract 20 deep hyperspectral features from near-infrared (NIR) hyperspectral images to non-destructively predict the total viable count (TVC) of peeled Pacific white shrimp.

Convolution neural network

The structure of the visual system has been the motivation behind convolution neural network (CNN). CNN uses a group of parameters called kernel function to extract a specified feature from an image. The reason behind the vast success of CNN is attributed to the three characteristic properties that make it very powerful for feature representation. In a fully connected network, the full connection between two neural layers lends itself to unfriendly high-resolution spatial images. The disadvantage of full connected networks is overcome by the local connection property of CNN by reducing the number of trainable parameters to a great extent and hence lending itself suitable for processing high-resolution images. The second characteristic property is to share the same parameters by the same kernel, which further offers a reduction in the number of parameters. As in traditional neural networks, the parameters of the output are independent of each other. CNN cuts down on parameters by using the same parameters for all other outputs, which, in turn, leads to the third characteristic property of CNN referred to as shift-invariance. CNN models capture features irrespective of the position of the features in the space. The architecture of the CNN has an alternating convolution layer and pooling layer followed by a number of fully connected layers (Figure 6). Convolution layers perform convolution operations between image patches and kernels, generating feature maps. These feature maps are reduced in size by pooling layers creating more general and abstract features. The final stage involves transforming feature maps to feature vectors (Li et al., 2019). A description of each layer is as follows:

1. Convolution Layers: The most important aspect of CNN is the layers. Convolution function involves convolving input cube with a number of learnable filters leading to a generation of multiple feature maps.

2. Pooling Layers: To eliminate the information redundancy prevalent in images, pooling layers are deployed periodically after many convolution layers in the CNN architectural layers tend to decrease the spatial size of feature maps along with the reduction in computational cost and number of parameters. The feature maps become more abstract because of suffering from shrinkage in size.

3. Fully Connected Layers: Feature maps generated from convolution and pooling layers are flattened to be fed to fully connected layers. They connect every neuron in the current layer to every other neuron in the next layer.

Figure 6. Architecture of a convolution neural network.

In applying CNN for HSI classification, 2D-CNN has displayed wide applicability in extracting spatial features (Yu et al., 2017). Due to redundancy in the hyperspectral cube, the convolution kernel size tends to enlarge. This limitation was removed by using a combination of 1D convolution and 2D convolution. Using a mixture of 1D and 2D convolution allows for spectral and spatial feature extraction. A fusion of these features forms an input for the classifier. Apart from collaboration between 1D and 2D convolution, 3D-CNNs have recently found applicability in the HSI classification with significant spatial-spectral fusion capability (Liu et al., 2018). The small sample size of hyperspectral cube leads to poor performance of 3D-CNN because of the involvement of the excessive number of parameters in the supervised paradigm. Fang et al. (2020) proposed 3D separable convolution to decrease parameters, whereas Mou et al. (2017) solved the excessive parameter problem by utilising autoencoders in the 3D convolution operation. This particular autoencoder is trained in an unsupervised way with the replacement of the classifier at the decoder module. Models that have been successful in extracting complex features from labelled samples of small size utilise convolution residuals and ResNet (Mou et al., 2017). The problem of small sample size has been handled by Yu et al. (2017) using the technique of augmentation. The authors have enlarged the dataset by rotating and flipping the images. This approach served two purposes: one, it increased the diversity within the data cube, and second, it helped the model to achieve rotational invariance. A variation in the augmentation technique was put forth by Li et al. (2018) which mines the difference between the images in pairs and hence increases the diversity of the dataset. Some other variations of CNNs include the use of dense connections for extracting sample features (Paoletti et al., 2018). As hyperspectral images are characterised by low spatial resolution, using the technique by Paoletti et al. (2018) gives rise to a mixed pixel problem. Combining data from two separate modalities such as HSI and light detection and ranging (LIDAR), a variation in the sample features is introduced which overcomes mixed pixel problems (Feng et al., 2019a). In the domain of food technology and HSI, deep learning has found applicability in detecting subtle bruises on winter jujube using a CNN. Pixel-wise spectra have been extracted from the hyperspectral image cube and after pre-processing, fed to a CNN to build pixel-wise classification model. The study has shown great potential for use in real-time application due to short prediction time (Feng et al. 2019b). Laborious and time-consuming tasks of classifying hybrid progeny of okra seeds and loofah seeds have been done by Nie et al. (2019) using a deep convolution neural network (DCNN) with an efficiency of 95%, suggesting acceleration towards the progress of related research. Deep learning along with HSI has replaced the traditional authenticity measures for the classification of meat (Al-Sarayreh et al., 2020). The authors have utilised 3D-CNN approach to classify red meat suggesting future scope of real-time meat authenticity. Visual Geometry Group16 (VGG16) CNN has been used to detect the freshness of fruits such as apples, bananas, guava and oranges, yielding an accuracy of 99% (Mehta et al., 2021). Various architectures of convolutional neural network such as ResNet18, MobileNetV2, MobileNetV3-Small and MobileNetV3-Large have been employed for freshness detection of hog plum (Arunachalaeshwaran et al., 2022). A tabulated structure showing various types of deep learning techniques and the results achieved has been given in Table 2. Apart from studies on Classification, there have been numerous studies on prediction as well using deep learning. One such study has been performed by Zhang et al. (2022)obtaining a large number of oil content reference values of maize kernels is time-consuming and expensive, and the limited data set also leads to low generalization ability of the model. Here, hyperspectral imaging technology and deep convolutional generative adversarial network (DCGAN. The authors utilised deep convolutional generative adversarial network (DCGAN) combined with partial least squares regression (PLSR) and support vector regression (SVR) to predict the oil content of single maize kernel. Due to limitations in the dataset, augmentation by DCGAN expanded the spectral data and oil content data, respectively, which proved advantageous for regression model improvisation and furnishing a large amount of data for model training. The regression results have been tabulated in Table 3. To meet the growing demand for food, there has to be an average increase in crop yield by 2.4% annually, with the current rate at 1.3%. These facts point out the urgency in crop production efficiency to meet food security concerns. In this regard, Moghimi et al. (2020) have utilised deep neural networks for selecting advanced varieties of wheat. The authors have divided the study area into plot scale and sub-plot scale. A produce of thousands of wheat plots was harvested and recorded as ground truth over two growing seasons along with aerial hyperspectral image acquisition of the fields. Deep neural networks were trained to extract features from the aerial images to estimate wheat yield. The coefficient of determination at plot scale and sub-plot scale was 0.41 and 0.79, respectively, which reveals that the study can facilitate remote visual inspection for high-throughput yield phenotyping.

Recurrent neural networks

Recurrent neural networks (RNN) can extract patterns in sequences of data that are dynamic and temporal in nature. The ability to extract patterns from sequences is provided by recurrent hidden states. One disadvantage of conventional RNN is the vanishing gradient or exploding gradient due to long-term sequential data which downgrades performance of RNN. To overcome this issue, Long Short-Term Memory (LSTM) (Graves, 2013) and Gated Recurrent Unit (GRU) (Chung et al., 2014) were introduced.

A hyperspectral data cube is a 3D dataset; each sample of the data cube serves as sequential data for the RNN. LSTM has been employed for HSI spectral classification by Mou et al. (2017). Corresponding to a sample pixel vector, each band forms a sequential input to the LSTM model. To fuse spatial information into the spectral classification, Liu et al. (2018) proposed multilayer LSTM for spatial-spectral feature extraction. Pan et al. (2020) offered single gate recurrent unit GRU for combined spectral-spatial feature extraction in the HSI classification task. A combination of 1D convolution operation and RNN have been used by Wu and Prasad (2017), primarily for extracting spectral feature vectors, the 1D convolution operation is employed and then the spectral feature sequences are fed to RNN along with the support of fully connected layers and SoftMax function to achieve HSI classification. The reason behind the collaboration between 1D convolution and RNN is that individually each of these does not cater to the extraction of spectral-spatial features in an efficient manner. Other works mentioning collaboration between convolution operation and RNN is proposed by Hao et al. (2020). The authors extracted features using U-Net and then fed the input to LSTM to explore contextual information within features. A peculiar collaboration between techniques has been shown by Zhou et al. (2017) by employing PCA to extract spatial information. The first principle component (PC₁) lends itself in a sequential form in terms of several lines to the LSTM network. LSTM has been used by Kang et al. (2021) for the identification of food-borne pathogens from chicken carcass rinse. The authors have employed three LSTM blocks and achieved an accuracy of 90.4%, 92.6% and 92.9% for different types of identification (Table 2). A tabular representation showcasing the comparison and limitations among the three techniques of deep learning have been organised in Table 4.

Application of Hyperspectral Imaging in Foods

Sensory analysis of foods

Sensory properties of food, including appearance (shape, colour and physical defects), taste and texture (hardness, chewiness and cohesiveness) constitute the first impression of a food product perceived by the consumer. The sensory analysis includes the evaluation of the signals which are received through the sense of sight, taste, smell and touch. Determination of the sensory properties of a food item is important for satisfying the needs of consumers and providing them with quality and wholesome food products. Moreover, sensory analysis is important to determine the food quality after processing such as cooking, drying and freezing. Nowadays, in the food industry, sensory evaluation of foods finds a wider application as consumer’s desire to purchase foods that can assure acceptance and satisfaction. In this regard, food manufacturers have to evaluate the overall acceptability and specify the sensory attributes that are crucial for a food item (Ozdogan et al., 2021).

Traditionally, the assessment of the sensory quality of food products is done by trained human panellists. However, sensory evaluation by a trained panel is subjective and is found unsuitable for instant application when required. Instrumental methods like texture profile analysis using a texture analyser for determining textural attributes, colourimeters for colour analysis and GC-MS for determination of flavour compounds are also available, but these techniques are also time-consuming, destructive and applicable on a small scale. Therefore, there is a need to develop rapid and reliable methods for assessing of sensory properties of food products. Various imaging and spectroscopic technologies have emerged as alternative methods to be used in the food industry for the sensory assessment of food items. In recent times, HSI technology has gained prime importance as a novel rapid and non-invasive technique for the sensory evaluation of foods (Liu et al., 2017). HSI is less time-consuming and a non-destructive technique to determine the sensory properties of a diverse range of food products. The VIS-NIR (400–1000 nm) HSI is the most used method for the evaluation of sensory attributes (Ozdogan et al., 2021). Table 5 summarises the recent studies done to evaluate the sensory properties of food products using the HSI technique. HSI principally depends on chemometric analysis for evaluating sensory quality, and the relationship between molecular bonds and wavebands is very important to understanding the chemistry behind the models (Lin and Sun, 2020).

HSI as a powerful tool to check adulteration of foods

Over the years, several analytical methods have been considered to detect adulterants in food products. Unfortunately, these procedures are disruptive, time-consuming, labour-intensive and expensive and they necessitate specific sample preparation. Alternative analytical approaches for speedy, accurate and reliable quality control systems for determining food adulterants and preventing fraudulent practices involving food adulteration are becoming increasingly important in order to assure food safety. In this context, the combined use of optical imaging and spectroscopic techniques for examining food safety and quality analysis is gaining popularity as it provides non-destructive detection, chemical information and visualisation, all at the same time. HSI is a technology that combines both spectroscopic and photography approaches into a single system, allowing it to gather both spectral and spatial information about the studied object. HSI is a promising technology that enables the rapid and accurate detection of spectral and spatial information. To derive spectral, textural and morphological information from high-dimensional HSI data, chemometric approaches are necessary (Temiz and Ulas, 2021). In this regard, the combined use of non-destructive optical imaging and spectroscopic techniques for evaluating food safety and quality analysis is gaining popularity.

Shafiee et al. (2016) investigated the use of HSI system and data mining-based classifiers to detect honey adulteration. A VIS-NIR hyperspectral camera (400–1000 nm) was used to take hyperspectral images of pure and adulterated samples. After pre-processing the images, supervised image classification was performed using five distinct data mining-based techniques: ANN, SVM, linear discriminant analysis (LDA), Fisher and Parzen classifiers. The ANN classifier showed maximum classification accuracy of 95%, according to classifier test results. Other classifiers with acceptable results included SVM with radial basis kernel function, LDA, Fisher and Parzen having classification rate of 92%, 90%, 89% and 84%, respectively. This study highlights the potential of HSI in honey authentication. Verdu et al. (2016) examined the efficacy of a SW-NIR hyperspectral image approach to detect adulteration in wheat flour and bread with low-cost grains like sorghum, oats and corn. Hyperspectral information was also used to interpret the change in physicochemical properties. The SW-NIR imaging approach was fully able to detect adulteration, and substantial correlation significances were found between wavelengths from specific spectra zones and physicochemical attributes of sample. Adulterants (soda, water, urea and detergents) have been successfully detected in milk using hyperspectral radiometry coupled with machine learning techniques, as observed by Kimbahune et al. (2016). The spectral irradiance was recorded over a range of wavelengths from VIS to NIR (350–1050 nm). For quantitative detection of adulteration of limestone powder in tapioca starch, Khamsopha et al. (2021) used near-infrared HSI at wavelengths ranging from 935–1720 nm. Chemometrics was studied and used to develop a calibration model for predicting adulterant concentrations using PLSR. With a correlation coefficient (R) of 0.996 and a root mean square error of prediction (RMSEP) of 2.47%, the model’s prediction accuracy was found excellent. With the help of the model, images of pure tapioca starch, adulterated tapioca starch and pure adulterant were then created. Depending on the amount of adulterant present, various colours were displayed. Barreto et al. (2018) predicted that the HSI approach could detect the presence of starch in fresh cheese that has been added as an adulterant. The PLSR method was used in the modelling of starch content present in cheese. For a reduced model, a level of predictability of the starch content of 83.21% was achieved. The model also showed a high degree of precision, even with values of 6.65 mg starch per gramme of cheese on a dry basis (3.19 mg g-1 on a wet basis). The wavelengths of 584 nm and 976–1000 nm were found most preferable for detecting and predicting starch in fresh cheese. It was also predicted that a sample’s light intensity for a given wavelength is inversely proportional to its starch content. Black pepper adulterated with common adulterant papaya seeds could be identified using NIR-HSI coupled with multivariate analysis (Orrillo et al., 2019). Principal component analyses (PCA) and soft independent modelling of class analogy (SIMCA)-based classification models achieved 100% accuracy for berry samples and sensitivity of more than 90% for ground samples.

Assessment of physicochemical properties

HSI can be used to determine various physicochemical properties like moisture content, pH, acidity, total soluble solids (TSS), polyphenol content and antioxidant activity of food commodities. Efficiency of HIS in determining internal quality (TSS and pH) of mulberry was explored by Huang et al. (2011). Rungpichayapiche et al. (2017) also found that HSI could be used to determine the TSS and the titratable acidity of mango. Potential of HSI for determination of pH, total acidity and soluble solid content of table grapes was explored by Baiano et al. (2017). A HSI system was used to obtain the reflectance spectra of berries and a good correlation was found between each of the properties and the spectra information. Ma et al. (2018) applied NIR-HSI to assess the soluble solids content (SSC) of apples. The relationship between SSC reference data and NIR spectral data taken from each sample was determined using PLS regression analysis. HSI can also be used to determine polyphenol oxidase (PPO) activity, which is important for controlling the quality of the finished product. For indirect measurement of PPO activity in fresh-cut apple slices, Shrestha et al. (2020) employed the VIS-NIR. The authors found that HSI could be used as an alternative for conventional chemical evaluation of PPO enzyme activity. Lu et al. (2017) proposed that HIS technology could be used to determine the quantitative amount of starch in rice. The hyperspectral pictures of 100 rice samples of 10 starch grades were collected using a HSI device within a spectral range of 871–1766 nm. Using full wavelength spectra data, the SVR model was developed to determine the starch percentage. According to the authors, HSI technique for starch detection in rice is viable, and it can measure rice starch fast, effectively and non-destructively. ElMasry Sun and Allen (2011) applied NIR-HSI to assess water holding capacity (WHC) in fresh beef. The spectral characteristics of various beef samples from various breeds and muscles were recovered from hyperspectral pictures. To get an overview of the systematic spectral variations and to link spectral data of beef samples to its WHC determined by the drip loss method, both PCA and PLSR models were created. Xuan et al. (2021) confirmed that maturity index of okra fruit can be assessed using VIS-NIR HSI, which could be important for farmers to optimise harvest dates for good taste and economic return. Effective wavelengths, texture features and their fusion were used to create a library for support vector machines (LIBSVM) model. The LIBSVM model with the fused dataset showed highest total maturity classification accuracy of okra fruit, with a cross-validation accuracy of 91.7%. The fermentation index, total polyphenol content and antioxidant activity of individual dry fermented cocoa beans can be quantitatively predicted using HSI in the spectral region of 1000–2500 nm (Caporaso et al., 2018). The concentration of soluble solids, organic acids, sugars, polyphenols and antioxidant activity of fennel heads predicted using HSI in the VIS-NIR spectral range in relation to the difference in sheath layers and harvest periods (Amodio et al., 2017). The study showed that HSI in the VIS-NIR spectral range has greater potential for predicting interior constituents than HSI in the NIR spectral range. NIR-HSI was applied to quantify 27 distinct phenolic compounds in freeze-dried grape marc components. When PLSR was applied to the spectral data, coefficient of determination values of up to 0.98 were found, even when estimating minor compounds. Because of its speed and simplicity, this technology emerges as an appealing alternative for analysing the phenolic components in grape marc (Jara-Palacios et al., 2016). HSI was used to determine the levels of monounsaturated and polyunsaturated fatty acids in processed hog products. The regression coefficient curves of PLSR models were used to find optimal wavelengths. The least squares SVM models used showed a higher coefficient of determination (greater than 0.81) in the Monte Carlo validation set than the partial least squares regression models, and the least squares SVM models developed based on selected optimal wavelengths (Ma and Sun, 2020).

Evaluation of microbial contamination and toxicants

Food quality and safety issues are getting importance in both developing and developed countries; these challenges are regularly encountered in our daily lives because there is an increasing need for safe and quality food products in today’s marketplaces. The food industry is truly focused on generating harmless products and needs a continuous commitment to the creation and implementation of protocols and systems to manage numerous parameters in food products. Currently, existing analytical techniques are extremely slow and harmful. As a result, developing non-invasive, effective and rapid testing methods for monitoring food quality and safety is critical. HSI technology is one of the most promising alternatives, as it is a non-destructive analysis technique that may readily engage in productive operations (Vejarano et al., 2017). Numerous studies have demonstrated the significant potential of HSI in detecting microbiological quality and the presence of toxicants in food applying distinctive HSI acquisition methods and wavelengths varying from the VIS spectrum to NIR. Barbin et al. (2013) used NIR-HSI to assess microbial contamination in porcine meat. Plate count analysis was utilised as a complementary technique to assess the efficacy of HSI in detecting the microbiological quality of pork. Hyperspectral pictures were used to categorise the pork meat in fresh and spoiled forms. Appropriate results for spoilage detection with accuracy greater than 95% were observed when NIR spectra was combined with LDA, and thus the adopted technique could be used to calculate the shelf life of pork. The NIR hyperspectral pictures used to predict the TVC and psychrotrophic plate count (PPC) using quantitative PLS models with coefficients of determination of 0.82 and 0.85 for TVC and PPC, respectively. As a result of the good regression models between spectral data and bacterial counts, the NIR hyperspectral system has the potential to become an alternative tool for rapid and reliable shelf life determination and microbiological assessment in the meat sector. Cheng and Sun (2015a) studied the feasibility of using VIS and NIR-HSI in the range of 400–1000 nm to determine TVCs for the detection of microbial deterioration in fish fillets. Models based on full wavelengths, such as PLSR and least square SVM, performed well, with the least square SVM model outperforming others with a higher residual predictive deviation (RPD) of 3.89, determination coefficient of 0.93 and lower RMSEP of 0.49 log10 CFU/g. Cheng and Sun (2015b) used HSI in the spectral region of 400–1000 nm to detect E. coli count in grass carp fish for evaluation and visualisation of microbial quality. To create prediction models between the spectrum data and the reference E. coli loads obtained by the standard microbiological plating method, a PLSR model was used. With a RPD of 5.47 and a determination coefficient of 0.880, the PLSR model based on complete wavelengths performed well in predicting E. coli loads. Sricharoonratana et al. (2021) worked on determination of shelf life and classify sponge cakes on microbial infections during storage using NIR-HIS in the range of 935–1720 nm. NIR-HSI has been used to estimate the average spectrum from a region of interest (ROI) in each sample’s spectral image. The model was created using PLSR to predict the storage time of the cakes. With a correlation coefficient (R) of 0.835 and a RMSEP of 1.242 days, the model proved to be reliable. The classification model for discriminating between non-expired and expired sponge cakes was developed using PLS discriminant analysis (PLS-DA). According to the findings, the accuracy of prediction was about 91.3%. HSI has also been used to detect bacterial contamination in spinach leaves (Siripatrawan et al., 2011) and lettuce (Yang et al., 2010). HSI has been found to be efficient in analysing the fungal contamination in apples (Mehl et al., 2004) and dates (Teena et al., 2014). Microbial toxins are one of the most common pollutants identified in food products. The production of these poisons is carried out by fungi (mycotoxins) and bacteria (bacterial toxins). Microbial toxins are detected using a variety of techniques, including HPLC, TLC, GC, ELISA and fluorescence-based detection methods. HSI is a new, fast, non-invasive, non-destructive, and acceptable approach for detecting toxins in foods. Wu and Zu (2019) examined the potential of VIS/NIR HSI in detecting aflatoxin B₁ (AFB₁) in pistachio kernels. PCA was used to separate control (unpolluted) and all polluted kernels. For samples that were intentionally contaminated with varying concentrations of AFB₁, LDA yielded accuracies higher than 90.0% based on spectra from 694 to 988 nm which had been pre-processed with standard normal variate (SNV) and Savitzky-Golay (SG) smoothing. The calibration and validation correlation coefficients using stepwise multiple linear regression (SMLR) models were all greater than 0.9100. Kandpal et al. (2015) reported that aflatoxin contamination on corn kernels can be detected using a short wave infrared (SWIR) HSI approach. Corn samples were inoculated with four different concentrations of aflatoxin B₁ and SWIR hyperspectral device was used to scan both infected and control (uncontaminated) samples over the spectral range of 1100–1700 nm. To categorise control and infected kernels, a PLS-DA model was developed, and the maximum overall classification accuracy generated by the developed model was 96.9%. Due to rise in concentration of AFB₁, spectral divergence was found between the control and infected samples. Besides, the contamination map created with the PLS-DA model showed how contaminated samples appear visually. The authors concluded that SWIR HSI is a quick, accurate and non-destructive method for detecting hazardous metabolites in grains and that it could be used as an alternative to conventional methods. Using a NIR-HSI approach, the feasibility of detecting Aflatoxin B₁ in maize kernels inoculated with Aspergillus flavus conidia was investigated by Wang et al. (2015). The frequencies 1729 and 2344 nm have been found as key wavelengths for detection of AFB₁. Based on NIR spectral features, a full approach for detecting ergot bodies in cereals was conducted by Vermeulen et al. (2013). The results obtained utilising two NIR-HSI cameras were quite consistent and repeatable. Furthermore, the estimated values produced by NIR-HSI and those provided by the stereo-microscopic approach (reference method) had a correlation of greater than 0.94. The method’s transferability was demonstrated by the validation of the transferred process on blind samples, which revealed that it could detect and quantify ergot contamination.

Practical Framework for Applying Deep Learning to Quality Analysis Using Hyperspectral Imagery

In Section 4, we detail the framework of deep learning algorithms and highlighted studies achieving remarkable accuracies with advanced techniques. The subsequent section explores applications across various food quality domains, demonstrating deep learning’s potential in sensory analysis, adulteration detection, physicochemical property assessment, microbial contamination evaluation and toxicant identification. To aid researchers, we present a step-by-step guideline for conducting research in these phases:

Step 1: Data preparation phase

Step 2: Image acquisition phase

Step 3: Manual testing phase

Step 4: Development of deep learning models

Step 5: Comparative analysis and model deployment phase

Data preparation phase

The creation of a high-quality dataset is imperative. This involves meticulous sample selection, sample inspection and the removal of any samples exhibiting anomalies such as detrimental disorders, abnormal coloration, microbial growth or insect damage (Nogales-Bueno et al., 2021).

Image acquisition phase

The food material under examination must be captured using a hyperspectral camera with a spectral range of 900–1700 nm. During imaging, several factors require careful consideration, including source light intensity, exposure time and the distance between the food material and the lens. If the experiment necessitates a moving surface, the speed of this surface becomes a crucial factor (Nogales-Bueno et al., 2021). Once the hyperspectral camera captures the images of the food material, it generates a hypercube with dimensions x*y*z, where x denotes the number of lines in the image, y represents the number of columns in the image and z signifies the number of images captured for each sample. This hypercube subsequently undergoes pre-processing steps outlined in Table 1.

Manual testing phase

The nature of the study dictates the type of manual testing to be conducted. For research focused on estimating specific components in the food material, such as regression analysis, each imaged sample undergoes chemical analysis to determine the concentration of the target component. This process is repeated for all samples, forming a matrix where each row represents the spectral signature of a sample, with the last column indicating the component’s concentration (Mishra et al., 2022). Conversely, if the study aims to automate the classification of different food material varieties or distinguish between good and bad quality products, manual labeling of all considered samples is necessary. This results in a matrix where each row represents the spectral signature of a sample, with the final column denoting the sample’s assigned class (Nogales-Bueno et al., 2021).

Development of deep learning models

Numerous state-of-the-art models are available, and the choice depends on the problem’s nature and the coder’s expertise. For regression-based problems, a suitable approach may involve using machine learning algorithms or a combination of machine learning and deep learning techniques. Similarly, classification tasks can be tackled using either machine learning or deep learning. It is widely acknowledged that deep learning algorithms outperform machine learning when provided with extensive datasets. Advanced augmentation techniques have expanded the horizons, making dataset size less of an obstacle. Section 4 outlines some of the deep learning techniques.

Comparative analysis and model deployment phase

Refinement and validation take centre stage. The developed models undergo critical comparisons with previous studies to fine-tune their performance and achieve high-quality metric scores. Metrics for regression studies encompass the Coefficient of Prediction, Root Mean Square Error and more. For classification studies, key metrics include Total Accuracy, Precision, Recall and F1 Score. While impactful research can transform lives, transitioning from the lab to practical applications presents challenges. Direct transfer of deep learning models is rarely feasible, given the bulkiness of hyperspectral image datasets. Practical use necessitates dimensionality reduction techniques to retain essential features while reducing time complexity. This critical step facilitates knowledge transfer from the lab to practical applications.

Conclusion

In conclusion, this paper provides a comprehensive overview of the utilisation of deep learning in HSI for food quality and safety assessment. HSI, which amalgamates spectroscopy and computer vision, offers a swift and non-destructive means of evaluating the quality and safety of food products by simultaneously capturing surface and internal information.

Our review has primarily focused on recent advancements in HSI technology concerning food quality and safety. However, it’s worth noting that the potential of HSI extends far beyond this scope. To harness its full capabilities, future research endeavours should explore its application in various other facets of food quality and safety. By doing so, HSI has the potential to play a pivotal role in proactively averting food safety crises and minimising associated losses. Its versatility and non-invasive nature make it a valuable tool in ensuring the integrity and safety of our food supply chain. As we continue to delve into its possibilities, we open doors to enhanced food quality assessment and a safer, more secure food industry.

Author Contributions

Conceptualisation: B.N.D., N.G., S.Z.A.H., A.A. and H.A.M.; methodology: N.G’, K.M. and B.N.D.; writing original draft preparation: N.G. and K.M.; supervision: B.N.D., S.Z.A.S., A.A. and H.A.M.; writing, reviewing and editing: B.N.D., N.G., K.M., S.Z.A.S. and A.A.