A Study on Weeds Retrieval based on Deep Neural Network Classification Model

2.1 DNN

Simonyan and Zisserman [7] proposed a depth DNN architecture using stack of small size 3×3 convolutional filters, which now we called as VGG net. They showed that using stacks of small size of receptive fields 3×3 had similar effects to a larger 5×5 or 7×7 receptive field. By their model configuration, earlier layers studies low-level features such as micro features, small textures on the object. In comparison, later layers studies high-level features such as edges, shapes, overall characteristics of the object. The last layer of stacking convolutional layers has 512 channels, the authors then added FC layers and softmax layer for training in the ILSVRC dataset. VGG net achieved the state-of-the-art on the ILSVRC competition in 2014.

2.2 Image retrieval

Recently, many scientific papers have shown the correlation between image classification and image retrieval. Xie et al. [8] proposed Online nearest neighbor estimation algorithm that could be used for classification and retrieval tasks. Tolias et al. [9] used a convolutional layer in a DNN model to encoded many regions on the image into short vectors used for retrieval as well as localization. Babeko and Lempitsky [10] investigated how to combine multiple local feature descriptors to create global descriptors efficiently in the image retrieval problem.

Our work was inspired by Babenko et al. [6], where the authors showed positive image retrieval results when using trained CNN on the classification task. They extracted feature vectors from the last max pooling and 2 FC layers as image descriptors. The retrieval results using this approach were slightly lower or even higher than previous descriptors approach, depending on datasets. They found that earlier descriptors were suitable for low-level retrieved texture, while later descriptor was sensitive to high-level textures. In their experiment, the original descriptors had a length of 4096, which made it time-consuming to retrieve. They solved this problem by applying dimensionality reduction techniques such as PCA and discriminative reduction (SIFT + nearest neighbor matching with the second-best neighbor test + RANSAC validation). The result showed a similar retrieval result after reducing to 256 dimensions. However, this discriminative reduction was not suitable in the weeds image because local features on the surface consist of many low-level features (shown in Fig. 5), which can make SIFT fail to capture useful descriptors. Instead, we use a nonlinear reduction technique such as autoencoder since this network can obtain diverse information available from descriptors.

3.1 DNN and image retrieval

Fig. 1 shows overall DNN architecture. For the classification task, training images are feed to DNN, where the last layer is GAP. Then, we add an FC layer with a length of 256 (denoted as FC256), followed by a batch normalization (BN) layer, and activation function is the rectified linear unit (RELU). Finally, we fed the output of RELU to an FC layer with length equal to the number of classes in the dataset (denoted as FCclass), followed by a softmax layer to determine the probability that input image belongs to species. After training the classifier, GAP, FC256, and FCclass are used as the descriptor of the input image for retrieval task.

Fig. 1.

DNN architecture used for classification tasks.

Fig. 2 shows the overall scheme for our study on weeds retrieval based on the DNN classification model. Assume that we select “descriptor 2” as the descriptor of images, given a query image and a gallery set, we collect descriptors by feeding those images to the DNN model. Then, we measure distances between the query image and all images in the gallery set, using these descriptors. Finally, k images in the gallery corresponding to k smallest distances, are returned as retrieval results of the query image.

Fig. 2.

Overall scheme for retrieval task. First, a query image and images in the gallery set are passed through the DNN model. Then, DNN model returns descriptors. Next, we can apply dimensionality reduction on descriptors (optional). Later, L2 normalization is use as a preprocessing step. We use Euclidean distance to measure the distance between the query descriptor and descriptors in the gallery set. Finally, we return k images in the gallery set corresponding to k smallest distances

3.2 Retrieval similarity metric

Given two descriptor vectors x = (x¹,x²,…xⁿ) and y = (y¹,y²,…yⁿ) have n dimensions in the Cartesian coordinate, the similarity between these two vectors is measured by Euclidean distance, d(x,y), in equation (1). The lower of d, the higher similarity of x and y.

L2 normalization is used as a preprocessing step before measuring similarity to deal with the high dispersity of descriptors set. This normalization is also used in cosine similarity metric to scale the length of a vector. After normalization, Euclidean norm of all descriptors is 1, where Euclidean norm is defined in (3).

3.3 Dimensionality reduction

3.3.2 Autoencoder

Autoencoder is a nonlinear approach for dimensionality reduction. Mathematically, it learns a function h_w,b(x)≈x, where x is the input, and W,b is a set of parameters. An autoencoder contains two modules: Encoder and decoder. Encoder learns a mapping $$ x\in {\mathbb{R}}^k$$ to lower dimension $$ z\in {\mathbb{R}}^k$$ with k < n. The decoder learns how to map back to x from the latent z. Both encoder and decoder can be parameterized by a neural network with a set of parameters. Each layer in the network has an activation function so that autoencoder can become a nonlinear mapping to reduce dimension. In this paper, we choose the parametric rectified linear unit (PReLU) [11] as the activation function. This activation function gives each node a specific trainable parameter to maintain essential values for the next layer shown in Fig. 4.

Fig. 3 shows an overall architecture of the autoencoder used for reducing the dimension of an input feature vector. This autoencoder contains two sub-neural networks: Encoder receive n-dimension input vector, then operate with n × k learnable weights W_encoder and k biases b_encoder for mapping input to k dimensions.

Fig. 3.

Architecture of autoencoder used for dimensionality reduction

A PReLU (as shown in Fig. 4) with k learnable parameters are used as the activation function for nonlinearly mapping. z is the output of PReLU (also the output of the encoder module). In total, the encoder module has n × k + k+ k = k(n + 2) learnable parameters W_encoder and b_encoder, then pass to the activation function PReLU. The output of PReLU is h(x) has the same dimension with x.

Fig. 4.

PReLU, where a is a learnable parameter (image from [11]).

Autoencoder uses regression optimization techniques such as mean square error (MSE) to calculate loss between h(x) and x, and backpropagation methods like adadelta [12] to find W and b that optimize the loss function, where W = {W_encoder, W_decoder} + {b_encoder, b_decoder} so that h(x) ≈ x.

4.1 Dataset

We used Chonnam National University (CNU) weeds dataset for retrieval experiments. This dataset has 21 species belong to five families, 208477 images in total [13]. As shown in Table 1, this dataset is imbalance, where the species has the largest number of images is Galinsoga quadriradiata Ruiz & Pav., around 24300 images, while species Bidens bipinnata L. has the smallest number of images, about 800 images. Fig. 5 shows images example in this dataset.

Table 1.

Contribution of species on the CNU weeds dataset

Fig. 5.

Images in the CNU weeds dataset (image from [13])

Given an image I, we retrieve k images in the training set that most “similar” with I through a metric distance function. The “similar” term refers to images that belong to a particular species.

For classification and retrieval experiment, 60% images in each species were used as the training set (also the gallery set for retrieval), 20% were the validation set to selected the optimal DNN parameters in classification task, and the remains 20% were the test set (also the query set for retrieval).

4.2 Weeds classification

We applied transfer learning on the VGG net for training weeds classification [14], in which parameters in this network were retrained on the ImageNet dataset, which already contains weeds images [15]. As shown in Fig. 1, we added the GAP layer, followed by an FC256, batch normalization layer, ReLU as the activation function, then passed to another FC layer before using softmax to get a probability vector. This FC layer had a size of 21 (equal to the number of species, we called it FC21), and the output of a softmax layer was used to calculate loss via the categorical cross-entropy loss function. The input was a 128×128 RGB color image, rescaled to [0,1] range. When applying transfer learning, we froze the first five layers and trained with a batch size of 128, and no data augmentation techniques were used. The learning rate was 0.001, and the optimization algorithm was stochastic gradient descent on 50 epochs.

Fig. 6 shows training and validation loss and accuracy on 50 epochs. VGG started to converge from 10th epoch, and validation accuracy varied around 0.975.

Fig. 6.

Training and validation performance on 50 epochs

After 50 epochs, Table 2 shows the accuracy, precision, recall, and F1 score on the test set, where the accuracy was 0.97643, and the F1 score was 0.96437.

Table 2.

Performance on the test set using VGG net

4.3 Weeds retrieval

To demonstrate the efficiency of retrieval by using features from the DNN learned by classification task, given an image I, we validated feature from GAP, FC256, and FC21 from VGG as the feature descriptor of I. Since the dispersity of descriptors may be extensive, we applied L2 normalization to scale descriptor x before retrieval. L2 normalization aims to make a k-dimension point lay on a unit ball centered at the origin, calculated by expression (2), where ∥x∥₂ is Euclidean norm. Euclidean distance was used to measure the similarity between x and each descriptor on the gallery set. k images with the shortest distance concluded as retrieval results of the image in the query set. Assume the number of images in the query set is Q, I_q and I_g is the image belongs to query and gallery set, we used mAP as the retrieve evaluation in terms of t, where t is the name of species or family of image. The mAP is calculated by expression (6):

where

Table 3 shows mAP results when we retrieved five images in the gallery set. The highest performance was GAP, followed by FC256 and FC21. Because of the natural characteristic of weeds, low-level feature plays a vital role in presenting weeds feature, which made GAP had the highest performance among all. The mAP performance shows that using DNN for classification task can also perform well on retrieval.

Table 3.

mAP value with five retrieval results on the CNU weeds dataset

Fig 7. show examples of retrieval results. Retrieved images were entirely identical to the outside characteristic of the weeds species, show that a DNN model on classification task can capture essential features, which can be used as a descriptor for retrieval.

Fig. 7.

Retrieval examples. 1st row used GAP as the descriptor, 2nd row used FC256, and the last row is FC21. images on the left-most column from the query set and the next five images from left to right are retrieval results in descending order of Euclidean distance

4.4 Dimensionality reduction

In this step, we experimented on dimensionality reduction on descriptors for faster retrieval. We did not apply this technique on FC21 because this descriptor has low dimensions. Instead, we used PCA and autoencoder to reduce the dimension of GAP and FC256 descriptors to 32, 64, and 128 (and 256 for GAP).

4.4.1 PCA

Table 4 and Table 5 show the mAP of GAP and FC256 after using PCA for dimensionality reduction. When reducing GAP dimension to 128, 256, and FC256 to 64, 128, the results were even higher than using original descriptors, except 32 dimensions were much lower. Fig. 8 shows examples of retrieved images after applying dimensionality reduction to reduce the descriptor dimension to 128.

Table 4.

mAP with five retrieval results, using PCA to reduce the dimension of GAP

Table 5.

mAP with five retrieval results, using PCA to reduce the dimension of FC256

Fig. 8.

Retrieval examples when applying PCA to reduce GAP and FC128 descriptors to 128 dimensions. 1st row shows GAP case, 2nd row shows FC256 case. Images on the left-most column from the query set and the next five images from left to right are retrieval results in descending order of Euclidean distance.

4.4.2 Autoencoder

In this experiment, we used the autoencoder demonstrated in Fig. 3. Given the input descriptor, we treated the output of the first PReLU (in encoder module) as a reduction result of the input descriptor, which was GAP and FC256. Similar to PCA approach, this PReLU had a length of 32, 64, 128 (and 256 for GAP). On training, we applied cyclical learning rates (CLR) triangular policy proposed on [16]. We trained on 1000 epochs per each dimension, with adadelta as the optimization method for MSE loss function. After 1000 epochs, we chose parameters on encoder and decoder modules having minimum on loss function on the validation set. By experiment, the optimal setting of batch size, max, and min learning rates boundary for CLR correspond to specific reduction are shown in Table 6 (for GAP descriptor) and Table 7 (for FC256 descriptor).

Table 6.

Setting of batch size, min and max learning rate, and loss of validation set on each dimension, applied on GAP

Table 7.

Setting of batch size, min and max learning rate, and loss of validation set on each dimension, applied on FC256

After reducing the dimension, retrieval results in Table 8 and Table 9 show mAP performance when retrieved five images from the gallery set. Applying autoencoder to reduce feature dimension attained performance higher than the original, except for the 32 dimensions that suffered a lack of information.

Table 8.

mAP with five retrieval results, using autoencoder to reduce the dimension of GAP

Table 9.

mAP with five retrieval results, using autoencoder to reduce the dimension of FC256

On GAP and FC256 descriptor, 64 and 256 dimensions that using autoencoder had better performance than PCA, and only 32 dimensions shows the dominance of PCA than autoencoder. While reducing descriptor to 32, 64, and 128 dimensions; in most cases, using GAP as descriptor yields better performance than FC256 since GAP is an earlier layer which can capture low-level features that play an essential role in weeds classification as well as weeds retrieval.

Fig. 9.

Retrieval examples in terms of species when applying autoencoder to reduce GAP and FC128 descriptors to 128 dimensions. 1st and 2nd row used GAP, 3rd and 4th row used FC256. L2 normalization applied to descriptor on odd rows. Images on the left-most column from the query set and the next five images from left to right are retrieval results in descending order of Euclidean distance