Home

Automatic assessment of sentiment analysis can be applied in many areas[1][2]. For example, an automatic tagging system can be constructed if the evoking emotion can be detected from an image someone has posted to social media. Most of the earlier studies on sentiment analysis has made use of textual contents[3]-[5]. However, Borth et al.[2] gives an example in which the valuable sentiment is conveyed primarily by images but texts. Thus, analyzing visual contents such as images and videos is an important aspect of sentiment analysis research.

Analyzing affective images is more challenging due to the different human responses evoked by the same stimuli. Previous studies on visual sentiment analysis mostly uses deep CNNs[1][6]-[8] due to their success in classification tasks since introduced by Krizhevsky et al.[9]. Nowadays, there are various types of CNNs, such as AlexNet[7], VGGNet[10], ResNet[11], and etc.

However, these large networks have one critical problem that they require large storage capacity and computing power. Although training and deployment of the model have very different objectives, typical neural network models use similar models for training stage and deployment stage[12]. For example, small networks should be utilized for automatic sentiment tagging systems because they need to operate in real time. However, the performance is decreased when using small networks. Therefore, knowledge distillation is a promising method to improve the performance of small networks that can be deployed with limited computing power.

To achieve this goal, we train a small network by transferring the knowledge in a large teacher network to the target small network. At training stage, the teacher network is trained with many layers. The teacher network inherits all the layers from ResNet[11] model to its second last fully connected layer and additional layers are added. Then the teacher network is fine-tuned with affective images dataset, ArtPhoto[14], that has 8 emotion classes. Once the teacher network has been trained, the knowledge is transferred from the teacher to the student network. The logits from teacher network preserve probabilities for each class and their values are greater than 0 and less than 1. Logits correspond to soft targets because they do not give one true answer like hard targets. Therefore, soft targets have more information than hard targets. Using this information from soft targets, student network can learn faster with less epochs and higher learning rate because they reduce the variance in the gradient during training. This method is originally introduced by Hinton et al.[12] and it is called distillation. Inspired by this work, we use distillation method, to train a scalable and real time visual sentiment detector with affective images.

This paper is organized as follows:

• In section 2, previous methods for visual sentiment analysis and distillation methods will be reviewed.
• In section 3, the details about model architecture will be explained.
• In section 4, the used dataset is described.
• In section 5, the comparative results are given.
• In section 6, we analyze the results.
• In section 7, the final conclusion is made.

Our efficient visual sentiment detector is trained with logits from a large teacher network. The model architecture of teacher network and student network is shown in Table 1 and Table 2, respectively.

To fine-tune the teacher network, we use ResNet[11] as the backbone network. From the input layer to the second last fully connected layer with 2048 units are used to build the teacher network. The layer is regularized by dropout with 50% and reduced to 8 dimensions, which is the final teacher network, to output the logits. Once the teacher network is trained, the student network is trained with the logits from the large teacher network to calculate the error rates between them.

The student network has two hidden layers and two fully connected layers, which is a lot smaller compared to the teacher network.

The overall architecture of matching logits between student network and teacher network is shown in below Fig. 1.

Fig. 1. 
Overall architecture of proposed system. The student network learns from a large teacher network by matching its logits to the logtis of its teacher

The first and second hidden layer of the student network have 64 and 128 units, respectively.

The first hidden layer is followed by ReLU activation and max pooling. The second hidden layer is followed by global average pooling and two fully connected layers. The last fully connected layer has 8 nodes. The respective logits are the outputs from the last fully connected layer of teacher and student network. The temperature of logits is set to 4 because it showed highest prediction accuracy. The error rates between the logits of the student network and the teacher network are calculated as in equation 1

The respective cross-entropy gradient is denoted as dCdzi for the corresponding logit of the distilled model, which is denoted as z_i. The logits from the teacher is denoted as v_i and the produced soft targets are represented with p_i. The temperature is denoted as T. By learning the parameters from the teacher network, the small student network can converge faster.

ArtPhoto [14] is a public benchmark dataset for affective images. The emotion of each image is defined by the owner. The images are collected from photo sharing sites and the total number of images is 806. We used this dataset because they are easily accessible and can be downloaded from the internet. In ArtPhoto, there are eight emotion classes as follows: amusement, anger, awe, contentment, disgust, excitement, fear, and sad.

Among the dataset, 90% samples are used for training and 10% samples are used for testing. The teacher network is trained with 30 epochs and the batch size is set to 20. We used adam optimizer with learning rate 1e-5 for the teacher network. On the other hand, the student network is trained with less epochs and smaller learning rate of 10 and 1e-3, respectively. To compare the accuracy and training time with other models, we also show the results of VGGNet and AlexNet.

The VGGNet model is enhanced with additional layers. The second fully connected layer of VGG16 is extracted followed by two fully connected layers. Dropout is applied to the first additional layer with 50%. Then the number of nodes is reduced to 8 nodes and softmax function is applied.

Regarding AlexNet, two fully connected layers are removed from AlexNet[7] model due to memory limitation. Instead, one fully connected layer with 2000 nodes is added before softmax function. Both VGG16 and AlexNet use the same hyperparameter settings as the teacher network.

Below Table 3 shows the respective elapsed time of training time for distilled model, teacher model, VGGNet and AlexNet model.

As shown in above Table 3, the training times for each temperature value of the proposed model is a lot less than the rest of the models by a significant margin.

Table 4 shows the comparative results of the accuracy for each model. The accuracy is the rate of predicting the correct emotions evoked from test images out of 8 emotion classes. The teacher model shows the highest accuracy compared to the rest of the models. VGGNet shows the next highest accuracy.

However, large networks such as the teacher model and VGGNet can not be deployed in mobile devices with less storage capacity and limited computing power due to their large number of parameters. Although there is about 6% decrease in the accuracy of the distilled model, it can reach performance as much as the VGGNet showing the next highest performance.

Due to the reduced number of parameters, the proposed model can predict much faster.

The number of parameters of the proposed model decreased roughly by 23M (99%) and 142M (99%) compared to teacher model and VGGNet, respectively. Thus, it is expected to contribute to real-time visual sentiment detector.

The distilled model can be deployed in devices with limited hardware. In addition, there is less latency during prediction due to their less number of parameters. Although there is a loss in accuracy compared to its teacher model, it shows similar performance as much as a large VGGNet. In addition, the accuracy is higher roughly by 13% compared to the variation of AlexNet without distillation. The proposed model is more scalable because it can be retrained with different affective images according to different user objectives due to the fast retraining time.

However, accuracies of existing models trained by directly mapping sentiments to the visual contents is not as good as image classification. It is due to the gap between low-level visual features and high-level emotion concepts. In the literature, this gap is defined as affective gap[14]. In the future, we plan to bridge this gap to improve the performance of large teacher networks.

In this paper, we propose an efficient visual sentiment detector by training a small network with distillation methods. Although a large network can be well trained to show a good performance in classification task, the large computation overhead makes it hard to be generalized to different user objectives. By utilizing distillation method, the deployment model of visual sentiment detector can be efficiently trained.

The performance of efficiently trained visual sentiment detector shows similar performance as much as the large VGGNet. In addition, the proposed model can predict much faster due to its less number of parameters. Thus, it is expected to contribute to a real-time visual sentiment detector.

The performance of student network is dependent on teacher network. Thus, improving the performance of large teacher network can act on the performance of small student network. However, when neural network models are applied to visual sentiment analysis, accuracies are not as good as image classification. To improve the performance of neural network models for emotion recognition, Yang et al. [1] utilized the local information, from which the sentiment is evoked. Inspired by this work, we plan to improve the performance of large teacher networks by giving attention to Region of Interests(ROI) that elicits the sentiment as a future work.