Research on Image Semantic Segmentation Based on FCN-VGG and Pyramid Pooling Module

2.1 Previously used algorithms

The method based on the edge detection is mainly used to realize image segmentation through the edge detection of the region. By using the inconsistency of features between regions, first it detects the edge points in the image and then connects them into a closed curve according to a certain strategy to form the segmentation region. The edges in an image are usually places that are not continuous in grayscale, color or texture. It often needs to use edge detection operators to carry out the edge detection. They include canny operator[5], laplace operator[6] and so on. The edge detection algorithm is more suitable for the segmentation of simple images whose edge gray value is more obvious and noise is smaller. Due to the more complex edges and the strong noise images, it faces the contradiction between noise immunity and detection accuracy. Therefore, this method is often used as a preprocessing algorithm for image segmentation. It is that first preprocessing is performed on image, and then subsequent method is used to deal with the image.

Conditional Random Field(CRF)[7] and Markov Random Field(MRF)[8] are semantic segmentation methods based on machine learning[9]. Markov Random Field is a theoretical model of undirected probability graphs, which has a wide range of applications in computer vision. The idea is to assign a random value to the pixel and then calculate and classify it in a probabilistic way.

The conditional random field is a discriminant probability undirected graph learning model based on the maximum entropy model and hidden markov model. It is a conditional probability model for annotating and segmenting ordered data. The conditional random field model uses the serialization form to optimize and decode the global parameters, solving the paranoid mark problem that other discriminant models can't avoid. It has a good application effect in the fields of natural language processing and image semantic segmentation.

2.2 Convolutional neural network

The method based on the machine learning through training to build the probabilistic model between image features and semantic categories, build a energy function and use the handwritten feature library to optimize the function through iterative computation, finally the image semantic segmentation model is obtained. This machine learning method rely on hand-marked feature library very much, so it is difficult to express the image features widely. Therefore, it has lot of limitations in practical application. The traditional image segmentation is to extract the features such as texture, spatial structure information, color and other features from different regions. Semantic information of the same area is suppressed and there are certain differences between different regions. Machine learning has restrictive in dealing with the original data. It require senior engineer or experts that for different target design different feature extractor and extract the appropriate feature vector.

Convolutional Neural Network (CNN) can automatically learn the distribution of image features and avoid the need of artificial feature extraction. Compared to traditional methods, CNN can learn more abstract image features from a large number of sample images. The traditional convolutional neural network is divided into two parts. The first part consists of multiple convolutional layers and the pooling layers, they are alternately responsible for the feature extraction. The second part is fully connection layers, its task is classification. For ordinary CNN, the input images need to be fixed due to the existence of the fully connection layer. For the convolutional layer and the pooling layer, the size of each layer's feature map is variable. If fully convolutional network is used to extract features of input image, the input image scale is arbitrary. Deep convolutional network is widely used to solve many image processing problems due to its excellent performance in image processing. The core of deep learning is feature learning. Its purpose is to obtain the hierarchical information through the hierarchical network so as to solve the problem that artificially design feature extraction problem. It is simple to say that make the machine to enter and own output. Learning process is to learn the most suitable network parameters weight and bias, under the conditions of the minimum loss to optimize their own learning parameters, finally it is able to accurately predict input data. Convolutional neural network is a neural network which consist of neurons with learnable weights and bias. The basic principles of convolutional neural networks are local connection, weight sharing, downsample. The proposed local connection is to reduce the training weight parameter. The nodes in the hidden layer are only connected with some nodes of the upper layer, it greatly reduces the amount of calculation. Weight sharing can reduce the computational complexity by convolution kernel applications, and downsample is a further reduction of network computation parameters. Fig. 1 is a simple convolutional neural network. It has input layer, convolutional layer, pooling layer, fully connection layer, output layer, but the last layer of convolutional neural network is commonly softmax layer.

Fig. 1.

Convolutional neural network

Here only the ReLU function is introduced, because most convolutional neural networks use ReLU activation function. Other activation functions are sigmoid and tanh functions and so on. Fig. 2 is the image of activation function.

Fig. 2.

ReLU function

Fig. 3 is the process of convolutional calculation. The calculation of convolutional neural network is that image data and convolution kernel are calculated to get a value.

Fig. 3.

Convolutional operation

Fig. 4.

Pooling calculation

Fig. 4 is pooling calculation. It contains max pooling and average pooling. The main role is to reduce the amount of computation.

Fig. 5 is a VGG convolutional neural network, this neural network has 16 layers, including 13 convolutional layers, 3 fully connection layers and 5 layers of pooling layer. The specific structure of the neural network in turn are 2 convolutional layers, 1 pooling layer, then 2 convolutional layers, 1 pooling layer, 3 convolutional layers, 1 pooling layer, 3 convolutional layers, 1 pooling layer, 3 convolutional layers, 1 pooling layer, and 3 fully connection layers.

Fig. 5.

VGG convolution network

2.3 The selection of the framework

The libraries for machine learning and deep learning are TensorFlow[10] Caffe, Theano, MXNet and so on. Here tensorflow is chosen. TensorFlow is an artificial intelligence platform open source software from Google in 2015. TensorFlow is a system that transmits complex data structures to an artificial neural network for analysis and processing. TensorFlow can be used in variety of deep learning areas such as speech recognition or image recognition, and it can be run on a wide range of devices, from as little as a smart phone to as many as thousands of data center servers. Tensorflow can be divided into tensor and flow, tensor is an N-dimensional array, flow is the meaning of graph, and tensorflow is calculated based on graph. TensorFlow running process is divided into two steps, one is construct the model and the other is training. In the construction phase, we need to build a graph that describes our model and then launch it in the session. The map can be understood as a flow chart that is the data input and output process, but this time it is not an actual operation, because TensorFlow is deferred execution model. It must know what you are calculating, and then your execution graph start sending computational tasks to computers. So first we should create a calculation graph in internal storage and start an execution session. Finally actual training tasks is performed by session.run. Fig. 6 is chart of a calculation that use tensorflow.

Fig. 6.

The use of tensorflow

2.4 Fully convolutional networks

Now most people use convolutional neural networks in image processing. The most used frequently convolutional neural networks are Alexnet, VGG, Googlenet[11] and so on. In the image classification task, VGG convolutional neural network can well achieve the task of classification. FCN-VGG is the change of VGG convolution neural network. By converting the normal VGG network’s fully connection layers into convolutional layers, FCN-VGG can be got. FCN-VGG is used to realize image semantic segmentation. Image semantic segmentation is that divides the pixels into different groups according to semantic meaning in the image.

3.1 Pyramid pooling module

The pyramid pooling module can learn different features of multi-scale and then combine, it have stronger expression ability. Fig. 8 is the structure of pyramid pooling module.

Fig. 8.

The structure of pyramid pooling module

Due to the increase of the parameter quantity, the 1*1 convolutional layer is used in the pyramid pooling module to reduce the parameters. Then fuse the feature maps and original feature maps to obtain advanced features, finally the new features is used to perform the pixels classification. Here 4 scales pools are used in the pyramid pooling module, they are respectively 1*1, 2*2, 3*3 and 6*6. Then pooling is performed in feature map respectively. In this module it has used upsample module. Fig. 9 is upsample operation.

Fig. 9.

Upsample operation

Because the ordinary pool will reduce the size of the images, such as VGG’s five pooling layers can reduce the input image by 32 times. In order to get a output image that the size is as the same as the original image, it need upsample. The size of extracted 4-parts using pyramid pooling module is decrease. So first upsample is used to revert to the size of the original feature map, then classification is performed.