Journal Archive

The detection and segmentation of cracks on a pavement surface are the important tasks in a pavement maintenance system. The traditional manual inspection method is time-consuming, labor-insensitive, and potentially hazardous for both inspectors and road users. Moreover, manual inspection completely depends on the individual specialists experience and knowledge, which lead to low objectivity in quantitative analysis [1]. For these reasons, image processing based techniques for crack detection have attracted significant attention in computer vision and other research communities. In this work, we focus on developing a new approach to automatically detect and segment cracks from gray-scale images.

Various crack detection methods have been suggested based on image processing. These methods can be classified into three categories, namely: conventional methods, current methods, and deep learning-based methods. The conventional methods basically attempted to find suitable thresholds to isolate cracks from input images.

However, these methods did not result in good detection due to numerous crack shapes, different sizes, and various noises such as shading and intensity inhomogeneity. In order to overcome these problems, many recent techniques, such as the minimal path-Based techniques, have been proposed to detect cracks based on the assumption that cracks appear darker than surrounding contexts. In general, machine learning based methods have been introduced to classify cracks on input images based on nominated features. The major disadvantage of machine learning-based methods is that the system performance relies on features and parameters which are usually difficult to select. Recently, deep learning-based approaches have achieved significant performances in detection, classification, and segmentation problems without any assumption of data distribution. Deep neural networks are able to learn and adjust trainable parameters based on numerous of training data samples. Therefore, millions of data points are necessary to train a deep neural network. More specifically, deep learning methods have been successfully applied on pixel-wise detection and segmentation tasks for medical applications [2][3]. In addition, U-shaped networks [2][4] have shown good performance with fewer trainable parameters than other traditional convolution neural network (CNN) based methods for segmentation tasks.

In this paper, we address the problems of crack detection and segmentation on gray-scale images with a small number of training images using a deep learning method based on an end-to-end training approach. Our approach is inspired by the well-known U-net network [2]. The original U-net network structure includes millions number of trainable parameters which is not suitable for practical applications. Therefore, a modified version of the U-net network is introduced in this paper including the encoder and decoder branches called the modified U-net network, which contains a small number of learning parameters.

In addition, we figure out that the modified U-net network cannot guarantee the system robustness, especially in the case of cracks which have small sizes. In order to overcome these limitations, a high-level features network is proposed to learn deeper features from input images independently in another branch. A fusion layer is applied to merge matrix features from the encoder branch of the modified U-net network with the High-level features Network. This combination dramatically improves system performance with a small amount of training data and a small number of learning parameters.

In the rest of this paper, we review the related works in Section II. In Section III, the proposed method and network architecture are discussed in detail. Two open data sets and experimental results are given in Section IV. Finally, Section V concludes this paper.

The problems of crack detection and segmentation can be considered to be the first step of pavement evaluation and maintenance systems, where the purpose is to distinguish cracks from other areas on pavement surfaces. Crack detection problems are naturally challenging due to the inhomogeneity of intensity, low-contrast, shadows, and cracks’ complexity.

In early works, crack detection and segmentation problems were addressed using conventional edge detector algorithms. Zhao et al. proposed an improved Canny edge detection algorithm for pavement edge detection applications [5]. They mainly used Mallat wavelet transform to strengthen the weak edges of input images and set the proper threshold based on the quadratic optimization of a genetic algorithm. Threshold-based approaches were applied due to their simplicity and low computation time. Oliveira and Correia [6] proposed the use of entropy and dynamic threshold methods to automatically segment cracks from input images. First, they applied morphology filters to reduce the variance of pixel values. A dynamic threshold was adopted to extract dark pixels which were considered to be crack candidates. After dividing threshold image into non-overlapping blocks, they computed entropy and applied another dynamic threshold in order to obtain final crack results. A fast crack detection method based on percolation image processing for a large-size concrete surface image was introduced in [7]. The termination- and skip-added procedures were proposed to reduce the computation cost. A Local Binary Pattern was introduced in [8] to detect cracks, whereby local neighbors were first classified into smooth areas and rough areas. The local pattern was applied on the rough area in order to obtain structure information. The Laplacian of Gaussian algorithm was presented for detecting cracks on road surfaces [9][10]. Generally, these conventional methods are sensitive to noise, which leads to unreliable results in poor lighting conditions.

More recently, Minimal Path-Based techniques were widely applied to detect cracks from images. In these techniques, an image was considered to be a graph of pixels weighted by pixel intensities. A minimal path was defined as the path in which the sum of pixel intensities was the smallest value. Nguyen et al. introduced a method of calculating features along every free-form path which was able to detect cracks with any form and any orientation [11] namely Free-Form Anisotropy (FFA). A dynamic programming implementation of the FFA approach was presented by Avila et al. in [12]. Amhaz et al. presented a Minimal Path Selection (MPS) approach where the end points of a crack were selected from a local scale and the minimal part of the crack was selected from a global scale [13][14]. Another more general method was presented by Kaul et al. in [15], in which complex curves were estimated without knowledge of either the endpoints or the topologies of curves. Their algorithm required only one arbitrary initial point to detect the complete curve. A fully-automatic method was also proposed [16].

They developed a geodesic shadow-removal algorithm to remove pavement shadows from input images and generated probability maps based on a tensor voting algorithm. A set of crack seeds was sampled from a crack probability map. Finally, minimum spanning trees were used to extract the final crack curves. A multiple-scale fusion crack detection (MFCD) approach based on the minimal intensity path was introduced by Li et al. [17]. The authors estimated the crack candidates at each scale and evaluated them based on several statistical hypotheses. The main disadvantage of minimal path-based method is that the computation cost is too high for real-time application.

In additional, a numerous of approaches relied on machine learning algorithms to classify crack and non-crack images or detect the cracks’ positions from input images were proposed. A fully automatic crack detection system was proposed called CrackIT [18][19], where different classification strategies were applied to classify the cracks on non-overlapping image blocks. Cord et al. proposed an approach using texture patterns based on AdaBoost classifier. In addition, the Support Vector Machine (SVM) classifier was applied to classify crack and non-crack images based on texture and shape descriptors [20] and graph-based features [21]. The Support Vector Machine-based method was applied in order to compute probability maps using information on multi-scale neighborhoods; this was called a Probabilistic Generative Model - Support Vector Machine (PGM-SVM) [22]. In addition, the Random Structure Forest-based method demonstrated good performance as well [23]. However, features extraction and selection steps, which are challenging tasks, are necessary to applied machine learning-based approaches.

In recent years, deep learning has demonstrated significant performance enhancements in image classification, detection, and segmentation tasks [24]. Zhang et al. proposed the ConvNets method [25], which included four convolution layers, four max-pooling layers, and two fully connected layers, to classify an individual pixel in terms of whether or not it belonged to a crack, based on local patch information. A combination between CNNs and the sliding window technique for detecting cracks was proposed in [26]. A neural network classifier trained on ImageNet pre-trained VGG-16 DCNN features was proposed in order to distinguish between crack images and non-crack images [27]. Fan et al. [28] addressed the problem of automatic crack detection based on a structured prediction with CNN. On the other hand, CNN was applied to calculate the mean texture depth (MTD) of a crack without computing the surface texture features statistics [29]. The proposed network included only one convolution layer, one pooling, and one fully connected layer.

However, thousands to millions of images are necessary for training the network. Zhang et al. proposed CrackNet [30] to detect cracks on 3D asphalt surfaces with four layers and included more than one million parameters that were trained in the learning process. A recurrent neural network (RNN) version called CrackNet-R was proposed in [31] for fully automated pixel-level crack detection on three-dimensional (3D) asphalt pavement surfaces. An end-to-end style processing approach was proposed in [32] called a fully convolutional network (FCN). The down-sampling branch was from the famous VGG19 networks [33], and the up-sampling branch was implemented by adding specific layers from convolutional layers and deconvolution layers. Due to the use of VGG19, the number of learning parameters was up to millions, which much more than in our proposed approach. In general, the aforementioned deep learning-based methods required substantial amounts of input data images, and the learning time was extensive due to the excessive number of parameters that need to be learned.

In order to overcome these weaknesses, we propose a new network architecture for crack detection and segmentation at the pixel-level. The parameters in the proposed method are trained as an end-to-end style with only one input channel. The outputs of high-level features and low-level features, which were extracted independently, were fused so as to enhance the system performance. Moreover, our proposed network architecture can be considered a lightweight network, since the number of parameters trained during the learning task is around 500,000 parameters. We carefully tested and compared our proposed network architectures with other methods, including recent methods and other deep learning-based methods, on the same data sets.

Our approach mainly involves three steps. First, the input image is converted to gray-scale and pre-processed so as to reduce the effects of noise before being the input of the proposed network. The min-max normalization, histogram equalization, and Gama correction methods are adopted for the preprocessing task. Second, we employ a self-design network in order to learn the crack features from the input image. The proposed deep neural network includes two different feature extraction branches based on a convolution neural network with a small number of trainable parameters. In detail, the modified U-net network is introduced in the first subbranch. On the other hand, we propose an additional branch network to extract high level features. The learning features from the encoder branch and high-level features network are independent. The fusion layer allows us to merge the low-level features from our modified U-net network with the high-level features for further processing based on sharing parameters. The entire system is trained in an end-to-end fashion. The output of the system is the probability segmentation map.

Finally, we simply apply a threshold in order to obtain the binary output map. With a small amount of training data, our method achieves impressive detection results. A schematic overview of all of the processing steps is depicted in Fig. 1. The following subsections discuss these aforementioned steps in details.

Fig. 1. 
Overview of crack prediction

3.2 Network architecture

The proposed network architecture is inspired by the U-shaped structure networks. A modified version of U-net network is firstly introduced. In additional, we proposed a high-level feature network in order to learn deeper features separately.

The fusion layer is then adopted to combine the low-level features and high-level features. These modifications in the network architecture boost the system performance by a surprising amount. We separate the network architecture into two main networks: the modified U-net network and the high-level features network, as shown in Fig. 2.

Fig. 2. 
Schematic architecture of the proposed network

3.2.1 Modified U-net network

The first part of the network architecture is based on the U-net architecture called the modified U-net network. It starts with a down-sampling (encoding) subbranch including four states. Each state includes two 3×3 convolution layers followed by a rectified linear unit (ReLU) [35] activation function, which has computational efficiency and is suitable for deep networks. Compared with the original U-net, we propose the use of zero-padding to retain the image shape after convolution layers. Moreover, Batch Normalization [36][37] is also adopted after each convolution layer in order to prevent an internal covariant shift as well as to ensure faster training convergence. A dropout layer [38] is applied between the convolution layers in the same state so as to prevent over-fitting of the system and improve the generalization of the network architecture. The dropout ratio is set to 0.2 after several experiments. Similar to U-net, the 2×2 max-pooling layer halves the resolution of the input matrix at the end of each stage, except for the last state. The number of filters in each layer is selected after several attempts, and the selected filter sizes are turnings of 4, 8, 16, and 32, as depicted in Fig. 2. In the deepest stage, the output layer matrix is merged with a high-level features network.

The up-sampling branch is based on a 3x3 de-convolution layer and convolution with ReLU, as well as zero-padding to recover the original size of input image. The architecture includes a contracting path to capture the context and a symmetric expanding path, as stated from the original U-net. Batch normalization and dropout layers are also applied on the up-sampling branch. However, the number of filters is decreased as follows: 32, 16, eight, and four. Finally, at the end of the network, the 1x1 convolution layer is adopted in order to obtain the final probability map.

3.2.2 High-level feature network

As aforementioned, the high-level feature network is proposed to extract deeper features and work independently with the modified U-net network. A very deep convolutional network for image recognition was proposed in [33]. While this approach is well-known, the number of trainable parameters was relatively high due to the increased number of filters. We propose using a network called the high-level features network with only four stages of 3×3 convolution layers with ReLU, zero-padding, and 2×2 max-pooling layers, as presented in Fig. 2.

In the first two stages, we use two convolution layers in each stage, with the number of filters being 16, 16 and 32, 32, respectively. In contrast, in the last two stages, we apply three convolution layers in each stage with the filter sizes of 64, 64, 64, 128, 128, and 128, respectively. At the end of each stage, a 2×2 max-pooling layer is applied so as to reduce the sizes of the feature maps. We apply Batch-normalization immediately following each convolution layer. In the final stage, the feature map is merged with the feature maps from the encoder branch of the modified U-net network by the fusion layer. None of the layers used in our proposed architecture are fully connected.

An effective approach for automatic road crack detection at the pixel level based on a deep neural network was introduced in this paper. We propose a novel network architecture which combined the modified U-net network and the high-level features network using a fusion layer. The training is implemented in an end-to-end style. After applying conventional pre-processing algorithms, we train our network with a small number of input images. The proposed network architecture outperformed eight state-of-the-art methods on two published data sets. In terms of cross data generation, we observe that the trained models are affected by label data. The thinning of cracks in the training sample images led to degradation of the segmentation results. In fact, the ground truth label was made manually. Therefore, the system is able to achieve better performance in the case of a carefully labeled dataset. Based on these achievements, the proposed network architecture can be applied in further studies on different types of dataset, such as vessel segmentation, road segmentation from satellite images. In future, this work is able to use in embedded monitoring system for real applications such as road or bridge surface monitoring systems.