Current Issue

Real-world visual processing applications require collecting information on the state of different objects to localize and identify specific objects for the required application. However, incomplete and inaccurate localization and identification can create a system failure that can hamper the performance of a real-time visual processing application. For example, driving assistant systems and satellite imaginary examination systems rely on real-time visual processing models. Current research interest in autonomous driving systems accelerated the development of modern technologies [1], [2] growing an exciting research domain. Critical challenges in this area are usually related to computer vision, like understanding ongoing progress and managing unpredictable situations even under difficult weather conditions. Fig. 1 shows an example of scene classification and assisting system for autonomous car.

Fig. 1. 
Scene classification and assisting system for autonomous car

Autonomous driving assistants deal with the real-time video processing system and numerous weather conditions, which is the most challenging task for the vision application. Extreme weather conditions can affect the vision system of the autonomous driving system to localize objects and to making a decision.

Essential components of the driving assistant system heavily depend on constant and intense image-based object recognition abilities building with cameras. Object recognition has been specified by a variety of different methods and is currently ruled by supervised artificial neural network architectures, such as Faster R-CNN[3], EfficientNet[4], and YOLO[5].

However, these models are still insufficient for operating under uncertain conditions like severe weather conditions especially during rainy nights, and foggy weather is the main challenge for vision algorithms[6]. The mentioned issue can be overcome by linking the scene understanding and image reconstruction strategy to evolve the vision-based application's performance. Recent developments in Generative Adversarial Networks (GANs) introduced a promising approach to image generation[7], image reconstruction[8], and style transfer[9], leading to tremendous computer vision applications to solve real-world vision tasks.

This paper proposed an adversarial scene reconstruction and recognition algorithm to assist an autonomous driving assistant system.

The proposed model consists of two essential components: the GAN-based adversarial network and the object detection module as for scene recognition. These two approaches are employed for scene reconstruction by adversarial network and scene feature extraction, computing optical flow information of the output images for object detection.

Scene understanding and activity recognition are core problems of computer vision, which involve several applications, including robot vision, aerospace monitoring, and automotive cars. High-performance computers became the passkey to compute a large amount of data at high speed. The deep learning method derives the feature-based classification approach named the image local features and applies the learning method to perform understanding specific scenarios and action recognition.

Two-stream convolutional models are one of the most typical computer vision solutions for scene classification and recognition. In this paper, two convolutional models are used, one for scene reconstruction. Another one adopts the temporal feature extraction strategy, which examines the actions from images for the feature extraction, which learns from the visible movement vectors of different frames. Then, outputs of the two models are consolidated at the end with the detection task.

2.2 Temporal-Spatial Feature Extraction

The reconstructed images are first fed into a temporal-spatial feature learning module, where the detection framework learns features information. An illustration of the components of the temporal-spatial feature learning module is given in Fig. 2.

Fig. 2. 
Network architecture of inherent augmented style encoding adversarial network

According to the typical architecture for feed-forward neural networks, these convolutional neural networks are built by interlacing convolution layers and average pooling layers. First, the convolution layer performs a nonlinear mapping, and the pooling layer learns object information. Then, referring to the learned object information, the detection model examines and performs the detection task.

f ⊗ ni=1N i  ∑M ε NifM. nui,j

(1)

In equation 1, features denote f_(i), N(i) denote the set of neighbors of node i, and n(u(i,j)) denote the weight parameter constructed from the kernel function n.

Image context transformation is one of the crucial tasks in adversarial learning. Several robust algorithms [11], [12] were introduced to solve the image context transformation task. However, those methods face difficulties translating a single image to a set of different images in a target domain. Therefore, the proposed network introduce transforming extreme weather scenes to the normal scenario and performing the detection task through the image.

The proposed model performs well in scene reconstruction with wider diversity. However, the conventional methods are less effective in scene reconstruction tasks that include significant shape variations, e.g, translating the background objects into the image images. The scene reconstruction results are shown in Fig. 3 and 4.

Fig. 3. 
Comparison of reconstructed results between proposed model and SingleGAN

Fig. 4. 
Reconstructed results of proposed network (foggy to normal day scenario)

The proposed method is evaluated against the CycleGAN[18], AINN[19], DRIT[10], and SingleGAN[20] models. The CycelGAN and AINN network is capable of the multi-context reconstruction task, but the proposed network achieved a higher evaluation score than the convention models for multi-context reconstruction. The scene reconstruction evaluation results are illustrated in Table 2.

Meanwhile, the detection model is trained with the Cityscape and Alderly dataset to recognize the traffic scenes to identify the object and notify about the detected objects.

The proposed network's detection results are shown in Fig. 5, and the comparison of detection results with the other robust detection model is illustrated in Table 3.

Fig. 5. 
Detection results of the proposed network

Analyzing all the reconstruction results, the proposed model achieved enhanced performance in translating rainy night scenes into clear daylight scenes and foggy scenarios to normal day scenarios.

Moreover, the reconstructed image detection task was conducted, and the object detection module is trained with six different classes of objects. The scene reconstruction task detection task also achieved higher accuracy in detecting and understanding the scenes.

This paper introduced a scene reconstruction and scene understanding framework applied to reconstruct the rainy-night image scene to add daylight and foggy images to the clear day scenario image. The network adopts an adversarial network for visual scene reconstruction and detecting objects a temporal-spatial feature extraction detection model used for the driving assisting system. The network reconstructs the daylight image to enhance visibility without harming any context information of the base image, which leads to assisting the module's mainframe in notifying the detection information. The proposed network achieved 88.1 percent accuracy in the rainy-night image to day image reconstruction task and 88.6 percent in the foggy image to day image reconstruction task. In addition, the network achieved 89.2 percent accuracy in the scene understanding object detection task. In the future, we will aim to include the pedestrian and road patterns information to increase the working area of the proposed mode by improving the network architecture.