Current Issue

US President Donald J. Trump wrote to Tweeter on December 10, 2016, six weeks before his presidency, that reports by CNN had been ridiculous and untrue - FAKE NEWS![1] This tweet has sparked controversy over whether CNN broadcasters are producing fake news and has triggered the worldwide issue of “what is really fake news?” Through active use of social network services in recent years, fake news, a story made for the purpose of deception, is spreading rapidly, causing frequent social disruption and becoming a global problem. For instance, the total number of comments on the top-20 fake news distributed through Facebook over the three months of the US presidential election campaign was 8.7 million, while the number of comments on the top-20 major news media was 7.36 million[2]. This fact clearly shows that fake news has more influential than news articles issued by major existing news media. According to the research report of Hyundai Research Institute in 2017, the domestic economic impact of fake news was estimated to be close to 30 billion dollars per year, assuming that the proportion of fake news among all news was 1%. As such, fake news has become one of urgent issues to be addressed as a national and social issue, such as threatening democracy around the world and fostering social division.

Clickbait detection is one of the major issues in the big category of fake news detection. Clickbait takes ad revenue based on the number of clicks that lead to clicks on links that are not interesting or worthwhile for readers. This is causing social disruption due to the spread of fake news and rising as a global problem. For example, most of Google’s revenue comes from Google AdSense, which monetize websites. However, there is an abuse that relies solely on the revenue of the ad to induce the reader to click on the link regardless of the value of the content or the quality of the information. If these occur frequently, users can no longer trust information on the web because many news articles may often contain exaggerated advertising, sexual, or spam content.

Technically, we view clickbait detection to a typical Big Data problem because clickbait articles essentially have four attributes of Big Data - (1) Large volumes of clickbait articles are generated from a variety of sources, including online sites, social media, and mobile messengers; (2) Real-time processing (velocity) is required to block the rapid spread of clickbait articles; (3) Clickbait articles are one of typical unstructured data (variety); and (4) Clickbait articles are filtered out to improve the quality of all news data (veracity). In fact, through these four Vs, Big Data is defined by Gartner and IBM. Obviously, it is time-consuming for human evaluators to manually identify clickbait articles among large volumes of news articles. On the other hand, deep learning models on behalf of human evaluators can detect a lot of clickbait articles automatically, whereas the accuracy of the deep learning models to classify clickbait articles is relatively low, compared to humans. In this work, we focus on improving the accuracy of the deep learning models used for detecting clickbait articles among massive news data.

In the clickbait detection problem, there are two different approaches to avoid the overfitting problem that degrades the accuracy of the deep learning models including FNN, CNN, and RNN[3]. The first approach is to reduce the complexity of the deep learning models. The other is to use large-scale training data. In the former approach, deep learning models themselves have high complexity with a number of hidden units and weight parameters. The accuracy would be decreased if we reduced the complexity of the models. On the other hand, our work focuses on the latter approach. As a good example, the accuracy of Google translation services has recently improved significantly because a sequence-to-sequence model, one of RNN-based deep learning models, is trained with a huge number of training data including one billion pairs of Korean sentences that match English ones. However, it is non-trivial to obtain large-scale training data in which each news article has its class label indicating that it is clickbait or not. Even though we collect many news articles automatically, human evaluators must manually check each news article to determine whether it is really clickbait or not. In this way, collecting large-scale training data by human judgement is not possible.

To solve the above problem, we first propose a new deep learning model that works based on Recurrent Neural Network(RNN) with multi-layers of Bi-LSTM, max-pooling layers, and fully-connected layers. Then, to more improve the accuracy of the proposed deep learning model, we propose a novel algorithm of pseudo-generating the large-scale but yet high-quality training data, thereby considerably improving the accuracy of the existing deep learning models. We also show that the pseudo-generated training data almost match those made by human evaluators. To the best of our knowledge, this is the first study of proposing the deep learning model that solves the clickbait detection problem, gathering the training data automatically.

To evaluate the proposed method, we compare it to SOTA methods such as (1) SOLAT in the SWEN[4], (2) Athene(UKP Lab)[5], and (3) UCL Machine Reading[6]. Our experimental result shows that the proposed method outperforms the SOTA methods in the clickbait detection problem.

The contributions of the proposed method are as follows:

　• Recently, the fake news detection problem is one of the most urgent issues in data engineering. In this article, we propose two novel approaches to solve the clickbait detection problem, the most essential problem among various subproblems in fake news detection. One is a new deep learning model that detects clickbait articles well and has the highest accuracy compared to the SOTA methods. The other is a new approach of automatically generating large-scale training data, given a collection of news articles. To the best of our knowledge, our solution suggests a new methodology that has not existed in the clickbait detection problem.

　• Unlike the SOTA methods, the proposed deep learning model makes use of only word embedding features. Because the SOTA methods use various features in addition to word embedding features, their accuracy is not high. In general, collecting various features is a time-consuming and labor-intensive task and some features are not directly related to solving the clickbait detection problem. In our deep learning model, such hand-crafted features are not necessary.

　• The proposed deep learning model improves the accuracy of the SOTA methods by 36% and our new approach of automatically generating training data significantly boosts up the accuracy of the deep learning model. Thus, if the proposed approaches can be applied to real applications, it will be useful for blocking news articles containing ad and spam.

The rest of this paper consists of the followings: In Section II, we discuss related work. In Section III, we formally define the clickbait detection problem. Furthermore, we present the details of the proposed methods in Section IV. Subsequently, Section V shows our experimental set-up and results. Concluding remarks and future works follow in Section VI.

For clickbait detection, Mengxi et al. integrated knowledge graph with deep learning techniques to achieve explainability[8]. Dimpas et al. presented a simple method based on only Bi-directional LSTM for determining if it is clickbait from Filipino and English news articles[9]. Our experimental results show that our proposed model is better than Bi-directional LSTM. We will discuss the detailed results in the experimental validation section. In Wei and Wan work, an ambiguous headline is a headline whose meaning is unclear relative to that of the content of the story and a misleading headline is a headline whose meaning differs from that of the content of the story[10]. The definition of their clickbait detection problems is different from our problem that is defined as deliberately clickbait detection. Especially, the main contribution of Wei and Wan is to exploit surface-level linguistic features collected by human annotators. In general, since this task is time-consuming and labor-intensive, it is difficult to collect large-sized label dataset, where some labels are biased. Thus, this hand-crafted feature engineering limits to improving accuracy of learning algorithms. Because of these reasons, we excluded Wei and Wan’s methods as baseline method. Instead, we compared the proposed method with the three clickbait methods below.

Firstly, the SOLAT in the SWEN proposed the ensemble method between a deep neural network and decision tree models. This model predicts either true or fake news, based on an weighted average between the two models. As illustrated in Figure 1 (a), two deep neural networks are made for news headlines and bodies independently. Each deep neural network is composed of three hidden layers on top of five convolutional and max-pooling layers. Each word (w) of a headline(e.g., ‘local’) is transformed to k-dimensional word vector using word2vec. For example, the first dimension of the word ‘local’ is 0.012 and the second dimension is 0.141. The k-dimensions indicate the similarity scores between w and each of k words related to w in the vocabulary of the corpus. k headline vectors are entered as the deep neural network for headlines. Through the five convolutional and max-pooling layers, the most discriminative word features are selected from k·(# of words) in the headline. This is, the objective of this process is to abstract the word features of the headline. In the same way, the deep neural network for news bodies is trained to abstract the word features of the news body. Then, the three hidden layers are used to find the optimal parameter values through back propagation. If the model predicts incorrectly, the weight values among nodes in different hidden layers are readjusted to minimize the errors between the actual values and the predicted values. In most cases, this deep learning model works well but another method called gradient-boosted decision tree method(XGBoost) is proposed to improve the accuracy of the deep learning model. XGBoost is an ensemble of weak prediction models - decision trees.

Fig. 1. 
SOTA clickbait detection methods

The feature set for XGBoost is Count, TF/IDF, SVD, Word2Vec, and Sentiment. The count feature is # of uni-grams, bi-grams, and tri-grams that appear simultaneously in the news headline and body. TF/IDF is the normalized value of the count feature, quantifying the importance of the uni-grams, bi-grams, and tri-grams. Through SVD, top-k important uni-grams, bi-grams, and tri-grams are selected. The word2vec feature is the similarity scores between w (one of uni-grams, bi-grams, or tri-grams) and each of k words related to w in the vocabulary of the corpus. The sentiment feature includes the polarity values (positive, neutral, or negative) of the uni-grams, bi-grams, and tri-grams. The experimental results show that XGBoost supplements the weakness of the deep learning model. The reason is: Some words in a news article are so ambiguous that the news article is classified incorrectly. It turns out that such words are less commonly used in the corpus. The count and TF/IDF features of XGBoost can detect such words easily and prevent some sort of misclassification.

Secondly, the models in Figure 1(b) and (c) are quite simple, compared to the deep learning model presented by SOLAT in the SWEN. Athene proposed five-layers FNN model to train the headline vector, the body vector, and the joint vector. They uses embedding of the nouns and verbs of the headline and the body as word features. And they also uses features such as Non-Negative Matrix Factorization, Latent Semantic Indexing, Latent Semantic Analysis to derive latent meanings through the matrix. Each of the above five features is trained independently and composed as an ensemble model.

Thirdly, Athene’s model is based on deep feed-forward neural network, while UCL’s model is based on shallow feed-forward neural network. The UCL uses BoW-TF and the cosine similarity between headline and body. This model is concatenated to learn features and has a hidden layer. Unlike SOLAT’s model, the two models do not use convolutional and pooling layers to abstract word features. Besides, the models are close each other in that the input of the models includes joint feature vectors.

In this section, we formally define the clickbait detection problem which is one of the important problems in fake news detection, referring to Table 1.

A news article n_i is composed of news headline H(n_i) and news content B(n_i). The ad and non-text data are removed in the news content so the output of B(n_i) is the list of words in the news content. The order of the words is not also considered in the problem. In this set-up, a fake headline news H_f (n_i) is defined as ∃n_i such that H(n_i) ≠ B(n_i). Given a news article n_i as input, the goal of AI-based models is to automatically classify n_i to either H(n_i) or H_f(n_i), where H(n_i) and H_f(n_i) are the true and fake headline news, respectively. This classification problem is also known as the stance detection problem. As already discussed in Section II, most solutions to fake news detection are focused on social media, and the features used are user profile information, context, network structure, content, and multi-modal features. However, relatively the only content-based clickbait detection problem with news data is rarely studied until now [7]. In this work, we focus on the content-based clickbait detection problem with news data.

As described above, the goal of our research is to automatically determine if the title and content of a given news article are consistent or not. By detecting fishing titles or deliberately distorted titles, we can prevent disinformation from being transmitted due to news headlines. In our work, the fake headline news that corresponds to five levels of difficulty are classified by domain experts. Table 2 shows one of typical examples in the clickbait detection problem.

The very easy fake headline news is the level at which the revision of the title can be found in the article “Matching Vocabulary in a Sentence”; The easy fake headline news is the level at which the modification of the title can be found in the article “Changing a vocabulary within a sentence”;

The normal fake headline news is the level at which the correction of the title can be found in the article “Change sentence in one sentence”; The difficult fake headline news refers to the level of finding the revision of the title as “sentence inference in one sentence”; The very difficult fake headline news refers to the level at which the revision of the title can be found in the content of the article as “sentence reasoning more than two sentences.”

In the previous section, we explained the main SOTA models, all of which are based on deep learning models like FNN and CNN. However, we claim that such methods do not work well because they do not consider the sequence of words per sentence. In the language model, the following word are generated under the influence of the previous word[11]. For example, in a sentence “I eat XX”, the probability of generating the word XX is directly affected by the previous word ‘eat’. The probability of generating ‘steel’ as XX is very low, while that of generating ‘apple’ is very high. Since news headlines and bodies consist of sentences, the order of words per sentence should be considered in deep learning models. To handle this point, we propose a new deep learning model that is the hybrid of news headline and body models. As shown in Fig. 2, each model is composed of four hidden layers, where the first and third layers are Bi-LSTM (Bidirectional Long Short-Term Memory), and the second and the fourth layers are max-pooling layers. By Bi-LSTM, the generation of the current word is influenced by both previous and next words. Through max-pooling layers, word features are abstracted to a few features as the most discriminative features. The fully-connected layers are used to concatenate the headline and body models. The input of the proposed model is word2vec of words and the output of the model is the probability values of true and fake headline to a news article.

Fig. 2. 
Proposed model

As we already discussed in the introduction section, we also propose a novel method that improves the accuracy of existing deep learning models by pseudo-generating large-scale but yet high-quality training data. It is well known that all deep learning models work well if the size of the training data is increasing. In the clickbait detection problem, to pseudo-generate training data, we propose Algorithm 1.

Given a news article n_i, Algorithm 1 generates a fake headline. First, compute the similarity score between H(n_i) and each of sentences in the news body and then mark the sentence with the highest score as the key sentence K(B(n_i)). Next, extract nouns from H(n_i) and K(B(n_i)). Finally, generate a fake news headline to exchange the noun of H(n_i) with the noun of K(B(n_i)), where the nouns are chosen at random. In this way, Algorithm 1 can automatically generate a large amount of training data for deep learning models.

Recently, the fake news detection problem is one of the most urgent issues in data engineering. In this article, we propose two novel approaches to solve the clickbait detection problem, the most essential problem among various subproblems in fake news detection. Specifically, we propose a novel deep learning model that abstracts the word sequence in news title and body for detecting clickbait articles well. Furthermore, we propose a brand-new algorithm for automatically pseudo-generating large-scale but high-quality training data, given a collection of news articles. Our experimental results show that the proposed deep learning model improves the accuracy of the existing deep learning models by 36% and our new approach of automatically generating training data significantly boosts up the accuracy of the deep learning model.

Our future research direction is to expand the proposed models to another subproblems in the fake news detection problem. One of the typical problems is to detect news content that is not related to the context of news. In other words, detecting mixed advertising or fishing content in news texts prevents exposure to unintended information. In detail, we will attempt to three cases. In the first case, a news article includes some paragraphs of other field in the article’s overall content. In the second case, it includes some sentences of the other field in the overall article. In the last case, it includes some paragraphs of the same field in the entire article.