Home

Among the various comprehensive review sites currently available, we collected user review information from Metacritic. This website offers high reliability in terms of video game-related data, and its metascore evaluation index has a considerable influence on game evaluation. As of March 25, 2020, the site aggregated reviews for a total of 5,033 games. We collected all reviews—a total of 278,849 reviews—and each item of the collected content comprised the game title being reviewed, author ID, review content, user score, and date of creation.

Specific games were selected from all games that were reviewed, and the datasets were built as in Table 1. The screening criteria are as follows.

• 1. Metascore of 50 or higher, avoiding games with significantly low scores.
• 2. More than 1,000 reviews available, excluding games with fewer reviews.
• 3. Quality of the game reviewed is above a certain level. To ensure that games are limited to a certain level of quality, 100 reviews per game must be reviewed manually.
• 4. Limited to games with knowledgeable reviewers. To ensure that actual game quality was considered, only reviews written by reviewers with an experience of playing the actual game were considered.

If the game failed the first criteria, the low recognition and perceived quality resulted in a low number of user reviews. Fig. 2 shows one of examples for that. Even for highly recognized games as shown in Fig. 1, most were unilaterally criticized while the rest of the reviews included a high amount of slang. Thus, we excluded these games from the dataset screening process. The remaining games were then checked for the overall quality of the reviews before analysis. For example, Fig. 3 show low-quality reviews and high-quality reviews, respectively. Through the quality check process, low-quality reviews were filtered out. Next, the minimum number of reviews required to proceed with the selection was believed to be about 1,000, and therefore, only games with more than 1,000 reviews were selected.

Fig. 1. 
Game with many reviews but with reviews that include many slang terms, unique elements, or profanity – Warcraft 3: Reforged

Fig. 2. 
Example games with a score of less than 50 points and fewer reviews – Rollercoaster Tycoon 2: Wacky Worlds

Fig. 3. 
Examples of low-quality reviews and high-quality reviews

For example, as in Fig. 1, because it is difficult to obtain good summarization results when there are many slang words, special characters, or meaningless words in the dataset, the third criterion was used to investigate the proportion of irrelevant data. After randomly selecting 100 reviews for each game, the quality of the reviews was examined; games that did not pass the examiner's evaluation were removed. Because writing a relevant game review requires knowledge of the target game, the final criteria limits game selection to those with reviewers that had real play experience.

About 1,000 reviews were re-selected for each of the 5 games selected through the process discussed in Section B. The reason for selecting around 1,000 data points is that it is unreasonable to inspect a larger amount of data manually, and if too few data points are selected, the amount of data in the dataset is too small. Therefore, we limited the numbers to approximately 1000±100. In the selection process, non-English reviews were excluded, and reviews were selected based on the number of user endorsements received. The reason is that the fewer are the number of user endorsements, the higher is the amount of profanity or meaningless texts that is part of the reviews, which makes it difficult to use them as valid review data.

Reviews containing summary sentences were selected to use them as training data in machine learning-based game review summarization. Selection processes were conducted manually by four annotators who checked the criteria for each game review and its summary based on the conditions provided below. The annotators reviewed each document by cross-verifying it against these criteria.

• 1. Whether to re-refer the content in the sentence as mentioned above
• 2. If a sentence contains some of the assessment items
• 3. If the writer's opinion is included in the sentence
• 4. Exclude long sentences that cannot be called a summary
  a) Check for Rule 1: The summary was determined from the perspective of recall, and the assessment was possible from the same perspective when comparing with the Recall- based ROUGE.
  b) Check for Rule 2: The assessment items mentioned in Section III.C were used; reviews that could not be assessed by relatively objective evaluation were excluded.
  c) Check for Rule 3: The author determined from the review that the key content he wants to convey is contained in that sentence, which is the basis for the summary's fundamental role.
  d) Check for Rule 4: Criteria were applied to exclude sentences in exceptional cases such as when the text was too short or when the text was about 1,000 characters long.

Further, the analysis was performed to add a summary to the criteria where a particular linking phrase appeared in the process; however, this was excluded because it was not suitable as an absolute indicator.

In the final version, a total of five games were selected and utilized first; about 1,000 reviews were evaluated item-by-item with the criteria provided in Section 3.4.

Among these approximately 1,000 reviews, a total of 400 summaries were selected following the filtering procedure mentioned in Section 3.2. Table 2 shows the statistics of our game review dataset.

We conducted game review summarization using state-of-the-art pre-trained models (i.e., BERT and RoBERTa). The collected and refined game review dataset in Section 3 was used for the experiments. The dataset was randomly divided into the training set and testing set in a ratio of 8:2. Using the training set, the pre-trained models were fine-tuned as our game review summarization models. The summarization results were evaluated on the test set using the ROUGE score. There are several types of ROUGE scores. In this paper, ROUGE-1, ROUGE-2, and ROUGE-L, which set scores according to the match between reference summaries and generated summaries for one (Unigram), two (Bigram), and N (N-gram), were used.

For a set of experiments, we performed a fine-tuning through transfer learning using our game review dataset based on the pre-trained model, BERT provided by Google.

In case of the basic BERT model, the BERT-large-cased model and the transformer model provided by HuggingFace were used as extractive and abstractive models, respectively. For the BertSum model, a pre-trained model was used as a combined extractive and abstractive model. Finally, the MatchSum model was used to extract summaries by matching the contextual representations of the document with the summaries by utilizing the RoBERTa model, which is a BERT model pre-trained with CNN/ DailyMail datasets .

The experimental results listed in Table 3 confirm that the overall score was lower than the existing state-of-the-art score. This result occurred because of the differences in the dataset. Most existing state-of-the-art results used standardized news data such as that from CNN or the Daily Mail. Therefore, relatively free-form reviews such as game reviews showed low scores.

In the extractive summarization method, the ROUGE score was calculated to be relatively high; this was attributed to the examiners using a method similar to the extractive method for generating the correct summary. In the case of MatchSum using RoBERTa, and this showed a score in the mid-20s, which is similar to the current result values.

Another reason besides the characteristics of the dataset is that the amount of game review data used for fine-tuning is small. In terms of fine tuning, the higher is the similarity of the data, the more effective are results that can be derived even with a small amount of data. However, we used game reviews having very different characteristics from general newspaper data, and the amount of data used for learning is small; these reasons may have contributed to a disparate result compared to those of the state-of-the-art models.

Most of the ROUGE-2 scores were low except for Transformer-Extract (BERT). Because the original text was not well used in the summary due to the characteristics of Abstractive Summarization. And also because the summary of the dataset that we built was long.

In the case of BERT (Transformer)-Extract, which had the highest average ROUGE value among the results, a high-quality summary sentence corresponding to the high score was generated as shown below in the result for (a) in Table 4.

However, there are frequent cases where the score is reduced because of the presence of longer sentences. (b) illustrates a representative example that includes the keyword “Amazing,” and if you read the entire contents of the extracted sentence, it is generally favorable for the game. However, the actual ROUGE score was evaluated for a summary with a low score of 11.11 based on ROUGE-1.

Among the abstractive techniques, the state-of-the-art BertSum-based model—BertSumExt Abs—showed a questionable result. In the case of (c), a summary of low quality was generated with a low score.

However, in the case of (d), i.e., in the case of the score, it is a low- quality summary sentence that repeats the same content by mentioning “voice” along with key points and frequently using the word “best.” This summary received a high score of 41.37.

As a result, in the case of the ROUGE score as shown in Table 4, when looking at each case, the quality of the generation summary was better, but the score was lower. And there were cases where the quality was bad but the score was high. As such, it was confirmed that there was a difference between the ROUGE score and the actual result, and through Case Study, the difference was compared using human evaluation in Section 5.

Like movies’ evaluation elements [13], as the production value of cut scenes in games is increasing, graphics and sound have started to weigh in on the gameplay experience, and therefore, it appears more consistently in reviews

In addition, the graphics, error, and system criteria reflect the game's other characteristics. For graphics, the visual quality of games has developed sufficiently to be evaluated as art, and related exhibitions have been held [14]. During our evaluation, we observed that graphical and visual elements were mentioned frequently in the reviews, and they were thus added to our evaluation criteria.

The Error criterion is a prerequisite for playing the game smoothly. Therefore, to describe user experiences, it is essential to determine whether it is possible to play the game smoothly. The system criterion was added to evaluate special systems in the game and the technical optimization of the game.

To find out existing gaps and similarities between ROUGE scores and our proposed QFs, we analyzed five different sample summaries selected from five different games.

For the analysis, QF scores for each of the original text, reference summary, and generated summary were scored through 4 human annotators. Five games in the data set were evaluated, and Table 5~9 show the results.

In the case of BioShock Infinity (Table 5), because the story was very important, there were many mentions about the game’s story itself.

The personalities of the game characters were excellent and the game itself was systematically excellent; therefore, there were many cases where system items were described in detail. Reflecting this fact, both the original text and the reference summary received a high score for the system item among the five QFs, but the generated summary received a low score. This indicates that the generated summary did not well summarize what was mainly mentioned in the review. Unlike these results, the ROUGE score received a high score. Other games also showed different aspects of QFs evaluation and ROUGE score. Through this, we confirmed once again that even a summary with a high ROUGE score could not be convinced in the field of game review.