Explainable Multimodal Visual-Contextual Framework for Restoration of Severely Damaged Korean Historical Rubbings

Based on the classified data, this study proposes a multimodal restoration architecture that fuses a Language Model and a Computer Vision (CV) model. To achieve precise restoration of damaged characters, the system integrally utilizes both the contextual information of the text and the structural features of the image. In this paper, 'Visual Restoration' does not imply the physical generation of new character forms; rather, it refers to the process of identifying the correct character by extracting structural visual cues from the damaged input. The detailed process is as follows.

The visual restoration model utilizes the Swin Transformer V2 as a backbone to restore damaged character regions. The entire inference process consists of three stages: input preprocessing, feature extraction, and probability calculation. First, the character patch extracted from the original image I and the bounding box b is resized to a fixed resolution 256 × 256 and normalized to generate the input tensor x_in, as shown in (8).

The generated tensor x_in is then input into the backbone function Φ_Swin, parameterized by θ_v2 as described in Section II, and encoded into a high-dimensional feature map Z containing structural information of the character, as expressed in (9).

Finally, the feature map z passes through a classification head to be converted into a restoration probability distribution P_v is over all character classes, as shown in (10). Here, GAP denotes Global Average Pooling, which aggregates the spatial information of the feature map into a single feature vector by computing the average value of each channel. This vector is then projected into the class space via a linear layer with weights W_h and bias b_h. Based on this distribution, the system selects the top-k restoration candidates and forwards them to the ensemble module.

The subsequent language model component performs context analysis based on SikuRoBERTa. By analyzing the bidirectional context surrounding the position of missing characters, the model calculates the probability distribution of Ancient Korean characters that fit the historical and grammatical context, thereby contributing to the derivation of the final restoration result.

To organically integrate the visual inference probability P_v analyzed by the CV model and the contextual suitability probability P_n derived by the language model, this architecture adopts the harmonic mean-based F1-score[14] as the final restoration metric, as defined in (11).

Based on the calculated combined score (Score_final), the system integrally reflects the probabilistic evidence from both models to propose the optimal candidate character most likely to occupy the damaged region. Once the original text is finalized and restored through this process, it proceeds to the final translation stage to convey the textual meaning. The finally restored source text is translated into modern Korean using the Gemini-2.5-flash-lite model. To ensure professional quality and historical accuracy, an 'Expert in Translating Rubbings' persona was established for the model. The specific prompt configuration and constraints are detailed in Table 1. Furthermore, to ensure consistency in translation style and enhance the processing of archaic language, a 3-shot In-context Learning technique was applied, as shown in Table 2.

Table 1.

Prompt strategy: instructional framework

Table 2.

In context learning: 3 shot example

The overall translation process begins with syntactic structure analysis to grasp the context of the source text, followed by key entity extraction to identify major proper nouns such as person names and location names. Subsequently, in the final translation stage, the system synthesizes the preceding information to generate a Korean translation grounded in the historical context. Notably, to preserve the damage state of the original text, strict constraints were imposed to maintain the exact count and positions of illegible symbols (e.g., □) without arbitrary estimation or deletion. This effectively suppresses hallucination phenomena caused by the model's arbitrary interpretation and secures the reliability of the translation.

The dataset used in this study is based on the AI-HUB Old Book Hanja OCR dataset[27], a public database released by the South Korean government. This dataset includes rubbing images along with bounding box coordinates and character labels for individual Hanja characters. A total of 11,416 rubbing images were utilized, from which approximately 2.02 million individual character instances were extracted. To ensure the reliability of the experiments, the dataset was partitioned into training, validation, and test sets at an 8:1:1 ratio. Furthermore, to guarantee reproducibility, a fixed seed value was used throughout all data splitting and training processes.

The rubbing data exhibits extreme variations in quality depending on the material of the stone monuments and the capturing environment. Additionally, damage to the character regions occurs frequently due to prolonged physical wear and environmental factors. Because it is difficult to obtain the original ground-truth images prior to the damage, synthetic training data was generated by applying advanced augmentation techniques to simulate realistic damage patterns. Specifically, six types of augmentations (three types of character damage and three types of background degradation) were integrated into the training data. However, to ensure that the reported performance metrics accurately reflect the model's ability to handle real-world degradations, all evaluations were performed solely on the original test set without any augmentations applied.

Furthermore, to address these domain-specific challenges and ensure the consistency of the model inputs, ImageNet-based normalization and intensity polarity normalization were applied. These processes are presented in Equations (12) and (13).

Here, μ_ImageNet and σ_ImageNet denote the mean and standard deviation of the ImageNet dataset, used to standardize the data distribution. In the training phase, to go beyond preventing simple overfitting and to precisely simulate the complex degradation patterns found in actual rubbings, we applied Advanced Augmentation techniques that integrate physical damage and environmental distortion to the original images shown in Fig. 4(a)[28].

Fig. 4.

Examples of the proposed degradation-based augmentation applied to stone-rubbing character images: (a) Original, (b) Scratch, (c) Blur, (d) Loss, (e) Aged texture, (f) CLAHE, and (g) Vignetting

First, the types of Character Damage are as follows:

1. Scratch: As shown in Fig. 4(b), complete black (pixel value 0) straight lines with random angles are overlaid on the original image. This simulates a "Deep Cut" where stroke information is completely lost, rather than a transparent overlay, thereby enhancing the model's ability to restore disconnected strokes[29]. This is expressed in (14), where ⊙ denotes the element-wise product, 1 is a matrix of ones, and M_cut is a binary mask representing the trajectory of the cut stroke.

2. Blur: As shown in Fig. 4(c), Gaussian Blur is applied to simulate the phenomenon where stroke boundaries become indistinct due to prolonged weathering[29]. This is defined in (15).

3. Loss: As shown in Fig. 4(d), this simulates paper tearing or omission. The missing areas are filled not with a simple solid color but with High-frequency Noise to blend naturally with the surrounding texture[29].

In (16), M_miss is a binary mask representing the irregular missing region, and N_hf denotes the high-frequency texture noise term used to fill the void.

Next, the types of Background Degradation are as follows: 4. Ancient Document Texture: As shown in Fig. 4(e), an aged texture synthesized with Gaussian noise and irregular spots is applied to reproduce the rough surface texture characteristic of traditional Korean paper (Hanji)[29].

In (17), T_ancient represents the ancient document pattern, $$ \text{}$$eta is noise, and $$ \text{}$$lambda is weighting coefficients controlling the synthesis intensity.

5. Contrast Enhancement: As shown in Fig. 4(f), local contrast is forcibly maximized using Contrast Limited Adaptive Histogram Equalization (CLAHE) to simulate adverse conditions where background noise becomes more prominent than the strokes[30].

In (18), J_HE refers to the contrast-limited adaptive histogram equalization function, with limit and grid representing the contrast limit threshold and grid size, respectively.

6. Vignetting: As shown in Fig. 4(g), this applies optical distortion where the image periphery darkens due to uneven illumination[31].

In (19), (c_x, c_y) are the center coordinates of the image, and $$ \text{}$$alpha is the vignetting intensity coefficient determining the degree of light fall-off at the edges. This integrated augmentation strategy ensures the model robustly extracts unique structural features of characters even when stroke information is physically severed or interfered with by background texture and lighting.

The optimization settings adopted to ensure stable convergence and reproducibility of the fine-tuned Swin Transformer in a resource-constrained environment are detailed in Table 3.

Table 3.

Hyperparameter settings for swin transformer fine-tuning

Additionally, to mitigate the weakening of training signals due to the long-tail class distribution, Weighted Cross-Entropy Loss was applied. Class weights were based on the inverse of the sample frequency for each class, with an exponential smoothing coefficient α=0.5 applied to prevent excessive weight bias. For stable training, weights were normalized by the mean of all class weights. The loss function and weight definition are given in (20).

The Macro-F1 performance of the Swin Transformer V2 model across training steps is presented in Fig. 5 and Table 4. The validation Macro-F1 score converges rapidly in the early stages and peaks at 19,720 training steps.

Fig. 5.

Validation Macro-F1 score curve during training steps.

Table 4.

Validation Macro-F1 score by training steps

The experimental results show a rapid performance improvement from 0.1576 at the initial 3,944 steps to 0.6619 at 7,888 steps. Subsequently, the performance steadily increased, recording 0.7421 at 11,832 steps and 0.7705 at 15,776 steps, finally achieving a peak Macro-F1 of 0.7777 at 19,720 steps. This indicates that the model entered a stable convergence phase around the 0.77 level after approximately 15,776 steps. Furthermore, the final recognition performance for damaged character patches recorded a Top-1 Accuracy of 96.63% and a Top-5 Accuracy of 99.38%. Considering the difficulty of rubbing data mixed with complex physical damage and environmental degradation, the Top-5 accuracy of 99.38% demonstrates that the proposed model generates candidate groups containing the ground truth character with high reliability and stability.

This study utilized a total of 6,751 rubbing decipherment records (approximately 3.84 million characters) collected from four major institutions: the National Institute of Korean History, Korea Heritage Service, Kyujanggak Institute for Korean Studies, and National Research Institute of Cultural Heritage. To ensure data consistency, preprocessing steps such as normalizing unknown character markers (e.g., □), removing unnecessary annotation symbols, and excluding non-Hanja data (Hangul, Hiragana) were performed.

The average length of the collected rubbing texts is 693 characters, which significantly exceeds the maximum input length of 512 tokens for BERT-based models. To prevent information loss in long sequences and preserve the full context, the text was reconstructed using a SikuRoBERTa-based punctuation restoration model[32], which segments the unpunctuated Hanja text into logical units. Given a document D composed of a character sequence $$ D=\left\{c_0,c_1,\dots ,c_{L-1}\right\}$$, we applied a Sliding Window technique with window size W = 400, overlap O = 100, and stride S = 300. The i-th window segment input_i is defined as in (21).

The model M predicts the punctuation tag sequence for each window c_k. Since a character input_j within the overlap region is predicted by multiple windows, we collect the set of predictions P_k for that position as shown in (22). The final punctuation y_k is defined in (23) by applying majority voting to the collected set over the label set L, which consists of none, comma(,), period(。), question mark(?).

By segmenting sentences based on restored periods and question marks, the average sentence length was adjusted to approximately 25 characters, resulting in a dataset of 228,630 sentences. This distribution not only satisfies the BERT input constraints but also forms an optimal structure for efficient context learning.

To select the optimal base model capable of effectively reflecting the characteristics of the rubbing domain, we conducted comparative experiments on three models pre-trained on different corpora: SikuRoBERTa[23], HUE[33], and SillokBERT[34]. Notably, SillokBERT was trained on the AJD, and HUE utilized both the AJD and The Diaries of the Royal Secretariats (DRS). Both models share the commonality of being built on Korean Hanja corpora from the Joseon Dynasty. First, analyzing the number of Unknown Tokens to compare the vocabulary coverage of each model's tokenizer revealed that SikuRoBERTa recorded the lowest count of 1,903 tokens, indicating its superior suitability for processing rubbing characters(Table 5).

Table 5.

Number of unknown tokens by model

The results of evaluating MLM performance in a Zero-shot environment prior to fine-tuning are shown in Table 6. While HUE and SillokBERT showed high performance around 90% on the AJD dataset, their performance dropped sharply to the 40% range on rubbing data, revealing vulnerability to domain shift. In contrast, SikuRoBERTa recorded the highest Top-5 accuracy on rubbing texts among the comparison models.

Table 6.

Top-5 accuracy comparison across models (zero-shot)

To optimize the performance of SikuRoBERTa, we conducted stepwise comparative experiments setting Input Unit and Data Duplication as key variables. Input units were categorized into Document level (using the entire text) and Sentence level (based on punctuation). Detailed results are presented in Table 7. Document-level training (Exp 1, 2) showed minimal performance improvement due to truncation caused by the input limit (512 tokens). Conversely, when segmented into sentences with an optimized input length of 128 tokens and allowing data duplication (Exp 3), performance significantly improved to 76.05%. Furthermore, applying the sliding window technique for context preservation (Exp 4) achieved the optimal restoration performance with a Top-5 Accuracy of 82.56%. This improvement is attributed to the sliding window technique effectively mitigating the 'Boundary Effect'—where contextual continuity is severed during simple sentence segmentation—through window overlapping. Specifically, this approach ensures that tokens located at the segment boundaries are processed within sufficient bidirectional context. Furthermore, the majority voting mechanism applied to the redundantly predicted regions induces an ensemble effect that cancels out local prediction errors from individual windows, thereby significantly enhancing the reliability of the final restoration. These results demonstrate that the combination of length optimization and context reinforcement is critical for enhancing model performance.

Table 7.

Comparative analysis of model performance based on preprocessing strategies

To enable the model to sufficiently grasp the context surrounding the target sentence (S_tgt), the input sequence I was constructed by concatenating (⊕) the previous context (C_prev) and the next context (C_next). The final input vector configuration including special tokens ([CLS], [SEP]) is defined in (24).

Here, the total sequence length |I| is limited to a maximum of 128 tokens (|I| ≤ 128). The dataset consists of 228,630 sentences, randomly split into Training, Validation, and Test sets in an 8:1:1 ratio to ensure experimental reliability. The number of sentences in each dataset is shown in Table 8.

Table 8.

Strategies statistics of the rubbing dataset split

To process unique characters specific to the rubbing dataset, new tokens were added to the existing vocabulary, expanding the tokenizer to 31,885 tokens, and the model's embedding layer was reconstructed. The specific hyperparameter settings used for training are detailed in Table 9.

Table 9.

Hyperparameters for SikuRoBERTa training

The changes in training and validation loss throughout the learning process are illustrated in Fig. 6. While the initial loss exceeded 6.0, it exhibited a sharp decline, recording 2.3441 (train) and 2.3102 (validation) by the end of the first epoch. The losses continued to converge consistently, finally reaching 1.6065 and 1.5973 at step 114,320. As training progressed, the rate of loss reduction plateaued, and the gap between the two metrics remained narrow, confirming that the model was stably optimized without overfitting. The context inference performance of the final model is presented in Table 10. The experimental results show that the model achieved a Top-1 Accuracy of 68.22%, representing single prediction accuracy, and a Top-5 Accuracy of 82.56%, indicating the probability that the correct answer is included within the top 5 predicted candidates.

Fig. 6.

Context recognition results

Table 10.

Final performance results of the SikuRoBERTa model

To quantitatively verify the performance of the multimodal restoration framework, a multimodal simulation was conducted in a controlled environment where the ground truth is clearly defined. Specifically, a target character was selected from a pristine rubbing image, and an actual damaged region extracted from another rubbing was overlaid on that position to construct an artificially damaged environment. At this juncture, to minimize the discrepancy between the artificially induced damage and real-world conditions, we applied the degradation-based augmentation techniques detailed in Section IV-A. By rigorously simulating not only stroke disconnections but also the texture distortions and noise patterns characteristic of actual rubbings, we effectively compensated for the limitations of artificial manipulation and ensured the realism of the simulation data.

This approach allowed for the objective evaluation of the model's restoration performance while reflecting the actual stroke loss and texture distortion of rubbings, utilizing known ground truth characters. The overall workflow of the simulation is illustrated in Fig. 7.

Fig. 7.

Workflow of the integrated restoration simulation

The simulated damaged rubbing image was segmented into characters, and the same damaged position was fed into both the context-based and visual-based models. Each model output the top-5 restoration candidates along with their probability values for the damaged character. The comparison of the predictions confirmed the existence of common candidate characters. Subsequently, utilizing the F1-score-based probability fusion method adapted from previous studies like Wang et al. [14], the final confidence score integrating both contextual and stroke information was calculated.

As a result, both context-based and visual-based recognition predicted the correct character as the top candidate. The combined final probability was calculated at 82.4%, matching the actual ground truth character. This demonstrates that, unlike single-modality approaches, the multimodal approach—which combines complementary information based on damage types—operates effectively even in complex damage environments. Furthermore, these simulation results suggest that the proposed framework can provide stable character restoration performance in actual cultural heritage restoration scenarios.

To verify the performance of the translation prompts from multiple perspectives, quantitative evaluations were performed using Noun-focused BLEU to measure morphological consistency and BERTScore to evaluate contextual similarity. First, to measure the transmission of key information such as person names, place names, and official titles, nouns were extracted using the Kiwi morphological analyzer, and the Corpus BLEU was calculated. The result showed a high accuracy of 44.42. This indicates that the "explicit entity extraction" step within the prompt strategy effectively prevented the distortion or omission of key information during the translation process. Next, to verify the preservation of contextual meaning, BERTScore was calculated using mDeBERTa-v3-base and KLUE-BERT as backbones. BERTScore is calculated based on the cosine similarity between token embeddings of the source text and the generated text, as represented by the recall metric R_BERT in (25).

The final performance evaluation utilizes F_BERT, which is the harmonic mean of precision P_BERT and recall R_BERT, as defined in (26). This metric provides a balanced assessment of the generated translation by simultaneously accounting for both accuracy (precision) and completeness (recall) regarding the source text.

The evaluation resulted in high scores of 89.04 with mDeBERTa and 81.35 with KLUE-BERT. These figures indicate that the generated translations maintain a high level of structural and contextual meaning relative to the original text.