Augmentation Strategies on Multi-Modal Lifelog Data for Smartphone Gait Authentication

Ⅰ. Introduction

Biometric authentication using human gait patterns is rapidly evolving as a secure method for identity verification, driven by integrating smart devices with advanced sensing technologies[1][2]. Early research primarily focused on basic sensors like accelerometers and gyroscopes, but the advent of deep learning models, such as RNNs, CNNs, and LSTMs, has significantly improved gait recognition accuracy[3]-[9]. Among these, CNNs have been particularly successful, outperforming the other algorithms by over 10% when trained with feature maps[10]. However, most of these studies have been conducted in controlled environments, failing to capture real-world conditions' diversity and complexity[11]-[13]. Existing public datasets like RecodGait, OU-ISIR, and WISDM are limited by fixed sensor positions and specific experimental settings, which restrict their applicability in broader, real-world scenarios[7]-[18].

Our previous research, currently under review for journal publication, focused on developing a personal authentication system using gait data. We utilized the ETRI Lifelog dataset[19], which includes a wide range of real-life activities and environmental conditions, to create a model that accurately reflects real-world scenarios. This dataset is particularly valuable because it captures diverse, unconstrained activities collected over extended periods, reflecting a realistic variability often missing in controlled datasets. Using ETRI Lifelog data allows our model to be evaluated under practical conditions, enhancing the relevance and potential applicability of the results. However, we encountered data imbalance issues across activities such as walking and running, leading us to apply data augmentation techniques, including noise addition, data shift, and stretching.

This study optimizes and evaluates these augmentation methods to enhance model robustness in real-world conditions. We explore six data augmentation techniques—Noise, Smoothing, Horizontal Shift, Horizontal Stretch, Vertical Stretch, and Permutation—each selected to address specific challenges in gait recognition. Additionally, we investigate the synergistic effects of combining these methods to identify the most effective strategies for improving model performance, particularly in handling real-life gait data variations. This research represents a significant advancement in smartphone-based gait authentication, emphasizing theoretical insights and practical applications. The unique contributions of our current study are as follows: First, we developed advanced data augmentation techniques, such as noise addition and stretching, specifically optimized to address real-world challenges in gait recognition. Second, we implemented a synergistic augmentation strategy, analyzing how combining various data augmentation methods impacts authentication accuracy. Finally, our research demonstrates improved authentication model performance and practicality, providing valuable insights and directions for the future development of gait-based authentication systems.

Ⅱ. Data and Authentication Model

Our research focuses on evaluating the applicability of biometric authentication systems in real-world environments, with a particular emphasis on individual gait patterns. This chapter describes the process of preparing and preprocessing real-world gait data, forming our authentication model's foundation.

We based our study on the ETRI Lifelog dataset[19], gathered by one of South Korea's premier research institutes. This rich dataset includes various real-life gait data from numerous participants over extended periods, providing comprehensive information such as Inertial Measurement Unit(IMU) data, GPS, heart rate, and environmental conditions. For our research, we specifically focused on accelerometer and gyroscope data from smartphones. Initially, the dataset included data from a larger group; however, due to the uncontrolled nature of real-world data in the ETRI dataset, with variations in user behavior and environmental factors, we observed inconsistencies across activity types. To ensure model reliability, we filtered the data to retain high-quality, usable walking and running data, which ultimately reduced the participant pool to around ten individuals. By concentrating on a smaller, controlled group of participants, we maintained the high data integrity essential for the reliability of our model.

Our data preprocessing strategy is a multifaceted approach that ensures the accuracy and reliability of our analysis. We start by cleaning the dataset to remove inconsistencies and anomalies, then use linear interpolation to fill gaps and maintain continuity. A Butterworth low-pass filter is applied to refine the dataset, focusing on gait-related features. Finally, we use a windowing method to segment the data, ensuring the effective analysis of continuous gait patterns. Please refer to [20] for a detailed description of these methods.

Our authentication model[20] is a Convolutional Neural Network(CNN) that analyzes one-dimensional time-series gait data. The model includes convolutional layers for extracting key gait features, pooling layers to reduce dimensionality and highlight essential features, dropout layers to prevent overfitting, and a dense layer with softmax activation for final classification.

Ⅲ. Data Augmentation and Result Analysis

This chapter examines the application of data augmentation methods to enhance gait authentication models. We compared the effects of six different augmentation methods as shown in Figure 1—Noise, Smoothing, Horizontal Shift, Horizontal Stretch, Vertical Stretch, and Permutation—each applied to the dataset with varying transformation rates ranging from 0% to 90%.

Fig. 1.

Augmentation methods used in our study

We aimed to identify optimal parameters for each method and determine the most effective combinations for improving authentication accuracy. We assessed the performance of each augmentation method by evaluating the Equal Error Rate(EER), a critical metric in biometric authentication that indicates the threshold where the rate of false acceptances equals the rate of false rejections. Our analysis involved optimizing transformation rates based on the mean, standard deviation, minimum, and maximum EER values. Additionally, we explored the synergistic effects of combining different augmentation methods to enhance the model's performance further. This comprehensive approach allowed us to identify the most effective strategies for dealing with real-world gait data variations and improving the robustness of the authentication model.

3.1 Augmentation methods and results

Let us look at Figure 2, which shows the EER results for each augmentation method.

Fig. 2.

Median EER trends for each augmentation method

Noise addition at a 20% transformation rate effectively introduced variability, enhancing model generalization. Smoothing, using a 30% transformation rate, balanced data refinement while preserving key gait information. Horizontal Shift, with a 60% rate, simulated timing discrepancies and yielded peak performance. Horizontal and Vertical Stretching, at 50% and 10% rates respectively, improved the model’s ability to handle variations in walking speed and step force. Permutation, with a 40% transformation rate, demonstrated the model’s robustness by maintaining accuracy even with shuffled data segments.

Our analysis showed that data augmentation significantly enhanced the base model’s performance, reducing the EER from 0.101196 to 0.057671 with permutation and 0.059469 with noise addition, as shown in Figure 3. Each augmentation method impacted performance differently: permutation and noise addition had the most significant improvements, while horizontal stretching, smoothing, and vertical stretching also contributed but to a lesser extent. This highlights how individual methods address specific variations in gait data, improving model robustness in different ways.

Fig. 3.

EER for each of the augmentation methods

3.2 Combining augmentation methods and results

We explored the effectiveness of combining different augmentation methods at their optimal transformation rates to improve the gait authentication model, as illustrated in Figure 4. Methods were combined in pairs (r=2), trios (r=3), quartets (r=4), quintets (r=5), and all six methods (r=6), with the mean EER calculated for each combination. The combination of noise and permutation (r=2) yielded the lowest mean EER of 0.056667, indicating that these methods complement each other well. In contrast, combinations involving Horizontal Shift generally resulted in higher EERs, suggesting less effective synergy. Increasing the number of combined methods further reduced the EER, with the trio of noise, horizontal stretch, and permutation achieving a mean EER of 0.053174, and the quartet of noise, horizontal stretch, vertical stretch, and permutation lowering it to 0.051977. The quintet reduced the EER even further to 0.049956.

Fig. 4.

EER for different augmentation method combinations

The lowest mean EER of 0.045905 was achieved by combining all six augmentation methods, demonstrating that a comprehensive approach significantly enhances the model’s ability to generalize from complex datasets, as shown in Figure 4.

Ⅳ. Conclusion

This study examined various data augmentation methods and their combinations to enhance gait-based authentication systems. We discovered that specific combinations significantly improved model performance, especially permutation, noise addition, and horizontal stretching. The use of each method individually improved the model's accuracy at optimal transformation rates. Combining them consistently reduced the EER, with the lowest mean EER achieved by combining all six methods. Our research advances the robustness of gait-based authentication models. However, practical implementation challenges, such as computational complexity, hardware limitations and potential delays, must be addressed. Future research could explore optimized algorithms or lightweight augmentation techniques to reduce computational load, especially for mobile devices.

References

• M. Abuhamad, T. Abuhmed, D. Mohaisen, and D. Nyang, "AUToSen: Deep-Learning-Based Implicit Continuous Authentication Using Smartphone Sensors", IEEE Internet of Things Journal, Vol. 7, No. 6, pp. 5008-5020, Jun. 2020. [https://doi.org/10.1109/JIOT.2020.2975779]
• Y. Watanabe and M. Kimura, "Gait identification and authentication using LSTM based on 3-axis accelerations of smartphone", Procedia Computer Science, Vol. 176, pp. 3873-3880, 2020. [https://doi.org/10.1016/j.procs.2020.09.001]
• Y. Hutabarat, D. Owaki, and M. Hayashibe, "Recent Advances in Quantitative Gait Analysis Using Wearable Sensors: A Review", IEEE Sensors Journal, Vol. 21, No. 23, pp. 26470-26487, Dec. 2021. [https://doi.org/10.1109/JSEN.2021.3119658]
• P. Connor and A. Ross, "Biometric recognition by gait: A survey of modalities and features", Computer Vision and Image Understanding, Vol. 167, pp. 1-27, Feb. 2018. [https://doi.org/10.1016/j.cviu.2018.01.007]
• J. P. Singh, S. Jain, S. Arora, and U. P. Singh, "Vision-Based Gait Recognition: A Survey", IEEE Access, Vol. 6, pp. 70497-70527, Nov. 2018. [https://doi.org/10.1109/ACCESS.2018.2879896]
• I. Bouchrika, "A Survey of Using Biometrics for Smart Visual Surveillance: Gait Recognition", in Surveillance in Action, Springer, pp 3-23, Nov. 2017. [https://doi.org/10.1007/978-3-319-68533-5_1]
• S. Sprager and M. B. Juric, "Inertial Sensor-Based Gait Recognition: A Review", Sensors 2015, Vol. 15, No. 9, pp. 22089-22127, Sep. 2015. [https://doi.org/10.3390/s150922089]
• F. Sun, C. Mao, X. Fan, and Y. Li, "Accelerometer-Based Speed-Adaptive Gait Authentication Method for Wearable IoT Devices", IEEE Internet of Things Journal, Vol. 6, No. 1, pp. 820-830, Feb. 2019. [https://doi.org/10.1109/JIOT.2018.2860592]
• A. R. Kothamachu and B. Chakraborty, "Real Time Gait based Person Authentication using Deep Hybrid Network", 2021 IEEE 4th international Conference on Knowledge Innovation and Invention (ICKII), Taichung, Taiwan, pp. 155-159, Jul. 2021. [https://doi.org/10.1109/ICKII51822.2021.9574763]
• M. Gadaleta and M. Rossi, "IDNet: Smartphone-based gait recognition with convolutional neural networks", Pattern Recognition, Vol. 74, pp. 25-37, Feb. 2018. [https://doi.org/10.1016/j.patcog.2017.09.005]
• S. R. V. Sudhakar, N. Kayastha, and K. Sha, "ActID: An efficient framework for activity sensor based user identification", Computers & Security, Vol. 108, Sep. 2021. [https://doi.org/10.1016/j.cose.2021.102319]
• N. Kala, T. Bhatia, and N. Aggarwal, "Person Identification and Characterization from Gait Using Smartphone", 2019 11th International Conference on Communication Systems & Networks (COMSNETS), Bengaluru, India, pp. 492-495, Jan. 2019. [https://doi.org/10.1109/COMSNETS.2019.8711131]
• W. Xu, Y. Shen, C. Luo, J. Li, W. Li, and A. Y. Zomaya, "Gait-Watch: A Gait-based context-aware authentication system for smart watch via sparse coding", Ad Hoc Networks, Vol. 107, Oct. 2020. [https://doi.org/10.1016/j.adhoc.2020.102218]
• S. Adhikary, S. Biswas, A. Ghosh, and S. Nandi, "TinyBioGait—Embedded intelligence and homologous time approximation warping for gait biometric authentication from IMU signals", Smart Health, Vol. 34, Dec. 2024. [https://doi.org/10.1016/j.smhl.2024.100515]
• K. Fukamachi, R. Narita, S. Sato, and T. Kawanami, "Proposal and Evaluation of a Personal Authentication Method Using Insole-Type Gait Sensors", 2024 International Conference on Activity and Behavior Computing (ABC), Oita / Kitakyushu, Japan, pp. 1-7, May 2024. [https://doi.org/10.1109/ABC61795.2024.10651837]
• L. Tran, T. Hoang, T. Nguyen, H. Kim, and D. Choi, "Multi-Model Long Short-Term Memory Network for Gait Recognition Using Window-Based Data Segment", IEEE Access, Vol. 9, pp. 23826-23839, Feb. 2021. [https://doi.org/10.1109/ACCESS.2021.3056880]
• G. Giorgi, A. Saracino, and F. Martinelli, "Using recurrent neural networks for continuous authentication through gait analysis", Pattern Recognition Letters, Vol. 147, pp. 157-163, Jul. 2021. [https://doi.org/10.1016/j.patrec.2021.03.010]
• P. Zouridakis and S. M. P. Dinakarrao, "Performance- and Energy-Aware Gait-Based User Authentication With Intermittent Computation for IoT Devices", IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 43, No. 2, pp. 600-612, Feb. 2024. [https://doi.org/10.1109/TCAD.2023.3313097]
• S. Chung, C. Y. Jeong, J. M. Lim, J. Lim, K. J. Noh, G. Kim, and H. Jeong, "Real-world multimodal lifelog dataset for human behavior study", ETRI Journal, Vol. 43, No. 6, pp. 426-437, Jun. 2021. [https://doi.org/10.4218/etrij.2020-0446]
• J. Choi, S. Choi, and T. Kang, "Identification of Gait Patterns using Convolutional Neural Networks for Personal Authentication", Journal of Korean Institute of Information Technology (JKIIT), Vol. 20, No. 4, pp. 13-23, Apr. 2022. [https://doi.org/10.14801/jkiit.2022.20.4.13]