Home | JKIIT Archives | About the JKIIT | E-SUBMISSON |
Sorry.
You are not permitted to access the full text of articles.
If you have any questions about permissions,
please contact the Society.
죄송합니다.
회원님은 논문 이용 권한이 없습니다.
권한 관련 문의는 학회로 부탁 드립니다.
[ Article ] | |
The Journal of Korean Institute of Information Technology - Vol. 22, No. 11, pp. 39-45 | |
Abbreviation: Journal of KIIT | |
ISSN: 1598-8619 (Print) 2093-7571 (Online) | |
Print publication date 30 Nov 2024 | |
Received 27 Aug 2024 Revised 05 Nov 2024 Accepted 08 Nov 2024 | |
DOI: https://doi.org/10.14801/jkiit.2024.22.11.39 | |
Augmentation Strategies on Multi-Modal Lifelog Data for Smartphone Gait Authentication | |
Jiwoo Choi*
; Sangil Choi**
; Taewon Kang**
| |
*Employee, INTERFO Co., Ltd. | |
**Professor, Department of Computer Science & Engineering, Gangneung-Wonju National University | |
Correspondence to : Sangil Choi Dept. of Computer Science & Engineering, Gangneung-Wonju National University, Korea Tel.: +82-33-760-8670, Email: schoi@gwnu.ac.kr | |
Human gait patterns are promising for biometric authentication but have mainly been studied in controlled environments, lacking real-world variability. Our previous research on authenticating individuals through gait data used the ETRI Lifelog dataset, which includes various real-life activities, but faced data imbalance issues with the imbalance of gait data across activities such as walking and running. We applied data augmentation methods like noise addition and stretching to address this. This study refines and analyzes these methods to improve gait authentication models under realistic conditions. By optimizing permutation and noise augmentation, the Equal Error Rate(EER) was significantly reduced. The results show that combining multiple augmentation strategies can enhance model robustness and accuracy, achieving the lowest Equal Error Rate(EER) of 0.045905 when using all methods together.
사람의 걸음걸이 패턴은 생체 인증에 유망하지만, 주로 통제된 환경에서 연구되어 실제 환경의 변동성을 반영하지 못하였다. 걸음걸이 데이터를 통해 개개인을 인증하는 우리의 이전 연구에서 ETRI Lifelog 데이터셋을 사용했으나, 걷기와 달리기와 같은 활동 간의 데이터 불균형 문제가 있었다. 이를 해결하기 위해 노이즈 추가와 스트레칭 등의 데이터 증강 방법을 적용했다. 이번 연구는 이러한 방법을 개선하여 현실적인 조건에서 걸음걸이 인증 모델의 성능을 향상시키는 것을 목표로 한다. 순열과 잡음 추가 기법을 최적화함으로써 EER이 감소했으며, 그 결과, 여러 보강 기법을 결합하면 모델 견고성과 정확성을 향상시켜 모든 방법을 함께 사용할 때 가장 낮은 EER 수치인 0.045905를 달성할 수 있음을 보여준다.
Keywords: human gait, convolutional neural network, authentication, machine learning, robustness |
Biometric authentication using human gait patterns is rapidly evolving as a secure method for identity verification, driven by integrating smart devices with advanced sensing technologies[1][2]. Early research primarily focused on basic sensors like accelerometers and gyroscopes, but the advent of deep learning models, such as RNNs, CNNs, and LSTMs, has significantly improved gait recognition accuracy[3]-[9]. Among these, CNNs have been particularly successful, outperforming the other algorithms by over 10% when trained with feature maps[10]. However, most of these studies have been conducted in controlled environments, failing to capture real-world conditions' diversity and complexity[11]-[13]. Existing public datasets like RecodGait, OU-ISIR, and WISDM are limited by fixed sensor positions and specific experimental settings, which restrict their applicability in broader, real-world scenarios[7]-[18].
Our previous research, currently under review for journal publication, focused on developing a personal authentication system using gait data. We utilized the ETRI Lifelog dataset[19], which includes a wide range of real-life activities and environmental conditions, to create a model that accurately reflects real-world scenarios. This dataset is particularly valuable because it captures diverse, unconstrained activities collected over extended periods, reflecting a realistic variability often missing in controlled datasets. Using ETRI Lifelog data allows our model to be evaluated under practical conditions, enhancing the relevance and potential applicability of the results. However, we encountered data imbalance issues across activities such as walking and running, leading us to apply data augmentation techniques, including noise addition, data shift, and stretching.
This study optimizes and evaluates these augmentation methods to enhance model robustness in real-world conditions. We explore six data augmentation techniques—Noise, Smoothing, Horizontal Shift, Horizontal Stretch, Vertical Stretch, and Permutation—each selected to address specific challenges in gait recognition. Additionally, we investigate the synergistic effects of combining these methods to identify the most effective strategies for improving model performance, particularly in handling real-life gait data variations. This research represents a significant advancement in smartphone-based gait authentication, emphasizing theoretical insights and practical applications. The unique contributions of our current study are as follows: First, we developed advanced data augmentation techniques, such as noise addition and stretching, specifically optimized to address real-world challenges in gait recognition. Second, we implemented a synergistic augmentation strategy, analyzing how combining various data augmentation methods impacts authentication accuracy. Finally, our research demonstrates improved authentication model performance and practicality, providing valuable insights and directions for the future development of gait-based authentication systems.
Our research focuses on evaluating the applicability of biometric authentication systems in real-world environments, with a particular emphasis on individual gait patterns. This chapter describes the process of preparing and preprocessing real-world gait data, forming our authentication model's foundation.
We based our study on the ETRI Lifelog dataset[19], gathered by one of South Korea's premier research institutes. This rich dataset includes various real-life gait data from numerous participants over extended periods, providing comprehensive information such as Inertial Measurement Unit(IMU) data, GPS, heart rate, and environmental conditions. For our research, we specifically focused on accelerometer and gyroscope data from smartphones. Initially, the dataset included data from a larger group; however, due to the uncontrolled nature of real-world data in the ETRI dataset, with variations in user behavior and environmental factors, we observed inconsistencies across activity types. To ensure model reliability, we filtered the data to retain high-quality, usable walking and running data, which ultimately reduced the participant pool to around ten individuals. By concentrating on a smaller, controlled group of participants, we maintained the high data integrity essential for the reliability of our model.
Our data preprocessing strategy is a multifaceted approach that ensures the accuracy and reliability of our analysis. We start by cleaning the dataset to remove inconsistencies and anomalies, then use linear interpolation to fill gaps and maintain continuity. A Butterworth low-pass filter is applied to refine the dataset, focusing on gait-related features. Finally, we use a windowing method to segment the data, ensuring the effective analysis of continuous gait patterns. Please refer to [20] for a detailed description of these methods.
Our authentication model[20] is a Convolutional Neural Network(CNN) that analyzes one-dimensional time-series gait data. The model includes convolutional layers for extracting key gait features, pooling layers to reduce dimensionality and highlight essential features, dropout layers to prevent overfitting, and a dense layer with softmax activation for final classification.
This chapter examines the application of data augmentation methods to enhance gait authentication models. We compared the effects of six different augmentation methods as shown in Figure 1—Noise, Smoothing, Horizontal Shift, Horizontal Stretch, Vertical Stretch, and Permutation—each applied to the dataset with varying transformation rates ranging from 0% to 90%.
We aimed to identify optimal parameters for each method and determine the most effective combinations for improving authentication accuracy. We assessed the performance of each augmentation method by evaluating the Equal Error Rate(EER), a critical metric in biometric authentication that indicates the threshold where the rate of false acceptances equals the rate of false rejections. Our analysis involved optimizing transformation rates based on the mean, standard deviation, minimum, and maximum EER values. Additionally, we explored the synergistic effects of combining different augmentation methods to enhance the model's performance further. This comprehensive approach allowed us to identify the most effective strategies for dealing with real-world gait data variations and improving the robustness of the authentication model.
Let us look at Figure 2, which shows the EER results for each augmentation method.
Noise addition at a 20% transformation rate effectively introduced variability, enhancing model generalization. Smoothing, using a 30% transformation rate, balanced data refinement while preserving key gait information. Horizontal Shift, with a 60% rate, simulated timing discrepancies and yielded peak performance. Horizontal and Vertical Stretching, at 50% and 10% rates respectively, improved the model’s ability to handle variations in walking speed and step force. Permutation, with a 40% transformation rate, demonstrated the model’s robustness by maintaining accuracy even with shuffled data segments.
Our analysis showed that data augmentation significantly enhanced the base model’s performance, reducing the EER from 0.101196 to 0.057671 with permutation and 0.059469 with noise addition, as shown in Figure 3. Each augmentation method impacted performance differently: permutation and noise addition had the most significant improvements, while horizontal stretching, smoothing, and vertical stretching also contributed but to a lesser extent. This highlights how individual methods address specific variations in gait data, improving model robustness in different ways.
We explored the effectiveness of combining different augmentation methods at their optimal transformation rates to improve the gait authentication model, as illustrated in Figure 4. Methods were combined in pairs (r=2), trios (r=3), quartets (r=4), quintets (r=5), and all six methods (r=6), with the mean EER calculated for each combination. The combination of noise and permutation (r=2) yielded the lowest mean EER of 0.056667, indicating that these methods complement each other well. In contrast, combinations involving Horizontal Shift generally resulted in higher EERs, suggesting less effective synergy. Increasing the number of combined methods further reduced the EER, with the trio of noise, horizontal stretch, and permutation achieving a mean EER of 0.053174, and the quartet of noise, horizontal stretch, vertical stretch, and permutation lowering it to 0.051977. The quintet reduced the EER even further to 0.049956.
The lowest mean EER of 0.045905 was achieved by combining all six augmentation methods, demonstrating that a comprehensive approach significantly enhances the model’s ability to generalize from complex datasets, as shown in Figure 4.
This chapter delves into the practical implications of data augmentation methods for enhancing the gait-based authentication model.
The findings reveal that proper calibration and combination can significantly boost accuracy, with the combination of permutation, noise, and horizontal stretch proving particularly effective. However, while the use of all six augmentation methods led to the lowest mean EER of 0.045905, the practical implementation in real-world applications poses challenges. Specifically, the computational demands of processing multiple augmentations in parallel may strain devices with limited hardware, potentially slowing down the authentication process. Balancing accuracy with operational efficiency remains a key challenge for real-time applications.
This study examined various data augmentation methods and their combinations to enhance gait-based authentication systems. We discovered that specific combinations significantly improved model performance, especially permutation, noise addition, and horizontal stretching. The use of each method individually improved the model's accuracy at optimal transformation rates. Combining them consistently reduced the EER, with the lowest mean EER achieved by combining all six methods. Our research advances the robustness of gait-based authentication models. However, practical implementation challenges, such as computational complexity, hardware limitations and potential delays, must be addressed. Future research could explore optimized algorithms or lightweight augmentation techniques to reduce computational load, especially for mobile devices.
1. | M. Abuhamad, T. Abuhmed, D. Mohaisen, and D. Nyang, "AUToSen: Deep-Learning-Based Implicit Continuous Authentication Using Smartphone Sensors", IEEE Internet of Things Journal, Vol. 7, No. 6, pp. 5008-5020, Jun. 2020. |
2. | Y. Watanabe and M. Kimura, "Gait identification and authentication using LSTM based on 3-axis accelerations of smartphone", Procedia Computer Science, Vol. 176, pp. 3873-3880, 2020. |
3. | Y. Hutabarat, D. Owaki, and M. Hayashibe, "Recent Advances in Quantitative Gait Analysis Using Wearable Sensors: A Review", IEEE Sensors Journal, Vol. 21, No. 23, pp. 26470-26487, Dec. 2021. |
4. | P. Connor and A. Ross, "Biometric recognition by gait: A survey of modalities and features", Computer Vision and Image Understanding, Vol. 167, pp. 1-27, Feb. 2018. |
5. | J. P. Singh, S. Jain, S. Arora, and U. P. Singh, "Vision-Based Gait Recognition: A Survey", IEEE Access, Vol. 6, pp. 70497-70527, Nov. 2018. |
6. | I. Bouchrika, "A Survey of Using Biometrics for Smart Visual Surveillance: Gait Recognition", in Surveillance in Action, Springer, pp 3-23, Nov. 2017. |
7. | S. Sprager and M. B. Juric, "Inertial Sensor-Based Gait Recognition: A Review", Sensors 2015, Vol. 15, No. 9, pp. 22089-22127, Sep. 2015. |
8. | F. Sun, C. Mao, X. Fan, and Y. Li, "Accelerometer-Based Speed-Adaptive Gait Authentication Method for Wearable IoT Devices", IEEE Internet of Things Journal, Vol. 6, No. 1, pp. 820-830, Feb. 2019. |
9. | A. R. Kothamachu and B. Chakraborty, "Real Time Gait based Person Authentication using Deep Hybrid Network", 2021 IEEE 4th international Conference on Knowledge Innovation and Invention (ICKII), Taichung, Taiwan, pp. 155-159, Jul. 2021. |
10. | M. Gadaleta and M. Rossi, "IDNet: Smartphone-based gait recognition with convolutional neural networks", Pattern Recognition, Vol. 74, pp. 25-37, Feb. 2018. |
11. | S. R. V. Sudhakar, N. Kayastha, and K. Sha, "ActID: An efficient framework for activity sensor based user identification", Computers & Security, Vol. 108, Sep. 2021. |
12. | N. Kala, T. Bhatia, and N. Aggarwal, "Person Identification and Characterization from Gait Using Smartphone", 2019 11th International Conference on Communication Systems & Networks (COMSNETS), Bengaluru, India, pp. 492-495, Jan. 2019. |
13. | W. Xu, Y. Shen, C. Luo, J. Li, W. Li, and A. Y. Zomaya, "Gait-Watch: A Gait-based context-aware authentication system for smart watch via sparse coding", Ad Hoc Networks, Vol. 107, Oct. 2020. |
14. | S. Adhikary, S. Biswas, A. Ghosh, and S. Nandi, "TinyBioGait—Embedded intelligence and homologous time approximation warping for gait biometric authentication from IMU signals", Smart Health, Vol. 34, Dec. 2024. |
15. | K. Fukamachi, R. Narita, S. Sato, and T. Kawanami, "Proposal and Evaluation of a Personal Authentication Method Using Insole-Type Gait Sensors", 2024 International Conference on Activity and Behavior Computing (ABC), Oita / Kitakyushu, Japan, pp. 1-7, May 2024. |
16. | L. Tran, T. Hoang, T. Nguyen, H. Kim, and D. Choi, "Multi-Model Long Short-Term Memory Network for Gait Recognition Using Window-Based Data Segment", IEEE Access, Vol. 9, pp. 23826-23839, Feb. 2021. |
17. | G. Giorgi, A. Saracino, and F. Martinelli, "Using recurrent neural networks for continuous authentication through gait analysis", Pattern Recognition Letters, Vol. 147, pp. 157-163, Jul. 2021. |
18. | P. Zouridakis and S. M. P. Dinakarrao, "Performance- and Energy-Aware Gait-Based User Authentication With Intermittent Computation for IoT Devices", IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 43, No. 2, pp. 600-612, Feb. 2024. |
19. | S. Chung, C. Y. Jeong, J. M. Lim, J. Lim, K. J. Noh, G. Kim, and H. Jeong, "Real-world multimodal lifelog dataset for human behavior study", ETRI Journal, Vol. 43, No. 6, pp. 426-437, Jun. 2021. |
20. | J. Choi, S. Choi, and T. Kang, "Identification of Gait Patterns using Convolutional Neural Networks for Personal Authentication", Journal of Korean Institute of Information Technology (JKIIT), Vol. 20, No. 4, pp. 13-23, Apr. 2022. |
2023. 2 : BS degree, Department of Computer Science & Engineering, Gananeung-Wonju National University
2023. 2 ~ Present : Employee, INTERFO Co., Ltd.
Research interests: Artficial Intelligence
2000. 2 : BS degree, Department of Computer Science & Engineering, Gangneung-Wonju National University
2008. 8 : MS degree, Department of Computer Science, Iowa State University
2015. 5 : PhD degree, Department of Computer Engineering, University of Nebraska
2015. 8 ~ 2016. 2 : Assistant Professor, Department of Computer Science & Engineering, Swaziland Christian University
2016. 8 ~ 2019. 2 : Full-time Lecturer, Department of Software, Ajou University
2019. 3 ~ Present : Professor, Department of Computer Science & Engineering, Gangneung-Wonju National University
Research interests : Internet of Things, Machine Learning and Deep Learning, Gait Analysis
1985. 2 : BS degree, Department of Mathematics, Yonsei University
1988. 2 : BS degree, Department of Computer Science & Engineering, Korea University
1991. 2 : MS degree, Department of Mathematics, Korea University
1996. 8 : PhD degree, Department of Computer Science & Engineering, Korea University
1997. 3 ~ Present : Professor, Department of Computer Science & Engineering, Gangneung-Wonju National University
Research interests : Complex Systems, Artificial Life, Artificial Intelligence, Soft Computing