Korean Institute of Information Technology

Current Issue

The Journal of Korean Institute of Information Technology - Vol. 19, No. 8

[ Article ]
The Journal of Korean Institute of Information Technology - Vol. 19, No. 6, pp.21-28
Abbreviation: Journal of KIIT
ISSN: 1598-8619 (Print) 2093-7571 (Online)
Print publication date 30 Jun 2021
Received 08 Apr 2021 Revised 07 Jun 2021 Accepted 10 Jun 2021
DOI: https://doi.org/10.14801/jkiit.2021.19.6.21

A Study on the Two-Dimensional Graph Data and Its Effectiveness in Human Behavior Classification Deep Learning
Seo-Yung Lee* ; Kwangmin Hyun** ; Baek-Ki Kim** ; Hwan-Seog Kim**
*KCNET Intelligence Information Division
**Information & Communication Engineering, Gangneung Wonju National University

Correspondence to : Seo-Yung Lee Dept. of AI R&D, Ehost Information & Communication Technology, Korea Tel. +82-31-969-4574, Email: wdragon277@gmail.com

Funding Information ▼


Recently, a lot of research has been conducted using machine learning technology in various fields. It also develops models by applying various learning techniques. We propose a method to develop a CNN model using a two-dimensional graph image, focusing on learning using CNN to process time series information faster than LSTM. Process the data to create a two-dimensional graph and use it for training AI models. It collects data on 4 specific motions with a 6-axis sensor (Arduino nano 33 ble) and creates a two-dimensional graph. The result of testing the model trained in this study is 99.6% accuracy. This was able to achieve higher accuracy than other studies that did not image the data. The difference from the time series data training model using other CNNs is that you can analyze all items in the graph from one kernel. Conventional methods make it difficult to convolution all items into one kernel. Additionally, through experiments with various parameters, inferences between parameters and results and optimal settings are suggested.


최근에 다양한 분야에서 머신 러닝 기술을 활용하는 많은 연구가 진행되고 있다. 또한 다양한 학습 기술을 적용하여 모델을 개발한다. CNN을 이용한 학습이 LSTM보다 빠르게 시계열 정보를 처리하는데 착안하여, 2 차원 그래프 이미지를 활용한 CNN 모델을 개발하는 방법을 제안한다. 데이터를 가공하여 2차원 그래프를 만들고 AI 모델 학습에 사용한다. 6 축 센서 (Arduino nano 33 ble)로 4개의 특정 동작에 대한 데이터를 수집하고 2차원 그래프를 만든다. 이 연구에서 훈련된 모델을 테스트한 결과는 99.6%의 정확도를 가진다. 이것은 데이터를 이미지화하지 않은 다른 연구보다 높은 정확도를 구현할 수 있었습니다. 다른 CNN을 이용한 시계열 데이터 학습모델과 차이점은 하나의 커널에서 그래프의 모든 항목을 분석할 수 있다는 것이다. 기존의 방법은 모든 항목을 하나의 커널에 합성곱하기 어렵다. 추가적으로 다양한 매개 변수를 사용한 실험을 통해 매개 변수와 결과 사이의 추론과 최적 설정값을 제안한다.

Keywords: CNN, image, deep learning, human behavior recognition, machine learning

Ⅰ. Introduction

Recently, the development of real-life technology using AI has received much attention from researchers as well as from all over the country. Advanced AI algorithms, computing resources, and rapidly increasing storage of data are being used to solve various problems across industries. In particular high entry barriers to artificial intelligence development have been greatly lowered through various examples and intuitive concepts of CNN. Accordingly, discussions are also taking place on how to utilize data efficiently. This is because there is a way to achieve faster and more accurate performance even when using the same data.

In this paper, I develop a model for classifying human behavior with CNN learning with data collected using 6-axis sensors for human multi-motions. Time series data collected via a six-axis sensor is stored in two-dimensional graphs. We develop a model using CNN for this image. As a result, we were able to predict the results more intuitively than other CNNs, and we confirm that we can also develop an effective model with two-dimensional graphs. In addition, learning in a direct input of data has solved the problem of not being able to synthesize all node values into one kernel. Through this study, it is expected that it can be used using a number of two-dimensional graph images that have been released and used in the healthcare field. Data such as heartbeat, ECG, and oxygen saturation are being graphed.

I expect this to be effective in detecting anomalies, such as detecting abnormal heart reactions. Graphs are also used for monitoring in areas such as industrial safety and water quality/atmospheric management, where people require a lot of concentration and know-how to discover abnormal symptoms. Therefore, it will be easy to learn AI models using two-dimensional graphs, and it will help build multiple safety nets because it complements and informs people of what they may miss. In this paper, I test the performance of our model using two-dimensional graph data as CNNs. The advantage of this is that, unlike GNN or GCN, the model can be developed using simpler examples.

Ⅱ. Related Work

There are various studies on human movement, and the sensor is collected at a specific point to classify the movement[1][2]. Related studies include Saiskul's attempt to classify behavior using SVM[3][4]. In this work, they proceed with neural network learning by collecting tri-axial data. We can see that accuracy increases as the type of input data increases. In a study conducted. [5], time series data was learned using LSTM (Long short term memory). LSTM feature long-term memory of data and re-use it for learning. The more repeated the number of lessons, the more accurate the convergence to 1 and the loss to 0. Using this, research on behavioral classification is being conducted. However, 200 million studies were needed to stabilize the system[6].

LSTM performs serial operations that require much more resources than CNN[7]. Therefore, it is inevitably slower and dull than CNN. Since the model to be developed in this study requires fast learning that can achieve high accuracy even with edge computing, CNN was adopted as a more suitable algorithm learning method. The corporation of PUSH in the United States used CNN to classify human movements[8]. Nine- axial data were utilized, with accuracy reaching 92.14%. It was also characterized by the general level of IT resources used. They learned from imaging 784 normalized data over 3.92 seconds[Fig. 1]. After attaching a sensor to a specific body part for 50 exercises, the collected data is used. It has high accuracy and monitoring effect enough to provide services to professional athletes. A similar study was conducted by Recofit[9].

Fig. 1. 
CNN kernel progress conducted by PUSH

In this process, if the stride is greater than 3, there is a problem that the data does not synthesize at all.[Fig. 2]

Fig. 2. 
CNN input node conducted by PUSH

Ⅲ. Classify Human Activitys

In this paper, we study how to learn more intuitively and accurately human behavior-aware artificial intelligence models developed in various ways. In that way, learning was carried out using two-dimensional graphs. We introduced a system to develop a model that can recognize human behavior using 6-axis sensors used in this study.

After device development for data collection, data is collected through the subject's movement. We then apply the data to two-dimensional graphs to generate images. The generated images are classified as training and verification data and model learning using CNN is conducted. The final step is the process of classifying and feed backing new data using developed models. In this study, only the verification of the model was carried out using the verification data. Finally, four behaviors were classified, with accuracy reaching 99%.

3.1 Training Data Preparation

Data were collected by specifying the behavior to be measured and positioning the sensor to be attached.

The collected data was normalized via maximum-minimum normalization, and the normalized data was soon converted to a two-dimensional graph image. The data are prepared by dividing them into learning and validation.

Fig. 3. 
Motion classification model development workflow

3.1.1. Measurement posture definition and sensor attachment location

Fig. 4 shows the bird dog position. Attach the sensor to the right ankle with a lot of movement. Fig. 5 shows a straightening waist. Attach the sensor behind the back.

Fig. 4. 

Fig. 5. 
Traightening waist

Fig. 6 shows the bridge posture. Attach the sensor to the nameplate. Fig. 7 shows the cobra position. The sensor is attached behind the most dynamic back.

Fig. 6. 
Bridge posture and sensor

Fig. 7. 
Cobra posture and sensor

3.2. Data collect and preprocessing

These are two random graphs with a split waist stretch in one time.[Fig. 8, 9] The graph shows tendency, but some graphs show significantly different shapes. This can be handled or used as an outlier during CNN learning. If used as it is, many results can be covered due to the nature of the body that cannot be accurately repeated.

Fig. 8. 
Random split graph 1

Fig. 9. 
Random split graph 2

Ⅳ. Experiment Result

Table 1 is a detailed description of the conditions of the experiments using Optimizer Adam. Various experimental environments were set using the hyper-parameters Kernel, Learning Rate, and Input nodes. Kernel used two things: 3*3 and 5*5 and a learning rate of 0.005,0.0005.

Table 1. 
Experimental results with Adam optimizer
No Acc Val_Acc Epochs Max time(s)/Epoch
1 0.3563 0.3538 - 12
2 0.3425 0.3596 - 12
3 1 0.9962 15 12
4 1 0.9904 15 12
5 0.3506 0.3577 - 13
6 0.3420 0.3558 - 12
7 1 0.9923 25 12
8 1 0.9827 30 12

The number of input nodes has been used by various experimental values to find the minimum value. If the number of nodes became extremely small, it would not be learned, so the appropriate value had to be found. We can see that 100*100 is the smallest number of nodes that do not degrade performance. Epoch was equally executed 100 times.

Table 2 is the result of an experiment using Adam. Accuracy and verification accuracy are entered for each setting. Furthermore, since the model with the shortest time to optimization is an effective model, we found the model with the smallest number of epochs needed to optimization. And the time taken per epoch was recorded. The time taken per epoch did not vary much. However, it was not learned in four of the eight settings. This was deeply related to the Learning Rate. The learning was not completed only in the setting with 0.005. The closest verification accuracy to 1 and the least epoch and the maximum time spent per epoch were set three times. Model 7 was successful in optimization and the number of epochs to optimization was 25. The time spent per epoch was generally similar and was not significant. More epochs will have a greater impact.

Ⅴ. Conclusion

Learning using two-dimensional graphs will allow for tight implementation of composite products between data items. And this contributes to accurate model learning. We also run experiments considering various hyper-parameters, identifying the variables that can learn the model the fastest and most accurately. Furthermore, we investigated the effect of parameters on the result value and the rate at which they produce results, and found that Learning Rate had the greatest impact on learning. If the Learning Rate is too high, learning is not done properly. Kernel's value affected the time it took to stabilize learning. It showed better performance at 3*3 than 5*5. This may be due to more gaps affecting the composite product when using kernel of 5*5 on the graph.

The technique of learning graphs GNN is too complex to apply to two-dimensional graphs. Two-dimensional graphs will be more productive in the direction of generating images and learning using direct CNNs. It is a suitable method for systems that monitor a large number of less correlated items. A number of items can be used by learning to show a certain pattern in a particular situation. This can be used as a reminder and a record of a particular situation. It can also be used by learning ordinary situations and recognizing and classifying special situations.

Unlike the Black Box, which is commonly used for artificial intelligence, researchers or managers have the opportunity to learn based on results. Manager's judgment and quickness can be applied in the industrial safety and disaster sectors where it is important.

The task of creating new images for learning using two-dimensional graphs can be a factor in delaying the speed of model development. However, two-dimensional graphs are often used for monitoring and feedback on smart healthcare, exercise, and indoor activities. Given that many products already provide two-dimensional graphs for visualization, the above problem can be solved using existing data. Therefore, studies applying AI to developed services are expected to be appropriate. If continuous observation is applied to plants such as hazardous materials monitoring, wastewater treatment system, and fine dust observation system, AI is expected to be able to warn people of abnormal reactions first before confirming them. In addition, learning about normal situations can be conducted to detect small outliers that a person might miss and to send warnings to managers. However, overfitting becomes a major obstacle to this study, so it is recommended to prepare countermeasures to proceed with the study. This study has a limitation that there were fewer samples of data. To address this, it is recommended to utilize public data or two-dimensional graphs deposited in plants


This research was supported by X-mind Corps program of National Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT(NRF-2017H1D8A1029391)

Thank you to researcher Park Chae-eun for her hard work for the data. I will remember everyone's help and grow into a more humble and confident educated citizen.

1. FahriddinMarnatov, Min Gyu Song, Jin Young Kim, and Seung You Na, "Human Activity Recognition Using New Multi-Sensor Module in Mobile Environment", Korean Institute of Information Technology, Vol. 9, No. 5, pp. 167-174, Mar. 2016.
2. Jongwook Si, Hyeoncheol Son, Daseul Kim, Moonnyeon Kim, Jiyeon Jeong, Gyuree Kim, Younghyung Kim, and Sungyoung Kim, "Fall Detection using Skeletal Coordinate Vector and LSTM Model" Journal of KIIT, Vol. 18, No. 12, pp. 19-29, Dec. 2020.
3. Saisakul Chernbumroong, Shuang Cang, Anthony Atkins, and Hongnian Yu, "Elderly activities recognition and classification for applications in assisted living", Expert Systems with Applications, Vol. 40, pp. 1662-1674, Oct. 2012.
4. Saisakul Chernbumroong, Anthony S. Atkins, and Hongnian Yu, "Activity classification using a single wrist-worn accelerometer" 2011 5th International Conference on Software, Knowledge Information, Industrial Management and Applications (SKIMA) Proceedings, Jan, 2011.
5. Myeong-Chun Lee and Sung-Bae Cho, "Mobile Activity Recognition using Bidirectional Recurrent Neural Network with Long Short-Term Memory", The Korean Institute of Information Scientists and Engineers, 38(28)th ed. Korea, pp. 263-266, Oct. 2011.
6. Soo-Yeon Shin, "Human Activities Recognition System Using Multimodal Sensor and Deep Learning" M.E dissertation, Department of Mechanics & Design, Kookmin Univ. Seoul, Korea, Feb. 2016.
7. Young-Jin Nam and Ha-Hyun Jo, "Prediction of Weekly Load using Stacked Bidirectional LSTM and Stacked Unidirectional LSTM", Journal of KIIT, Vol. 18, No. 9, pp. 9-17, Sep. 2020.
8. Terry Taewoong Um1, Vahid Babakeshizadeh, and Dana Kulic, "Exercise Motion Classification from Large-Scale Wearable Sensor Data Using Convolutional Neural Networks", 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sep. 2017.
9. Dan Morris, T. Scott Saponas, Andrew Guillory, and Ilya Kelner, "RecoFit: Using a Wearable Sensor to CHI '14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems", Toronto, pp. 3225–3234, Apr. 2014.

Seo-Yung Lee

2017. 2 : Yonsei University Department of Environmental Engineering (Engineering)

2021. 2 : Gangneung Wonju University Medical Device Convergence (Master of Engineering)

2021. 5 ~ present : KCNET Intelligence Information Division

Research interests : machine learning, CNN, cloud computing and big data analysis.

Kwangmin Hyun

2004. 8 : Ph.D. degrees in electronics and computer engineering from Hanyang University

2005. 3 ~ present : professor in the Dept. Information and Communication Eng. at Gangneung-Wonju National University

Research interests : in the areas of communications theory, communication systems, and signal processing for digital systems

Baek-Ki Kim

1997 : Ph.D. degree in electronics engineering from KyungHee University

1990. 9. ~ present : professor in the Dept. Information and Communication Eng. at Gangneung-Wonju National University

Research interests : include signal processing and medical equipment field

Hwan-Seog Kim

2008. 8 : Ph.D. degrees in electronics engineering from CheongJu University

1992. 9. ~ present : professor in the Dept. Information and Communication Eng. at Gangneung-Wonju National University

Research interests : in the areas of AI, IoT and New media communication systems