Real-Time Workout Posture Correction using OpenCV and MediaPipe

3.1 Squat

The squat is a representative weight training exercise in which the lower quadriceps, lower quadriceps, femur triceps, femur muscles, and mesenchymal muscles are developed by repeatedly flexing and stretching the knee joint. Squat is effective in developing the lower extremities, but it can cause injury if performed without a proper selection of posture or weight.

The most common squats include wide squats with 120% wider stances, parallel squats with both feet together, and split squats with both feet spread back and forth in a lunge position.

3.2 Push-up

Push-up is one of the representative muscle exercises, which can be done without the use of equipment. While lying face down, the body is lifted by the force of stretching both arms by focusing the entire body's weight on the four spots of the hands and toes. The primary method is to bend the motion and elbow joints so that the body is not attached to the ground, and to repeat.

The most common types of push-ups are classical push-ups, grip-operated Hindu push-ups, grip-operated diamond push-ups, decline push-ups, incline push-ups, and clapping push-ups.

3.3 MediaPipe Pose

MediaPipe Pose is an ML solution for body pose tracking that uses BlazePose research to infer 33 human body key points from RGB video frames. BlazePose is a lightweight convolutional neural network architecture.

Pose tracking uses a two-stage detector-tracker ML pipeline. First, the pose detector finds a pose ROI (region of interest) within the frame. Next, the pose tracker predicts 33 pose key points from ROI [11]. Applications can be built by applying it to various fields that require posture analysis, such as yoga or fitness.

3.4 OpenCV

OpenCV is a programming library aimed at real-time computer vision, focusing on real-time image processing. It supports a variety of programming languages, including C++, Python, and Java, and a variety of OS platforms, including Windows, Linux, OS X, Android, and iOS. It can be used for many algorithms, including machine learning, math operations, video capture, and image processing.

OpenCV is algorithmically designed to focus on computational efficiency and real-time applications, making it easy to create high-quality commercial programs. It also supports multicore processing, which makes it applicable to a variety of situations.

4.1 Exercise Posture Correction System

Fig. 1 shows a flow chart of posture analysis process. Exercise posture recognition system converts input images generated using a computer webcam from BGR images to RGB images using OpenCV. Using the Landmark model of MediaPipe Pose detects body landmarks required for postural analysis and saves the position. Calculate the angle of the posture using x, y, z coordinates of the analyzed position and verify that it meets the criteria for the correct exercise posture. The guideline suggests a correction is refreshed and drawn on the screen every second while the user's posture does not meet the criteria.

Fig. 1.

Posture analysis flowchart

If the posture is correct, the correction guideline is not printed on the screen. Also, the number of workouts is increased and output by screen and voice.

When a user executes a program, the client program sends a real-time exercise video frame from the webcam. The server finds the landmark in the image, verifies that it meets the criteria for correct posture, and, if not, forwards the correction contents back to the client. If the user is in the proper exercise posture, only the number of workouts is forwarded. The user can check the contents on the screen whenever the value is transferred from the server, and when the correct posture is taken, the number of workouts is output by voice. The guide for posture correction provides how to change which body part to be in the right posture.

The development tools used to implement the exercise posture correction program are Docker, React (v.17.0.2 version), flask (v.1.1.1), gunicorn (v.20.4), Nginx, MongoDB, Visual Code, OpenCV (v.4.5.3) and MediaPipe.

React has the performance benefit of rapid rendering using Virtual DOM. Flask is a microframework based light web framework and has the performance benefit of generating fewer generic errors. Nginx is easy to obtain server performance and reliability. OpenCV and MediaPipe are used to estimate posture.

Fig. 2 shows the system architecture of the program. When a HTTP request is sent from the client framework React to the web server nginx, the connection between the nginx and the server application flask is relayed by gunicorn.

Fig. 2.

System architecture

In flask, OpenCV and MediaPipe are used to analyze image frames delivered on request from clients and store user records in MongoDB, where DB-related tasks are processed as asynchronous tasks using Redis and Celery so that the processing time can be performed quickly.

For this program, we chose squat and push-up, which are the most popular exercises that we can exercise without tools.

4.2 Definition of Correct Posture

There are three proper posture conditions for squats. The first condition is that the hip angle would be close to 90 degrees, and the second condition is that the thigh is almost horizontal to the floor. And the third condition is that the knee and toe lines would be close to a straight line. Therefore, the program determines that the squats were performed correctly only when all the following conditions were met.

• Condition 1) The hip angle (knee-hip-shoulder) is between 60 and 120 degrees.

• Condition 2) Difference of y-values between knees and thighs is within 0.2.

• Condition 3) The difference between the x-values of the knee and toe line is not more than 0.1.

Fig. 3 shows the algorithm for generating guidance for squat posture correction. In the case of Condition 1, it is guided to make the height of the hip higher when the angle of the hip is less than 60 degrees, and to make the height lower when it is greater than 120 degrees.

Fig. 3.

Algorithm to generate squat posture correction contents

In the case of Condition 2, when the difference between the y values of the knee and the thigh is more than 0.2, it is guided to keep the thigh horizontal to the floor. In the case of Condition 3, when the difference between the x values of the knee and the tip of the toe are more than 0.1, the knee is guided not to exceed the tip of the toe.

There are two correct posture conditions for push-ups. The first condition is that the elbow angle will be close to 90 degrees during the push-up, and the second condition is to keep the body close to a straight line during the push-up. Therefore, the program determines that the push-up was performed correctly only when all the following conditions were met.

• Condition 1) Elbow angle (wrist-elbow-shoulder) between 70 and 100 degrees.

• Condition 2) Body angle (shoulder-hip-ankle) is between 160 and 200 degrees.

Fig. 4 shows the algorithm for generating guidance for push-up posture correction. In the case of Condition 1, it is guided that the body has fallen too much when the angle of the elbow is less than 70 degrees, and that the body has to go down a little more when it is greater than 100 degrees. In the case of Condition 2, when the angle of the body is less than 160, it is guided to lower the hip to align the body, and when it is greater than 200 degrees, it is guided to raise the hip to align the body.

Fig. 4.

Algorithm to generate push-up posture correction contents

The posture correction algorithm analyzes the user's image to estimate the pose landmark, calculates it through a function that calculates the angle and value required for the exercise posture condition, and then generates the posture correction guide content through the correction guide content generation function.

4.3 Pose Landmark

Use the key points of the MediaPipe landmark model to determine whether the squat meets the correct posture conditions. Since the angles of the left and right hips can be different, the average of the two values ​​is used as the hip angle as (1). Fig. 5 shows the key points used in the conditions for validating correct squat posture.

• Condition 1) Left hip angle (left knee(25), left hip(23), left shoulder(11)), Right hip angle (right knee(26), right hip(24), right shoulder(12))

Fig. 5.

Landmarks for validating correct squat posture

• Condition 2) Left knee-hip height difference (left knee(25), left hip(23)), Right knee-hip height difference (right knee(26), right hip(24))

• Condition 3) Left foot-knee width difference (left knee(25), left foot(31)), Right foot-knee width difference (right knee(26), right foot(32))

Use the key points of the MediaPipe landmark model to determine whether the push-up meets the correct posture conditions. Since the angles of the left and right elbows can be different, the average of the two values ​​is used as the elbow angle as (2). Body angle uses the average value as in (3) for the same reason. Fig. 6 shows the key points used in the conditions for validating correct push-up posture.

Fig. 6.

Landmarks for validating correct push-up posture

• Condition 1) Left elbow angle (left wrist(15), left elbow(13), left shoulder(11)), Right elbow angle (right wrist(16), right elbow(14), right shoulder(12))

• Condition 2) Left body angle (left ankle(27), left hip(23), left shoulder(11)), Right body angle (right ankle(28), right hip(24), right shoulder(12))

4.4 Camera Calibration

When an image is acquired through a webcam, the position, direction, and angle of the camera are different for each user, thus obtaining an image of a modified object that is different from the object of the real world. Therefore, images with calibration for the camera should be used. For correction to the camera, a method is used to obtain a modified rotation angle using a fixed indicator of the exercise posture and then rotate the image to the negative value of the rotation angle.

Since squat is a standing exercise, the angles of both feet should be kept horizontal. Since push up is a lie-down exercise, the angles of both feet should be kept vertical. The angle of the foot uses a straight line between the heel and the tip of the toe. Since the angles of both feet can be different, the rotation angle uses the average value of the rotation angle of each foot as (4).

Fig. 7 shows the key points used to calculate the angle of the foot. Key points used for calculating left foot rotation angle are left heel(29) and left foot(31). Key points used for calculating right foot rotation angle are right heel(30) and right foot(32).

Fig. 7.

Landmarks used to find the angle of the foot

After obtaining the rotation angle, calculate the rotation conversion matrix. The rotation transformation matrix is defined as (5). And α and β values can be defined as (6).

Center is the coordinate of the center point, scale is the ratio, and θ is the rotation angle. The horizontal and vertical lengths of the image can be obtained using OpenCV, and the rotation transformation matrix is calculated with the negative value of the rotation angle obtained before, the rotation center, and the scale as 1. This is then applied to the warpAffine function to obtain the transformed image.

4.5 Calculate Angle

To estimate the pose, x, y, and z coordinates are used to calculate the angle between the three points. Fig. 8 shows the angle between the three points. Let the three points be A, B, and C. And the angle to be obtained is θ.

Fig. 8.

Angle between three points

To obtain θ, we use the dot product of vectors.

Vector with point C as the point of view and A as the endpoint is called $$ \overrightarrow{CA}$$ and vector with point C as the point of view and B as the endpoint is called $$ \overrightarrow{CB}$$. Vector $$ \overrightarrow{CA}$$ and $$ \overrightarrow{CB}$$ are defined as (7).

So, θ is obtained by substituting it for the internal formula of the vector. In this way, the angle between the three points can be defined as (8).

5.1 Squat

Fig. 9 shows the output from the program when the correct squat posture is taken. It was found that the hip angle between 60 and 120 degrees, the thighs positioned close to the floor, and the knees not crossing the toe line are required to avoid guidelines. And when the condition was satisfied, the number of workouts was increased on the screen. It was confirmed that the number 8 was output by voice.

Fig. 9.

Correct squat posture

Conversely, Fig. 10 shows the output from the program when the incorrect squat posture is taken. Because the user’s posture did not meet the condition to keep the thigh horizontal, a guideline was printed to make sure that the thigh horizontal with the floor. And the number of exercises did not increase.

Fig. 10.

Incorrect squat posture

5.2 Push-up

Fig. 11 shows the correct user's posture when executing push-ups. When performing push-ups, the elbow angle between 70 and 100 degrees and the body angle close to 180 degrees was required to prevent the guidelines and to increase the number of workouts. And it was confirmed that the number 3 was output by voice. No correction is printed on the screen, and an increased number of workouts is printed. In addition, the increased number of workouts is output by voice.

Fig. 11.

Correct push-up posture

On the other hand, Fig. 12 shows the output from the program when the incorrect push-up posture is taken. Because the user’s posture did not meet the condition that elbow angle was below 100 degrees, a guideline was printed to bend the arm more to approximately 90 degrees. And the number of workouts did not increase.

Fig. 12.

Incorrect push-up posture

And we conducted an experiment to analyze the performance of the program. The detail of the experiment is to calculate the success rate whether the program accurately calculates the number of workouts when the user performs exercise 1 to 5 times in the correct posture. The formula to calculate the success rate is defined as (9). And Table 1 shows the experimental results.

Table 1.

Calculation success rate