Online Finger Circumference Measurement System using Semantic Segmentation with Transfer Learning

2.1 Online Finger Circumference Measurement System

Finger circumference measurement methods to get appropriate ring size can be broadly categorized into: (i) Traditional ways including paper strip methods and visiting offline stores which are inconvenient. While the latter ensures high accuracy, it is not easy to visit stores every time and it is almost impossible when purchasing rings at online shops; (ii) Applications to measure the size of the ring that one already has which is of no use when one does not have a ring; (iii) Measuring the circumference of a finger is another type of online measurement system[1].

Similar to the second type of online measurement system, we also compute the circumference of fingers. However, we use a semantic segmentation-based deep learning model to get the boundary of fingers, not leaving it users which only increases inaccuracy. In this paper, the comparison and improvements are directly made on a mobile application called PerfectFit by Radius Technologies that sizes the circumference of fingers.

2.2 Semantic Segmentation with Transfer Learning

Semantic segmentation, based on pixel-level image classification, is a fundamental approach in the field of computer vision for scene understanding. Compared to other techniques such as object detection in which no exact shape of the object is known, segmentation exhibits pixel-level classification output providing richer information, including the object’s shape and boundary[2].

Transfer learning is to recognize and apply knowledge and skills learned in previous tasks to novel tasks. In this regard, transfer learning aims to extract the knowledge from one or more source tasks and applies the knowledge to a target task[3]. It uses the knowledge gained while solving one problem (where we have access to a larger dataset) and applies it to a different but related problem[4]. With transfer learning, it is able to increase learning speed since there are fewer new things to learn and the algorithm is faster to generate high-quality output. On top of that, it reduces the amount of data required[5]. Therefore, the performance of one segmentation task can be effectively improved by transferring knowledge from one domain data to another.

3.1 System Overview

Development environment as seen in Fig. 1, our system uses React for frontend, Nginx for the web server, Gunicorn as Web Server Gateway Interface (WSGI), Uvicorn as Asynchronous Server Gateway Interface (ASGI), FastAPI for the backend API server, and MongoDB for the database. We containerized the application using Docker, got SSL certificate, and deployed our app to Amazon Web Services (AWS).

Fig. 1.

Diagram of system architecture

The app is developed in the form of a progressive web application (Fig. 1(a)). Handling HTTP requests, sending responses and making use of resources are done with FARM Stack (Fig. 1(b)). TensorFlow and OpenCV are used to implement the deep learning model for finger circumference measurement (Fig. 1(c)).

3.2 FARM Stack

FARM stack stands for full web application development stack consisting of FastAPI, React, and MongoDB. Here, we utilized FastAPI as the python web framework paired with React frontend and MongoDB as a NoSQL database.

FastAPI is a modern web framework for the python backend, released in 2018. Because of its remarkably high performance on par with NodeJS and Go, Netflix, Uber, Microsoft amongst many other corporations are using FastAPI[6]. As the name itself has fast in it, it is faster in speed than other major python frameworks such as Flask and Django. This is because Flask is built over WSGI (Web Server Gateway Interface) whereby FastAPI uses Uvicorn as an ASGI (Asynchronous Server Gateway Interface) server[7].

ASGI is the spiritual successor to WSGI and able to achieve high throughput in IO-bound contexts which cannot be handled by WSGI. With the advancement of technology, a lightning-fast ASGI server without routing capabilities called Uvicorn was born. Then Starlette came along, which provides a complete ASGI toolkit on top of Uvicorn[8].

FastAPI is built on top of two libraries, Starlette for the web parts and Pydantic for data validation[9]. Starlette lets us serve a request and return a response asynchronously[10]. Hence, we were able to build a high-performant async IO microservice without the need for additional asynchronous task queues such as RabbitMQ and Celery. And thanks to the full compatibility with the Pydantic library, FastAPI can automatically validate the input data from the external callers at runtime and generates errors to the clients in JSON format when it receives invalid data[6].

As it uses Pydantic known as the fastest validation library (which Flask lacks and 12.3 times faster than Django-rest-framework), objects can be directly passed from the client to the database and the same applies to the other way around[11].

Besides, FastAPI generates data model documentation on the go coming with either Swagger UI or ReDoc. This not only increases the productivity during development working in a team, but also allows developers to test the API endpoints more conveniently[12].

Key features of FastAPI can be summarized as follows:

• 1. NodeJS-like high performance using Python programming language.
• 2. Native asynchronous support allowing concurrency and coroutines.
• 3. Data validation and serialization.
• 4. Automatically generated interactive API documentation with Swagger UI.

Amongst many other advantages complementing limitations of existing python frameworks, the native support for async IO operations itself was enough to adopt FastAPI as the backend server for our web application.

MongoDB is a document-oriented NoSQL database that stores BSON (Binary JSON) documents. Compared to Relational DataBase Management System (RDBMS), NoSQL (Not only SQL) database is schema-less which is a synonym for the flexible data model[13]. This means a single collection can hold multiple documents and these documents may consist of heterogeneous data[14]. Embedded documents and arrays reduce the need for expensive joins, unlike relational databases[15].

Indexing is of great importance for improving the performances of search queries. Since every field in the documents is indexed with primary and secondary indices, it is easier and takes less time to get or search data from MongoDB[16].

It also supports replication. If the primary server goes down during a transaction, then the secondary server handles the transaction without human interaction[17]. This saves our time for maintenance and makes operations smooth. Indexing and replication enable query responses to be faster, which eventually leads to high performance[18].

High availability and easy scalability should not be left out as these are the main reasons the MongoDB database was developed for. Considering the continuous updates for our service afterwards, easy and horizontal scalability was one of the core features to use MongoDB as a database[19].

Particularly, the fact that both MongoDB and FastAPI work natively with JSON, making a good pair themselves made us use MongoDB as a database backend[20]. We used PyMongo, the official python driver for MongoDB to create its client. As we deployed the web application to Amazon Web Services, we used their cloud hosting service, MongoDB Atlas instead of a local MongoDB database.

3.3 Deep Learning Model and Computer Vision

In this section, we describe the core idea of our measurement method to compute the width of the finger in an image. Deep learning model based on semantic segmentation enables the contour recognition of a hand with high accuracy and the width of the finger is computed in pixels via computer vision.

3.4 Progressive Web App (PWA)

Progressive web app is a regular web application designed to look and function like a native mobile application[27]. PWA has features of web browsers with advanced enhancement strategies but allows users to have a native app-like experience on any device.

Another advantage about PWA is that it still works under network-disconnected conditions. IndexedDB is a way to persistently store data inside a user's browser. Since it allows us create web applications regardless of network availability, our application can work both online and offline[28]. When a web app makes an HTTP request for some data and the user doesn’t have internet connectivity, we can still serve this data from the local database cache[29].

4.1 Experiments with WebXR

WebXR Device API developed by W3C group implements the core of the WebXR feature set, renders the 3D scene to the chosen device at the appropriate frame rate, and manages motion vectors created using input controllers[30]. As shown in Fig. 2, it was impossible to get measurements in millimeters.

Fig. 2.

Result of computing the length using WebXR

Since the ring size is differentiated by a millimeter, we decided not to implement WebXR on our system.

4.2 Semantic Segmentation with Transfer Learning

In this section, we describe our approach to measure the finger circumference by implementing a pre-trained semantic segmentation model called DeepLabV3-MobileNetV2 and train it with hand datasets through transfer learning. Our goal is to measure the width of a finger in an image, compute the actual width, and convert it into circumference.

4.2.1 Transfer Learning Process

Transfer learning is an effective way to customize the pre-trained model according to the purpose of use. For the transfer learning, we gathered the hand gesture recognition dataset from Kaggle beforehand. Then, we labeled the training dataset using OpenCV and converted it into TFRecord format to increase the learning rate and reduce overhead while reading the data. After the transfer learning with 1500 training datasets and 8000 epochs, there has been a remarkable improvement in segmentation mask containing the outline of the hand as shown in Fig. 3.

Fig. 3.

Result of segmentation mask via DeepLabV3-MobileNetV2 pre-trained model (a) and the model after transfer learning (b)

4.2.2 Finger Circumference Measurement

As seen in Fig. 4, a segmentation map of 513×513 matrix is the output of our model. The Segmentation map here tells which pixels are in the class of hand and which are not. Through the (x, y)-coordinates sent from the client side, the semantic image segmentation model gets the two endpoints of the finger to get measured. Then it computes the pixel-width of the finger in the image.

Fig. 4.

Segmentation map as the output of the model

To convert the pixel-width into the actual width, we have adopted a 2.2×2.2cm size 100-won coin as the reference object. The formula called pixels per metric mentioned in Section 3.3 is applied at this point, where the numerator, the width of the object becomes the pixel-width and the denominator, the known width becomes 2.2cm.

The pixel-width between two endpoints of the finger is divided by the width of the 100-won coin, the actual length is computed as seen in Fig. 5. Lastly, the width of the finger is computed into the circumference based on the ring size chart.

Fig. 5.

Actual length computation using a 100-won coin as the reference object

4.3 Measurement Procedure and Main Features

Users can easily access our web application by searching‘ringfit’in web browsers. Followings are written in the order of actual service flow.

As soon as users are on our website, they are informed of how to use the service in 4 steps as seen in Fig. 6. Users have to prepare a 100-won coin as the reference object for the first step. Next, they may upload a photo of a hand with the coin placed on the left side of their hand.

Fig. 6.

Users’ guide in 4 steps

Once the image is successfully uploaded, users can choose which part of the finger to get measured by adjusting the red line on any part of their finger marked as the third step in Fig. 6. Soon, a proper ring size with finger circumference is provided.

At this point, users can easily find out the international ring size conversion chart retrieved from the database in the order of Korea, US, Italy, and the United Kingdom by tapping the screen as shown in Fig. 7.

Fig. 7.

International ring size conversion chart in the order of KR, US, IT, and the UK

Users can save the results in indexedDB with extra information consisting of which hand, which finger, and which part of the finger as shown in Fig. 8.

Fig. 8.

Users can save the results in indexedDB

Saved data can be checked anytime, anywhere, even when the devices are not connected to networks. Fig. 9 is an example of a user whose left middle finger fits size 13 with a circumference of 56mm.

Fig. 9.

Saved results can be checked even under network-disconnected conditions

4.4 Results

The comparison of measurement results between PerfectFit[1] and our proposed method is made in millimeters. We have made experiments on the cases of 51mm, 53mm and 55mm as seen in Table 1.

Table 1.

Comparison of the previous method and ours

PerfectFit not only showed low accuracy but also computed certain values particularly around 45mm. Through semantic segmentation model and computer vision, we were able to come up with much more reliable results and reduce the error from 7.7mm to 2.3mm on average. Accuracy calculated in percentage also showed improvement from 85.6% to 97.5% on average.