Mobile AR Contents Based on IoT(Internet of Things) Using Shading Language over Smartphone and Arduino Bluetooth Sensors

Ⅰ. Introduction

The performance improvement of the IT equipment and the development of the Internet, leading to the fourth industrial revolution, have created a new field of Internet of Things. Everything in everyday life is connected to the Internet to create new services, and the Internet of Things is combined with the Big Data to pioneer new forms of business that have not been seen before. The prospect of the Internet of things is unlimited in its utilization[1]. As time goes on, the Internet will become deeply involved in our lives and its importance will increase day by day. In this situation, the growth of exponential Internet devices is not difficult to predict. The number of devices supporting the internet is different for each market research organization and global company. And it is expected to reach approximately 30 billion to 50 billion by 2020[2][3]. Things Internet is a technology that has been widely publicized in recent years. IoT(Internet of Things) refers to the technology of connecting sensors and communication functions to various objects and connecting them to the Internet. It is a technology that enables Internet connected objects to exchange Internet data of objects, analyze it by itself, provide the learned information to the user, or remotely control the user. Here, things become various embedded systems such as household appliances, mobile equipment, and wearable computers. Things to be connected to the Internet must be connected to the Internet with a unique IP that can identify themselves, and a sensor can be embedded to obtain data from the external environment[4]. The core technologies of the Internet are the sensing technology, the interface technology, and the networking technology. Sensing technology is a technology that measures physical quantities such as temperature, humidity, heat, gas, position, speed, and illumination using various methods. In the object internet, basic signal processing and There is a need for a built-in smart sensor technology including a module capable of performing algorithms. Interface technology is a technology that works with application services that perform specific functions through human, object, and service, which are the main components of the Internet of things. Networking technology is a wired and wireless networking technology that performs physical connection of various devices in a distributed environment. It is composed of Wireless Personal Area Networks (WPAN), Wireless Fidelity, 3G · 4G, Long Term Evolution (LTE) Ethernet, BcN(Broadband convergence Networks), satellite communication, etc.[5]-[8], which have been used to demonstrate the usefulness of mobile devices such as smart phones and HMDs[8], Augmented Reality is a realistic media technology that naturally fuses virtual information or content to the real world and extends human sense and perception. In contrast to virtual reality, augmented reality reduces the burden of 3D modeling / rendering for realistic virtual world construction because it utilizes real space as media. In contrast, virtual reality or contents. The new technical difficulties must be addressed because they have to be provided adaptively[9]. Unlike the console game, the smartphone game uses the voice recognition function to allow the user to make a command to the game character, to use the interface using the touch screen, or to interact with the objects and objects using the network communication. And users can easily play the game[10][11]. The IoT technology, in which the interaction between these objects is achieved, will be facilitated by the rapid diffusion of smart phones and the realization of wearable computing, which will facilitate access to various new information using various sensors. Through these technologies, people and objects, Will interact with each other[12][13]. The Internet of things with these characteristics is actively studied by the standardization organizations for smooth connection of equipment. However, in order to control or acquire information on the Internet devices, there is a need to separately develop interworking issues (IP address, Wi-Fi, Bluetooth, NFC, etc.) and related application software or apps. In the smart digital era, the key is the communication of emotion and technology, and the future information service will become a sensibility-based service in which the value of products and services is determined by touching, feeling, and communicating with devices in viewing and listening. In particular, it is highly likely to be developed as a next-generation platform that provides intelligent services that provide contents according to the situation using augmented reality technology and object Internet technology, or provides data to users in real time in a changing environment. Considering the same technology trends, we produce mobile augmented reality IoT contents using mobile camera output as background. The collected contents are collected and analyzed using bluetooth based IoT sensor data, and the environmental index information is detected and transmitted to the mobile phone. In the mobile phone, the external environmental change according to the environmental index information is applied to the background of the augmented reality contents and output. We can provide emotional information for a more natural and deep immersion to users of mobile AR contents created by applying such environmental index information. The results are confirmed through experiments.

Ⅱ. 4.0 Bluetooth Interface

In this paper, we use bluetooth communication with low power consumption for interaction with smartphone. Bluetooth refers to short-range wireless standard technology that enables the exchange of information between devices and devices. After bluetooth version 1.0 was released in December of 1999, bluetooth announced EDR, which increased data transmission speed by up to three times in 2004. Bluetooth EDR has a disadvantage of high power consumption but high transmission speed. The bluetooth low energy(BLE) technology has been developed to secure these shortcomings. BLE technology is a technology that significantly reduces standby power consumption compared to existing EDR. In addition, it is possible to simplify the connection procedure of the device and to reduce the communication packet size. The advantage of BLE is that it is suitable for direct object communication between potable healthcare, wearable terminal and neighboring terminals[14][15]. In addition, currently available smart phones are equipped with BLE technology in the device, so they can easily communicate with other objects when they need to interact with objects using smart phones.

Ⅲ. System Configuration

Augmented reality is the latest technology that tried to converge physical space and digital space. Augmented reality provides multidimensional information to users exposed to physical environment by fusing information sensing technology of physical space and information data of digital space[16]-[21]. In the IoT environment using the bluetooth communication method with the tablet PC on which the AR environment is established, the user can freely move the virtual objects on the AR game implemented in the tablet PC using the joystick to play the AR game. We use Arduino and sensors to detect environmental information. A vibration sensor for detecting vibration and shaking information such as an earthquake, an MP (micro phone) sensor for detecting external noise, a CDS sensor for detecting the degree of darkness, and an LED device for confirming whether or not these sensors are stuck. For example, if there is an external earthquake, the vibration sensor on the Arduino senses it, and when it sends the earthquake information to the smartphone using bluetooth communication, the phone will use the information of the earth to detect the background.

To a state of earthquake. To do this, we apply shading language to background graphic in real time and output it. In addition, according to the size of the dark input information, the CDS sensor detects it and sends it to the smartphone to output the background of the game content in dark real time. When the external noise is large, it detects it and displays various effects such as crack on the background of the game. Can be implemented. Through this, it is possible to produce high-quality realistic contents that direct environment information to the user in real time and direct them on the currently used contents. This is shown in Fig. 1.

Fig. 1.

System configuration diagram proposed in this paper

Fig. 2.

Pairing for bluetooth connection with arduino and smartphone

Ⅳ. Experiments

4.2 Experiment 2

In Experiment 2, when the external environment is dark, the background of the currently used content is expressed differently in real time according to the dark information.

Experiment 2 adds a CDS sensor (see Fig. 3(a)) to detect light intensity information and apply it to the background of the augmented reality. Fig. 3(a) shows that when the light is bright, the resistance of the CDS sensor is low, and the red of the power supply and the green of the resistance are turned on. However, when the amount of light decreases and becomes darker, the resistance value becomes larger and the green light is turned off, which shows that the amount of light in the surrounding environment is reduced to dark state (see Fig. The phone detects the change of the light information according to the change of the environment and adjusts the background image according to the value to make it dark in real time. Fig. 3(c) shows the steady-state augmented reality image. Fig. 3(d) shows the darkened image according to the CDS sensor value. It is an edge-detected image to show surrounding object information.

Fig. 3.

Using the CDS sensor to detect the environmental change index according to the light intensity and experimental results applied to the background in real time

4.3 Experiment 3

Experiment 3 tests how to convert the camera color image of a smart phone into a monochrome image effectively. For this purpose, when the color image is converted into a gray image and the gray image is converted into a monochrome image, if the edge information of the image is represented, the object shape information of the color image can be outputted without much loss.

Fig. 4(a) shows the input image of the smart mobile phone. (b) shows the input image converted into gray image. In this case, we used the length function and gray-converted 3 channels of color.rgb1 channel. Fig. 4(c) shows the Prewitt algorithm and Fig. 4(d) shows the result using the dFdx/dFdy built-in function.

Fig. 4.

(a) input image, (b) gray image and (c) Prewitt image (d) the result image using the dFdx / dFdy function

The Prewitt edge detector in Fig. 5(C) is a suitable method to estimate the size, Direction of the corner. Differential gradient edge detection requires some time consuming computation to estimate the direction in the x and y directions, but compass edge detection takes direction directly from the kernel as a maximum response. The Prewitt operator is limited to eight possible directions and the edge detector is estimated in eight directions. Fig. 4(d) is a fragment shader function that calculates the partial derivative of the dFdx/dFdy screen in the X direction and Y direction using the built-in function dFdx/dFdy since GLSL1.1.

Fig. 5.

Arduino vibration sensor detects the vibration and sends it to the smartphone

4.5 Experiment 5

Experiment 5 is an experiment in which a large noise from the outside was detected by the Arduino MP sensor and applied to the background image of the AR game contents of the phone.

Fig. 6(c) shows the camera input image and Fig. 6(b) shows the mosaic image. The image of Fig. 6(d) is an image of 60 * 60 size (vec2 tile_num = vec2 (60.0,60.0);) tile changed by shader programming.

Fig. 6.

Mosaic result image according to input image and tile size by external noise

Applicable result clear the square form the file and book format are maintained, but the information such as the clear file name, book name, and judge can not be confirmed.

Fig. 6(e) shows an example of another input image. (f) has 60 tiles and (g) has 80 tiles are used. (g) has more object information than the figure (f). However, parts such as the character information of an object can be identified but it can not be read. Figure (h) shows the approximate lubrication of large objects when 30 tiles are used. Therefore, the number of tiles can be adjusted appropriately according to the amount of information that should not be exposed, so that the conversion into a desired image is easy.

Ⅴ. Conclusion

At the core of the smart era is the communication of emotion and technology, and the future information service will become a sensibility-based service in which the value of products and services are determined by touching, feeling, and communicating with devices. In addition, emotional information service technology that can enhance sense of realism and realism by stimulating human senses is essential element technology in various application services. For this purpose, this paper acquires environmental information using Arduino's sensor and makes it into environmental information, and transmits it to the smartphone using bluetooth communication. In the smartphone, the information of the background image is transformed in real time according to the change of the environment information in the game contents and the like, and the emotional information is considered in the output.

Experiment 2, experiment 3, experiment 4 and experiment 5 were used to detect various external environment changes such as darkness, earthquake, and noise through sensors and to provide various emotional information to the contents in augmented reality. We can propose a new type of digital interface to communicate with augmented reality. With this new form of environmental information recognition and emotional delivery mechanism, the user can receive current and realistic media consumption services by recognizing the current emotion and assisting in purification, sublimation, suppression of emotions. In the future, various researches are needed to collect, analyze and apply various types of external environmental information, and it is necessary to study not only the background but also various types of delivery techniques.

References

• Jongsub Park, June Seok Hong, and Wooju Kim, "A Study on Intuitive IoT Interface System using 3D Depth Camera", The Journal of Society for e-Business Studies, 22(2), p137-152, May), (2017.
• D. Evans, "The Internet of Things How the Next Evolution of the Inernet", White Paper, Cisco Internet Business Solutions Group, (2011).
• Mckinsey & Company, "Global Semiconductor Alliance / Mckinsey Report on IoT", (2015).
• Junjae Yoo Ji, and Sanghak Lee, "IoT, Big Data and AI", Communications of the Institute of Information Scientists and Engineers, 35(7), p43-50, May), (2017.
• Mansik Gong, et al., "Trends and Prospects of Internet (IoT) Technology", Journal of the KSME, 56(2), p32-36, Feb.), (2016.
• Bang, Jung-seong, Dong Chun Lee, Sang-Hyun Seo, Yong-Joon Lee, Heung-Joo Lee, and Sonook Ho, "VR/AR Game Technology Trends", Electronic Communication Trend Analysis, 31(1), p146-156, Feb.), (2016.
• Kim, Kyung Ho, Jong Chan Jang, and Han, Sang Chul, "Vehicle Augmented Reality Technology Status and Outlook", KEIT PD ISSUE REPORT, p13-29, Mar.), (2015.
• http://www.pokemongo.com/, . [accessed: Aug. 06, 2018]
• Analysis of technology trend of 3D fusion industry, Kyungpook National University, 3D fusion technology support center, Jun.), (2015.
• Hye-Won Han, and Hye-Young Yun, "A Study on Correlation Between Console Game Controller and Gam e Contents", Journal of the Korea Game Society, 11(4), p61-72, Aug.), (2011.
• Sunha Park, "Mobile Revolution-Effect and Role of Design Elements of Smartphone and Mobile Game based on Samples study of Smartphnoe in Korea", Journal of Digital Design, 10(3), p135-144, Jul.), (2010. [https://doi.org/10.17280/jdd.2010.10.3.014]
• Woo-Soo Jeong, Sa-Hyuk Kim, and Kyoung-Sik Min, "An Analysis of the Economic Effects for the IoT Industry", Korea Society for Internet Information, 14(5), p119-128, Oct.), (2013.
• Hae-Ryong Lee, Joon-seok Park, and Jeon-Woo Lee, "Sensibility UX Technology Trends", Electronic Communications Trend Analysis, 26(5), p83-93, Oct.), (2011.
• Jong-Tak Kim, Jae-Yong Soh, Jong-Hun Kim, and Un-Gu Kang, "Bluetooth based Portable Healthcare Gateway Development for Hypertension Patients", The Society of Digital Policy & Management, 11(12), p339-344, Dec.), (2013.
• Yong-Ho Oh, Jae-Shin Lee, and Soon-Ju Kang, "A Protocol Design for Opportunistic Direct M2M Protocol Design for Opportunistic Direct M2M Communication in Wearable Computing Environment", The Journal of Korean Institute of Communications and Information Sciences, 39(2), p151-163, Feb.), (2014. [https://doi.org/10.7840/kics.2014.39C.2.151]
• Park, Ki-young, "A Study on the Mobile Guide Program for Museums", Journal of Creative Industry, 2(3), p23-49, Dec.), (2015.
• https://gamedevelopment.tutsplus.com/tutorials/usingdisplacement-shaders-to-create-an-underwater-effect—cms-27191. [accessed: Sep. 10, 2018]
• S. K. Goudos, Panagiotis I. Dallas, and Sofoklis Kyriazakos, "A Survey of IoT Key Enabling and Future Technologies: 5G, Mobile IoT, Sematic Web and Applications", Wireless Personal Communications, 97(2), p1645-1675, Nov.), (2017. [https://doi.org/10.1007/s11277-017-4647-8]
• Antonio Celesti, Antonino Galletta, Lorenzo Carnevale, Maria Fazio, Aime Ĺay-Ekuakille, and Massimo Villari, "An IoT Cloud System for Traffic Monitoring and Vehicular Accidents Prevention Based on Mobile Sensor Data Processing", IEEE Sensors Journal, 18(12), p4795-4802, Jun.), (2018. [https://doi.org/10.1109/JSEN.2017.2777786]
• Tai-Dal Kim, "A Study on the Development of Intelligent Markup Indicator (IMI) Technology for Underground Facilities Management Using IoT", JIIBC, 17(3), p129-136, Jun.), (2017. [https://doi.org/10.7236/JIIBC.2017.17.3.129]
• Myeong-Soo Jang, Woo-Beom Lee, "Implementation of Hand-Gesture Interface to manipulate a 3D Object of AugmentedReality", JIIBC, 16(4), p117-123, Aug.), (2016. [https://doi.org/10.7236/JIIBC.2016.16.4.117]