Marker Concealment Using Print Color Correction and Its Application

Abstract

The markers used in the marker-based Augmented Reality have disadvantages that they are heterogeneous in their surroundings and difficult to detect in a dark environment. Therefore, it is necessary to study the markers that can be identified even in dark situations. We analyzed the studies that supplemented the weakness of markers and created new markers. In this method, it is produced by overlapping printing of markers and pictures based on the fact that infrared images vary depending on the type of printer. However, the printed color was darker than the original one, because the marker and the color of the picture were becoming a subtractive mixture. In order to reduce the color difference and to show a similar color, it is required to correct the picture color of the marker part and the picture color of the non-marker part. The color correction was processing by comparing the combined result of Printer-A RGB color code on the top of markers pressed by Printer-B and the sole result of Printer-A RGB color code. As a result, the color difference between the marker, the overlapping part of the picture and the picture part was reduced, and the marker was concealed so that it was not visible to the eye. The concealed markers are able to replace existing invisible markers and can be detected in dark environments. In particular, since the marker concealment method uses original ink or toner without modifying a general printer, it can be easily manufactured. Besides, given that printing work is done by a printer, it is possible to mass-produce uniform quality markers.

In this paper, we evaluated and analyzed the results of user evaluation to check the effectiveness of markers. The assessment was conducted by using the application program which was made with and without the concealed marker method. The survey results showed that there was a positive response to the AR content with the concealed marker and the rejection of the concealed marker decreased compared to the existing markers.

1 Introduction

With the advancement of a performance of mobile and HMD devices, the application areas of AR are becoming more diverse. It is essential for AR to keep track of the position and orientation of the object in the camera image. Marker-based Augmented Reality are most widely used in easy-to-use ways [1,2,3]. However, the marker is expressed in dark color and figure, hence it is heterogeneous with the surroundings. Research has been conducted to improve the disadvantages of these markers [4,5,6,7,8].

Additionally, it is difficult to detect the marker with the camera in a dark environment. The infrared camera has been a method for recognizing a marker in a low-light mode. This is a method of using an ink having a characteristic of absorbing infrared rays of a specific wavelength. By drawing the shape of the marker with a pen, the technique recognizes the marker. However, due to the color of the ink, we can adopt only invisible markers that are similar to the background color. Another method is to dilute the infrared absorbing ink with acetone to color the marker part.

The new markers improved the existing infrared markers, applying printing differences between printers without using infrared ink. There were differences in printed color and infrared image depending on how it works and the composition of the toner and the ink. If you print two markers on two printers with significant differences in image Infrared, you will see the markers on the infrared camera images, but the pictures are not visible. However, since the marker area is printed twice over, the color is darker than the other areas. In order to reduce the color difference and to show a similar color, it is necessary to correct the picture color of the marker part and the picture color of the non-marker part.

The color correction was processing by comparing the combined result of Printer-A RGB color code on the top of markers pressed by Printer-B and the sole result of Printer-A RGB color code. As a result, the color difference between the marker, the overlapping part of the picture and the picture part was reduced, and the marker was concealed so that it was not visible to the eye.

The characteristics of the concealed marker are as follows. First, the marker can be recognized even in the low-light mode. Second, the marker and background image look natural. Third, the mass production of uniform quality markers is possible. Lastly, it is easy to produce.

In this paper, we analyzed the description of marker concealment and the opinions and evaluation of users. The participants in the review were targeted to young 20s people who frequently use multimedia technologies such as Augmented Reality. Participants were able to directly view three samples of common markers, existing invisible markers and concealment markers, and test markers on projector-based AR applications. The majority of the users agreed with the need for invisible markers and evaluated that the markers concealment method was better than the current IR markers. A detailed analysis of user evaluation a given in Sect. 4.

The composition of this paper is as follows. Section 1 explains the need for marker concealment, its introduction, and its composition. In Sect. 2, we analyze the existing research on markers that can be used in invisible and dark places. Section 3 explains marker and image overprinting using IR image difference and printer color correction. In Sect. 4, the user evaluation process and its results are analyzed to verify the usefulness of marker concealment. Finally, conclusions and future research are discussed in Sect. 5.

2 Related Works

Previous studies on markers that can be used in the low-light mode without being visible to the naked eye use mostly Near-Infrared band. Near-Infrared rays are broadly utilized as markers because infrared objects can be seen in dark environments. Especially, considering infrared is widely used, there are many studies on Near-Infrared-based markers [4, 5].

Studies using IR cameras include using IR absorbing inks and using the infrared features of objects or prints. The IR pen has the advantage of being easy to use because it can draw the shape of the marker directly [6]. However, there is a drawback that the markers are not constant and the color of the IR pen is not transparent. IR inks are very dark in color, so when diluted with acetone, they appear to be a light, and you can draw large markers faster than pens. However, when diluted to a light color, the infrared absorption intensity is lowered, and uniform coloring is difficult [7].

The use of such inks does not show the color of the ink itself in a low-light environment but is visible to the public as a regular marker in bright light (Fig. 1).

Fig. 1.

Marker IR image painted with infrared absorbing ink

Some publications and printed papers are also seen in infrared images. Printing markers with printer ink with these features can be used as infrared markers. This has the advantage of being able to rapidly produce a large number of infrared markers. As it is visible, you can overlay the print marker with a different color, therefore it is similar to the surrounding tone. However, when sprayed with spray, the surface is not uniform and stains occur. These problems result in poor print quality when printing pictures (Fig. 2).

Fig. 2.

Left: RGB Camera Right: IR Camera

3 Marker Concealment

The marker concealment used in this paper is a form in which a marker is inserted in the picture. Marker concealment is the use of two printers with different intensity of infrared image, not using infrared ink, to overprint the marker and the picture [8]. This section describes the process of making marker concealment.

3.1 Overprinting Pictures and Markers

If the markers printed on different printers are viewed with an infrared camera, the image intensity may be different. This difference makes it possible to achieve the same effect as using infrared ink without using infrared ink. Overprinting of markers and pictures uses two printers. Printers that print the markers use what looks good on infrared images (Printer-B). Printers that print pictures use something that is not visible in the infrared (Printer-A). We compared printers and markers printed on multiple printers with infrared cameras and chose printers for overprinting. The printer used is a regular printer and has not been deformed or reassembled.

Since the color of the marker is black, we adjusted the grayscale level from 0 to 255. Even if the grayscale level is 220, it is possible to detect the marker with the infrared camera.

3.2 Printer Color Correction

The printed color was darker than the original one, because the marker and the color of the picture were becoming a subtractive mixture. In order to reduce the color difference and to show a similar color, it is required to correct the picture color of the marker part and the picture color of the non-marker part. Figure 3(a) shows the color difference between the marker part and the non-marker part. Color correction reduces this color difference (Fig. 3(b)).

Fig. 3.

Marker part color and Invisible Marker part color

Color patch images printed by different printers are in different colors that are caused by the printing method and the characteristics of the printing materials. For minimizing color difference between printed images, color correction algorithm is used.

For print color correction, the input color and output color of the printer must be compared, and the comparison can be performed using a color patch. Since color patches cannot be created in all colors, we used color patches that were divided into 7 spaces for all color ranges from 0–255. Figure 4 shows the color patches arranged in color spaces.

Fig. 4.

Color space placement of color patch

The generated color patch can be printed and captured by the camera, so that the RGB values of the digital data and printed data can be used for color correction. The two RGB values correspond to each other, and when the digital data of color patch is printed, corresponding printed data indicates printed value. With the correspondence, the desired printed color can be found in digital data of color patch, and the RGB value not in the color patch can be inferred by using the color patch.

Color patches are printed in two types. The first is that the color patch is printed using the printer A, the second is that the marker is printed using the printer B, and then the color patch is printed using the printer A. Figure 5 shows the two types of color patch RGB values printed in the color space. In Fig. 5, we can see that the color representation range on the left is wider than the color representation range on the right. To print the same color by different printers, the range of colors that the two printers can print should include the color range of the image. Since the invisible marker is narrower, the color range of the image is limited within the range of the invisible marker. The image was taken with the camera in a condition where the light was constant, and the image and the color patch should be shot in the same environment.

Fig. 5.

Left: Color patches printed on printer A, Right: Print the marker on printer-B and then print the color patch over to printer-A.

It is possible to calculate the color of the overlapped portion of the marker which is color-corrected by the inferred color patch RGB value. Figure 6 shows the measurement of color before and after color correction using a spectrophotometer. Figure 6(a) is the color value before color correction and Fig. 6(b) is the measurement value after color correction. It can be confirmed that the color difference is reduced.

Fig. 6.

Spectrophotometer color value compare

Figure 7 shows an example of applying marker concealment to a photograph.

Fig. 7.

Marker Concealment print comparison (Left: No Correction, Right: color correction)

4 User Evaluation

In this chapter, we conducted and analyzed user evaluations to verify the necessity and usefulness of marker concealment.

4.1 Assessment Methods and Evaluator Consist

Participants were 50 college students aged 20–22. Because they are familiar with mobile devices and frequently access multimedia technologies and contents, they will be more active in AR technology and content consumption than other age groups.

The AR system prepared for demonstration and evaluation is based on a mobile projector camera. There are four types of markers, general markers, two conventional invisible markers, and proposed concealment markers. Experiment (evaluation) was conducted in a bright environment and a dark environment so that invisible markers could be compared. The evaluation items are 10, and the necessity aspects, usability aspects are evaluated.

Figure 8(a) shows the gender of the participant. There are 33 men and 17 women. 96% (48 participants) of participants knew about AR (Fig. 8(b)). And 90% of participants (45) experienced AR (Fig. 8(c)). As shown in Fig. 8(c), no one used AR every week, and many participants experienced 1 to 3 times a year. This result shows that augmented reality is widely used but is not widely used in everyday life.

Fig. 8.

Evaluator configuration and background

4.2 Necessity Aspects

In necessity aspects, we evaluated the problems and improvement points of the concealment markers by evaluating participants who experienced augmented reality based on concealment markers.

In Fig. 9(a), “satisfaction” and “Neutral” are large numbers because they did not experience augmented reality in various environments and form. Figure 9(b) and (c) show the reason for this. After experiencing marker based augmented reality in a dark environment, the necessity ratio was high.

Fig. 9.

Survey of result on necessity

4.3 Usability Aspects

In usability aspects, the interest in AR technology and the concealment marker system used in this paper are investigated.

In usability aspects, 74% of the participants rated the concealment markers in dark environments very useful (Fig. 10(a)). In addition, the quality of the concealment marker was evaluated to be higher than that of the conventional invisible markers (Fig. 10(b)). And 72% of the evaluators suggested that an augmented reality system with a concealment marker could be recommended to others (Fig. 10(c)).

Fig. 10.

Survey of result on Usability

The evaluators evaluated that the concealment markers used in this paper are superior in performance and quality to the conventional invisible markers. Also, as shown in Fig. 11, the markers applied to the children’s storybooks used in the demonstration were evaluated very positively for the contents of the hiding augmented reality contents.

Fig. 11.

Question10. What do you think of the cloaking marker augmented reality applied to children’s storybooks?

5 Conclusion and Future Work

We analyzed and calibrated the printer colors of the two printers to reduce color differences, and consequently, hid the markers in the images. In this paper, a user evaluation was conducted to confirm the usefulness of a concealment marker. The user evaluations were performed to compare the proposed method with the conventional infrared absorbing ink method. And they perform evaluation and analysis on the concealment marker that we proposed. The survey results showed that there was a positive response to the AR content with the concealed marker and the rejection of the concealed marker decreased compared to the existing markers. In the future work, we plan to improve the color correction algorithm and find out how to apply it. Utilization research studies will be conducted to print pictures and concealment markers on different types of paper surfaces or clothing.