1 Introduction

1.1 Augmented Reality

Augmented Reality (AR) is the name given to the process of overlaying virtual and synthetic information on to the live world. Application areas include navigation, advertising, entertainment and education. With the convergence of technologies in modern mobile phones, accessibility to hardware that can support augmented reality has never been easier.

An example of AR applications includes Golfscape GPS Rangefinder by Shotzoom Software that uses the Global Positioning System (GPS) and compass within your mobile phone to provide golf course summary and information. This information is displayed when you look at the course through the inbuilt camera on the phone (Fig. 1).

Fig. 1.
figure 1

Example of AR application – Golfscape rangefinder by Shotzoom software

Full size image
Fig. 2.
figure 2

Drakerz-confrontation by Drakerz

Full size image

1.2 AR Games

Being able to mix real world and virtual environments can introduce interesting game play elements into the real world. Up until recently, the technology required to implement AR in a mobile gaming environment hasn’t been accessible to the general consumer. However, with the rapid advancements in mobile technologies, in particular high quality camera’s, accurate geo-location and increased processing capabilities on the mobile device, games that mix real world locations with innovative game play are beginning to emerge. Some examples include:

  • Google’s Ingress (iOS), a game that involves ‘hacking’ real world locations to gain control of portals for your faction. This game uses the players position, orientation and camera to overlay game play information onto the mobile device screen.

  • Drakerz-Confrontation, a Pokémon style game where the creatures ‘pop off’ the cards. This game uses the phone camera to capture and correlate the location of the card and place a 3D character model of creature on top of the card as soon through the phone (Fig. 2).

Current generation AR based games either use broad location settings, which are accurate to within a few meters, or fiducial markers, high contrast, distinct patterns that computer vision algorithms can accurately determine orientation and position to correlate between the real world and game worlds.

2 Ghost Hunter

Ghost Hunter is a short, augmented reality, game in which the player’s objective is to search around their real-world environment looking for ghosts. The game was designed to be fully mobile but retain some gameplay physical characteristics of a first person shooter as the focus is on a complete game that you can use in any real world indoor location and interact with the environment around you.

2.1 Design Challenges

As briefly discussed earlier, most AR games on the market rely on one of two technologies to track user movements and align the game world with the real world.

The first set of technologies rely on using the inbuilt location tracking of the phone or mobile device. Although accuracy does vary with each device, a position accuracy of less than a couple of meters is about average. Combined with the inbuilt compass (and potentially inertial sensors), the device is able to understand its position and orientation. This approach works very well outdoors, and has been used as the basis of navigation for the last ten years, however it doesn’t enable real time, accurate position tracking required for real time gaming. In addition, the system itself is not able to generate information about its environment so is entirely reliant on pre-built information.

The second set of technologies relies on a computer vision technique (often referred to as Fiducial markers). A fiducial marker is an object or image placed within the scene which the image system can use as a point of reference or measure. For instance, within the Drakerz-Confrontation game detailed above, the playing cards are used as the reference marker. The system knows the exact dimensions of the card and from the pattern is able to determine the orientation and position of the card within the image and from that extrapolate the position of the device in relation to the card. This allows the system to accurately overlay game world features over their real world counterparts. The system often has to recalculate and redraw the scene many times per second in order to maintain a stable image.

For Ghost Hunter we required the ability to not only know the accurate location of the device but also for the game environment to be able to interpret the real world around it. The preferred method for being able to generate this understanding of the real world location is to use 3D depth sensor. A number of depth sensors are available:

  • Structured Light sensor – Illuminates the real world with a structured IR light pattern. From the size of the dots and the pattern the system can determine 3D geometry.

  • Time of Flight Camera – Depth is measured based on the time it takes for an IR beam to transmit, reflect and return to the sensor. A scene can be built by building up this depth information over time.

  • Stereo – An image is captured from two cameras a small distanced apart and by comparing the two images depth and 3D geometry can be determined.

Combining these methods can enable for very accurate real time capture of the real world environment in a number of different lighting conditions (Fig. 3).

2.2 Google’s Project Tango Yellowstone Tablet

Google released its Project Tango “Yellowstone” tablet in June 2014 marking one of the first consumer devices to combine both depth sensing and color camera within a single mobile device.

Fig. 3.
figure 3

Google’s project Tango Yellowstone tablet

Full size image

The Yellowstone is a 7-inch tablet that features a 2.3 GHz quad-core Nvidia Tegra K1 processor, 128 GB flash memory, 1920 × 1200-pixel touchscreen, 4MP color camera, fisheye-lens (motion-tracking) camera, integrated depth sensing, and 4G LTE connectivity.

Tango uses a combination of motion tracking, depth perception and area learning to accurately map the physical location around the player and position the tablet exactly within the real world space. With this information we have the potential to accurately generate and incorporate, in real time, physical spaces into the game environment without having to spend significant time and money (Fig. 4).

Fig. 4.
figure 4

Point cloud image captured by Project Tango

Full size image
Fig. 5.
figure 5

Top shot elite firearm controller, modified to hold tablet

Full size image

Ghost Hunter was a short game designed to explore using real time real world map generation as a game play mechanic.

2.3 Top Shot Firearm Controller

For the game controller, the Top Shot Elite Firearm Controller from Activision was selected. This controller includes built in wireless technology, twin analogue sticks, D-Pad, action buttons as well as a pump action feature (Fig. 5).

The Top Shot was modified to accommodate the Yellowstone tablet as a ‘ghost sight’, and a wireless interface was created to pick communicate between the gun and the tablet.

2.4 Gameplay

The player uses the Yellowstone’s Depth Perception camera to ping a surface in front of them, such as a table, floor, wall, etc. and check if there’s a ghost inside. If the ping sensor is red, a ghost will appear and the ghost(s) will be forced out of the surface. The ghost(s) will then approach the player, and can be avoided by moving around in the environment, as well as use the gun to shoot them down.

The player scores points by successfully shooting the ghost and loses points if the ghost manages to reach the player. The player is also able to move within the physical space to avoid the ghost creating a dynamic environment where the player has to be conscious of not only the game space but also their physical surroundings. Each round is timed and the speed, number and health of ghosts can be increased with difficulty (Fig. 6).

Fig. 6.
figure 6

Ghost Hunter play through

Full size image

2.5 User Interface

The player is presented with a simple screen that details the time left in the current round, current score and players all time hi-score. In the middle of the screen is a red reticle which displays the current aim point of the player (Fig. 7).

Fig. 7.
figure 7

Early user interface for Ghost Hunter

Full size image

If the player moves the gun around the environment and presses the ping button (x on the controller) a small circle is drawn on the real world surface. This circle represents a search area and will remain gray if no ghost is present, but turn red if a ghost has been detected. After a small amount of time, a ghost will spawn out of any red circle areas and turn and advance towards the player. The player has to shoot the ghosts before they reach their location and each ghost takes a number of hits depending on the difficulty. The player is able to ‘super charge’ their weapon by using the pump action and is physically able to move around the room and dodge the ghosts in order to avoid being hit.

2.6 Challenges

During game development a number of challenges were encountered that highlighted some potential issues with our approach:

  • Accurately determining the direction of physical surface in relation to the camera so that the ghost perpendicularly floats out of the surface proved to be a demanding task. The Tango builds up a complex model of the room over time and often starts with incomplete information, so occasionally our system would either not locate a surface or misinterpret a surface so the ghost would come out at a strange angle. The longer the Tango was able to build the depth image then the less likely there was of errant ghosts.

  • Changes in lighting within the environment conflicts with the depth sensors. IR doesn’t work as well outdoors, in sunlight or in a very bright room and depth information can be degraded in these conditions. In addition, dark surfaces also will reduce the quality of the depth information again leading to strange ghost behavior.

  • In order to reduce processing requirements, the game would only interrogate the 3D model of the environment when a pulse was fired from the gun. However, this meant that if you moved around the room physical objects wouldn’t obscure the ghost as it advances towards you leading to a break in immersion. Moving to a design where the environment is interrogated each frame would allow real world objects to obscure the ghosts, even when in flight allowing players to truly interact with their environment (such as hiding behind the table). However, this significantly increases processing requirements and can reduce frame rate.

3 Conclusion

Although very simple, Ghost Hunter shows a good example of being able to mix virtual and game play elements with the real world. By exploiting the feature rich functionality of the Tango tablet and integrating with a commercially available gun controller we were able to rapidly produce a short game that enables you to hunt ghosts anywhere.

This short project has demonstrated that AR games can now be developed on today’s mobile platform and with the continued advancement in mobile technologies and computer vision techniques we will see a proliferation in AR based game play and games.