Keywords

1 Introduction

The problem in verifying ticket holders is how to simultaneously verify identities efficiently and prevent individuals from impersonating others at a large-scale event at which tens of thousands of people participate. To solve this problem, we previously developed two ticket ID systems that identify the purchaser and holder of a ticket by using face-recognition software, which require ID equipment such as a tablet terminal with a camera, card reader, and ticket-issue printer [1,2,3]. Since the two systems were proven effective for preventing illegal resale by verifying attendees at large concerts of popular music groups, they have been used at more than 100 concerts. However, simplifying the ID equipment is necessary from an operational view-point. It is also necessary to secure the cooperation of individuals in obtaining facial photos from a technical view-point of face-recognition accuracy because face recognition fails when unclear facial photos of individuals are obtained, i.e., when the individuals have their eyes closed, are not looking directly forward, or have hair covering their face. This paper proposes an identity-verification system that uses attendees’ selfies as input photos for face recognition, which simplifies ID equipment by using the attendees’ smartphone cameras and secures clear facial photos from attendees by allowing them to take their own photos.

2 Methods

2.1 Identity-Verification System Using Face Recognition from Selfies

The proposed system solves the following problems with conventional systems:

  1. (1)

    Simplifying ID equipment

Attendees open an identity-verification app with their smartphone cameras to show a venue attendant the verification result, as shown in Fig. 1.

Fig. 1.
figure 1

Admission by using identity-verification app

Full size image

The app recognizes attendees’ selfies and makes it possible for them to check-in. It is not necessary for an event organizer to prepare card readers for check-in, cameras for taking facial photos of attendees, tablet-terminals for face recognition, and ticket-issuing printers.

  1. (2)

    Securing clear facial photos

Selfies are helpful for securing clear facial photos because it is possible for attendees to take their own photos with the identity-verification app. The app helps attendees take acceptable photos by showing their registered facial photos as good examples together with an instruction message, as shown in Fig. 2 (left). The message suggests that they should directly face the camera without having their eyes closed or having their face covered by their hair. They are able to re-try if they were not successful.

Fig. 2.
figure 2

Self-photographing screen (left) and soft-flash screen (right)

Full size image

2.2 Requirements for Using Selfies

  1. (1)

    Pre-screening of Attendee’s Operational Skills

All attendees do not possess smartphones and do not always have sufficient skills to operate a smartphone camera for selfies. Therefore, attendees who would like to enter an event venue with the app have to be verified as to whether they succeeded in identity verification with the app in advance of the event. Successful attendees are allowed to verify themselves by using their selfies with the app on the day of the event. Unsuccessful attendees as well as those who do not want to use the app at an event site are arranged to be verified with conventional systems. The pre-screening makes it possible for an event organizer to estimate the necessary equipment and number of venue attendants by figuring out the number of attendees verified with the proposed and conventional systems.

  1. (2)

    Preventing impersonation

Selfies should be checked when and where they were taken because it is possible for attendees to use ticket winners’ smartphones with which identity verification succeeded by using the winners’ selfies in advance. Therefore, the app is designed to verify the time and location of selfies, i.e., whether the selfies were appropriately taken at the right place and time of the event, by using the built-in clock and GPS of smartphones.

  1. (3)

    Self-photographing under dark conditions

Face recognition from selfies at an event site is impossible when it is too dark to detect facial areas. Though flash photography is useful for face recognition under dark conditions, the front-facing built-in cameras used for selfies are not currently equipped with a flash. Therefore, the app provides a soft-flash screen for selfies under dark conditions, which has a wide white area except for a small viewfinder with the highest brightness, as shown in Fig. 2 (right). The soft-flash screen makes it possible for attendees to execute face recognition by illuminating their faces.

2.3 Identity-Verification Procedure

The proposed system meets the above-mentioned requirements according to the following operational procedure from the first step of ticket application to the last step of admission supported by the app:

Step 1

Individuals applying for tickets register their membership information as well as their facial photos in the same way as with conventional systems [1,2,3]. The photos are stored in the membership database.

Step 2

After an event organizer notifies ticket winners, i.e., successful applicants that have been selected, the winners can download the identity-verification app. The organizer gives permission to attendees who succeeded in the identity verification by using their selfies with the app in advance of the event. The permitted attendees can enter the event venue with the app.

Step 3

The permitted attendees open the app to execute face recognition by taking their selfies at a specified date and place. They show a venue attendant a message displayed on their smartphone when the verification was successful, as shown in Fig. 1. Checking the execution date and place prevents attendees from borrowing or obtaining ticket winners’ smartphones with which identity verification succeeded in advance.

Step 4

A venue attendant carries out the admission procedure in accordance with a message on the smartphone display. When the attendant invalidates a ticket on the attendees’ smartphone, the same way as an electronic tearing ticket [4], it is reported to the event organizer.

2.4 Configuration of Identity-Verification App

The identity-verification app consists of four modules, i.e., face recognition, time-location verification, identity-verification control, and check-in, as shown in Fig. 3. In the face-recognition module, the face-photographing function makes it possible for attendees to take selfies even under dark conditions by means of a soft-flash screen. The face-recognition module stores the encoded facial image of the attendee that is registered at the time of ticket application. The module collates the selfie with the registered facial photo then transmits the recognition result to the identity-verification control module together with the selfie and registered photo. When the face-recognition module transmits a signal of face-photographing to the time-location verification module, the module extracts the time and location data. The module checks whether the extracted data are consistent with the pre-stored time and location data of the event at which an attendee can participate as a ticket winner. After the check, the module transmits the verification result to the identity-verification control module. The identity-verification control module determines that the verification result is successful if both results of face-recognition and time-location verification are successful. Otherwise, the control module determines that the verification result is a failure. The control module transmits the result to the check-in module together with the selfie and registered facial photo. The check-in module stores the attendee’s data, such as name and seat, in advance. This module generates a success message from the attendee’s data, the transmitted selfie and registered photo, and a sentence telling the attendee to enter the venue if the verification was successful. Otherwise, it generates an unsuccessful message from the attendee’s data, the transmitted selfie and registered photo, and a sentence prompting the attendee to re-try identity verification. When a venue attendant carries out the admission procedure on the attendee’s smartphones, the app transmits the attendee’s check-in information to the event organizer as well as expressing an appreciation message on the display.

Fig. 3.
figure 3

Configuration of identity-verification app

Full size image

3 Preliminary Tests and Results

3.1 Face-Recognition Parameters

The proposed system should be evaluated before the identity-verification app is used for actual events from the view-point of face-recognition accuracy. Though time-location verification is reliable because commercial smartphones have practically proven results, face-recognition accuracy has to be scrutinized regarding the feasibility under actual event conditions because attendees take photos of themselves using their smartphones in various environments. Face recognition is controlled using intrinsic, extrinsic, and operational parameters [2]. The intrinsic parameters are due purely to the physical nature of the face and are independent of the observer. They include age, expression, and facial paraphernalia such as facial hair, glasses, and cosmetics. Extrinsic parameters are related to the appearance of the face. They include lighting, pose, background, and imaging such as resolution and focus. Operational parameters are related to the interaction between attendants and attendees. They include how many times the face-recognition process should be repeated per attendee until his/her identity is verified, whether an attendee should stop for the face-recognition process, and whether an attendee should face the camera. The proposed system makes it possible for attendees to control the intrinsic and operational parameters using selfies they have taken. However, attendees are not able to control several extrinsic parameters such as resolution, lighting, and background. Though resolution is not a problem for a commercial smartphone from the view-point of face recognition, it is necessary to evaluate face-recognition accuracy under actual venue conditions with regard to brightness and background conditions.

3.2 Test Methods

The identity-verification app was developed based on the tablet-based face-recognition system [1, 2]. It can be installed on smartphones commercially available in Japan with Android OS and iOS. As preliminary tests, 30 examinees executed face recognition with the app under different brightness and background conditions. The examinees carried out the tests according to the following steps:

Step 1

Examinees registered their membership information as well as their facial photos. The photos were stored in the membership database.

Step 2

In the same way as ticket winners, they could download the identity-verification app on their smartphones. They were permitted to operate the app at any time and any place for the tests. They had their operational skills pre-screened after downloading the app.

Step 3

They started the app to execute face recognition under the following two conditions: bright enough to detect their faces and too dark to detect them. Under the dark condition, they used the soft-flash screen for face recognition. Both conditions contained four backgrounds, i.e., indoor, outdoor, crowds, and under-umbrella. This means that all the examinees tested eight selfie patterns, i.e., taken under the two conditions multiplied by four backgrounds. The total number of selfies were 240, i.e., 8 patterns multiplied by 30 examinees.

3.3 Results

The identity-verification app was downloaded and operated normally without any problems by all examinees. The face-recognition accuracy was 97.5%, (the false reject rate was 2.5%); 6 photos failed in face recognition among the 240 photos. Table 1 lists the results in the form of the number of failure photos/total number of photos with regard to the two brightness and four background conditions. One failure under the bright outdoor condition was due to the examinees closing their eyes during photographing. There were no failure photos due to the fact that examinees were not looking directly forward or that they had hair covering their faces. Five failures in crowds were due to face-detection errors. Since the five photos contained several people behind the examinees, faces of the different people were detected for face recognition. There was no failure photo among those taken with soft-flash or under-umbrella.

Table 1. Face-recognition results under various brightness and background conditions
Full size table

4 Discussion

4.1 Problems with Conventional Systems

The identity-verification app was downloaded and operated normally for the preliminary tests with the commercial smartphones of all examinees. It was not difficult for the examinees to operate the app. The face-recognition accuracy was 97.5%, which is higher than that of the conventional systems [1,2,3]. This could be helpful for simplifying ID equipment in comparison with conventional systems. Though one facial photo happened to be unclear, the others were all clear for face recognition. Therefore, selfies are regarded as helpful for securing clear facial photos.

4.2 Background Conditions

Face detection exhibited a problem in that faces of incorrect people were detected when selfies contained other people behind the examinees. It could be practically solved by choosing the face with the largest face area among all the detected faces. The detected face areas could be equal when two people are photographed abreast intentionally. The app will be improved with the re-try function with a message telling attendees to take a photo of one person again when it detects same-sized faces.

4.3 Brightness Conditions

The soft-flash screen made it possible for attendees to execute face recognition even under conditions in which it was too dark to detect their faces. In general, brightness is more than 1000 lx in bright offices such as department stores, 750 lx under shopping arcades at night or just after sunset, 200 lx under street lights or in bedrooms, and 30 lx in moonlight or candlelight [5]. The soft-flash screen provided a brightness of more than 80 lx for face recognition. Since the soft-flash illuminated the face of only the person close to the smartphone screen, it prevented the detection of those of other people in crowds.

4.4 Future Issues

The preliminary tests could ensure the feasibility for simplifying ID equipment by using selfies with attendees’ smartphones. They also clarified that selfies are helpful for securing clearer photos than with conventional systems. After improving face detection, we will address the following issues before applying the proposed system for actual large-scale events:

  1. (1)

    Time of identity verification by a venue attendant

The proposed system is expected to decrease identity-verification time because venue attendants do not have to execute face recognition and ticket issuing. We are currently measuring how long attendants should spend for admission procedure per person in step 4 mentioned in Subsect. 2.3 to estimate the number and allocation of necessary venue attendants according to the event scale.

  1. (2)

    Rehearsal at actual event sites

Larger-scale tests are planned at actual event sites where the proposed system is expected to be used. All the modules of the identity-verification app will be checked as a rehearsal for actual events. At the rehearsal, we will install the proposed system as well as conventional systems in case of any problems that will make it impossible for attendees to use the identity-verification app. We are also developing operational guidelines for dealing with disruptive individuals.

5 Conclusion

We developed an identity-verification system for attendees of large-scale events using face recognition from selfies. The proposed system simplifies ID equipment by using attendees’ smartphone cameras and ensures clear facial photos from attendees by allowing them to take their own photos. The system achieved 97.5% face-recognition accuracy in preliminary tests of 240 photos taken by 30 examinees under various brightness and background conditions. The soft-flash screen made it possible for attendees to execute face recognition under conditions in which it would be too dark to detect faces. However, it is necessary to improve the face-detection method for choosing the foremost face when several faces are detected. We are developing clear guidelines for system introduction by estimating the number and allocation of necessary venue attendants through larger-scale tests at actual sites.