Keywords

1 Introduction

In the tourism domain, mobile and wireless technologies have been pointed out as one of the most influential technological innovation because of the fast growing of intelligent devices are in a huge user base [1]. In the past, tourists got traveling information only from few stationary channels such as tour guide books, television programs, and discussion among friends, so the information retrieving process could be monotonous and time wasting [2]. After the late 1990s, thanks to the popularization of the computer and the developing of the internet, most people can easily search for and get the traveling information by using internet connected computers. In the kind of Internet Content Provider (ICP) environment, traveling information were usually shared by formal content provider or website owner, causing the insufficient richness of information. After 2000, wireless communication and smart mobile device technology have grown rapidly, and mobile services have turned into mainstream and changed the traditional information environment from ICP into Wireless Content Provider (WCP). The change not only makes people interact with each other and get the information easily, but also revises the traditional tourists behavior and information service model, allowing the tourists to grasp the real-time and personal information anytime, anywhere [3].

In the era that mobile devices are taking over, more and more tourists search for travel information and make travel plans via mobile devices [4]. Mobile technology with the properties of ubiquity, timely, flexibility and localization frees the tourists from being restricted to passive and fixed-point information gathering [5], dramatically changing the information service model to be directly and personalized. There is a variety of mobile travel information services in the market. The services are categorized into before-travel, during-travel, and after-travel services based on when the users use them listed in Table 1. And the most emphasis is how to enhance the information interactive services, to fulfill tourists’ demand at the right place and right time. The smart mobile device is therefore become the most important connecting channel to this purpose [6].

Table 1. Mobile services in tourism domain
Full size table

The main task that most users perform via mobile devices is information search, especially for those which are in urgently needed. However, restricted by the screen size and the operation mode, searching for and browsing information via mobile devices is not as easy as via the traditional computers [7]. To resolve the difficulties in operating on mobile devices, the user interfaces, including information inputting, should be simple and easy enough [8]. In the past, researches have been focused on optimal design for inputting texts and searching via mobile devices. Recently, as hardware technology to smart mobile devices improves, non-contact inputting (such as, microphone, camera, or NFC wireless sensor technology) is becoming an alternative to text inputting and the new solution for information retrieving on mobile devices [9]. This paper will investigate the new travel information querying method, which combines mobile phone camera, image recognition and recommendation technology.

Almost all mobile phones come with an integrated camera or image acquisition device. Camera is typically used for taking pictures for posterity purposes; however, there are many other applications for which the images may be applied [10]. Instead of typing the key word to search for and get the information on mobile phone, tourists can now use a powerful image-based mobile search service, which functions by sending an image acquired by a phone camera to a server. The server hosts visual-based recognition and personalized recommendation engines, returns the personalized, appropriate search results back to the user, and helps the tourists know the attraction or finishes a suitable itinerary in convenience. The visual input in the real environment is a new and rich interaction modality between a mobile user and vast information [11], and it brings advantages as follows:

  • Compared to traditional text searching, the visual input saves a lot of time that typing may cost.

  • In most of the traveling conditions, the tourists are not familiar with the searching object. Visual input by camera can solve the main problem that users have no idea to come out an appropriate key word or suffer from the language problem.

  • Unique feature of image makes the searching results more precise than ever.

The tourism industry acknowledges that the launch of value-added mobile services is clearly identified as the main factors affecting the competitiveness of the tourism market [12]. Mobile phones with cameras present new opportunities and challenges for mobile information association and retrieval, and it is clear that visual information query will eventually be integrated with other mobile traveler supported services and change the way the tourism industry works today.

In this paper, we propose the framework of information query system, which is implemented by the prototype application (APP). The system consists of passive information query, active information query, trip scheduling, and information management. In the beginning, user browses information on APP. Furthermore, he/she can take a picture to get the corresponding information. Combined user’s preferences with location based service (LBS), the prototype recommends the proper trips to guide user easily. The rest of this paper is organized as follows: design and prototype implementation are described in Sects. 2 and 3, respectively. The resultants will be shown in Sect. 4, and the concluding remarks will be drawn in Sect. 5.

2 Design

The proposed prototype implements the framework of information query system for tourism. Figure 1 shows the block diagram of the proposed prototype. The details of functions in the prototype are introduced as follows:

Fig. 1.
figure 1

Block diagram of the proposed prototype

Full size image

  • Passive Information Query. In the conventional applications, theme-based query and context menu are common approaches to obtain information. Considerations of user’s preference and time consumption, seven types of themes have been set in the prototype. Those themes include “culture & heritage”, “gourmet guide”, “offshore islands”, “ecotourism”, “hot springs”, “LOHAS”, and “night markets”. User acquires information by pressing the theme of interest. For instance, when user presses the button of “gourmet guide”, he/she can browse the introductions of restaurants in Taiwan. Similarly, user clicks the item in the context menu to acquire the corresponding information.

  • Active Information Query. In order to provide friendly query, active information query are realized by two approaches, namely LBS-based query and recognition-based query. User clicks the button of “Nearby”, then, landmarks, restaurants, and exhibitions around user are marked on Google Map. He/she clicks the icon on the map to acquire the information of the spot. For recognition-based query, user clicks the button of “Scan” to take picture. Cloud server analyzes the shoot picture and then delivers the related information to mobile device.

  • Trip Scheduling. Pressing the button of “Quick Plan”, user picks the dates, the locations, and the themes of interest, and the prototype creates a new trip schedule against different periods. Shaking the mobile device, a new schedule is created. Furthermore, the other way to create trip schedule is referred to the result of recognition-based query. The recognition result is treated as a seed point of interest (POI) to create trip schedule.

  • Information Management. The tasks of information management have two: (1) systematically retrieval information via defined metadata, (2) flexibly update information. Especially for implementation of active information query, we flexibly add/delete the reference images and the information without altering the system.

3 Prototype Implementation

3.1 Information Management

For collecting and updating POI related information, a POI table is built and maintained. Figure 2 shows a screenshot of a partial POI table. This table integrates information collected from multiple sources and is used for further analysis and recommendation.

Fig. 2.
figure 2

POI table

Full size image

The structure of a record in POI table is shown in Table 2. For the x-th POI (denoted as poi x ), the denotations id(poi x ), name(poi x ), add(poi x ), and des(poi x ) represent the identity, the name, the address, and the description of poi x , respectively. The denotation ts(poi x ) is the expected number of hours that tourists spend on poi x , where ts(poi x )∈{1,2,3}. Each POI has its own business hours. In this system, a day is divided into three time intervals, i.e., period 1(8:00 ~ 11:59), period 2(13:00 ~ 16:59), and period 3(18:00 ~ 20:59). Three Boolean variables, namely vp 1(poi x ), vp 2(poi x ), and vp 3(poi x ), are used to indicate appropriate periods for visiting poi x , where vp i (poi x )∈{“true”,“false”}. The value of vp i (poi x ) will set to be “true”, if poi x is appropriate to be visited in period i ; otherwise, this value will set to be “false”. Each POI belongs to a kind of preference themes, such as foods, arts, natural, and shopping. We use prefer(poi x ) to denote the theme assigned to poi x and collect ratings, reviews, and tags for scoring POIs. The score of poi x denoted as score(poi x ) is a weighted sum of its number of browsed, saved, reviewed, and checked in on social media.

Table 2. Structure of a record in POI table
Full size table

3.2 Active Information Query

The prototype actively collects information of landmarks, restaurants, and exhibitions which are located around user. As we above mentioned, LBS-based query and recognition-based are two approaches for implementation of active information query. Figure 3 illustrates the block diagram of recognition-based query, and the query process to mobile device is described as follows:

Fig. 3.
figure 3

Flowchart of recognition-based query

Full size image

  • User takes a picture which is called as query image, and then the image is transmitted to cloud server.

  • After implementing image recognition, the application receives the information with respect to query image from the second database in cloud server.

The objectives of cloud server have two: (1) finding a reference image in database as like as the query image, and (2) delivering the information to the application. The query process to cloud server is described as follows:

  • Corner detection is applied to reference images and query image to find key pixels.

  • Extract features of key pixels. Histogram of oriented gradient (HOG) is employed to represent feature descriptor of key pixel [13, 14].

  • The first database stores feature descriptors of all reference images, and the information corresponded to the reference image are stored in the second database.

  • Compute feature similarity between the query image and the reference images. The most similar reference image is found with the maximum feature similarity.

  • The identity of the selected reference image (which is denoted as id(poi x ) in Table 2) is an index to acquire the related information in the second database. Then, the information is delivered to the application.

3.3 Trip Scheduling

Given date, location and user’s preference, the system recommends schedule of trip with respect to opinions on social media. The structure of schedule of trip is listed in Table 3. Each hour is regarded as a time slot. The three periods, period 1(8:00 ~ 11:59), period 2(13:00 ~ 16:59), and period 3(18:00 ~ 20:59), consist of 4, 4, and 3 time slots, respectively. Let slot(i,j) be the j-th time slot in the i-th period, and slotnum(period i ) be the total number of time slots in period i , where i∈{1,2,3} and j∈{1,2,3,4}. The rec(i,j) denotes the recommendation POI at slot(i,j). In our definition, a recommendation POI crosses one or more time slots. For each period, the seed POI is given by either recognition-based query or user’s selection.

Table 3. Structure of schedule of trip
Full size table

For example, the seed POI, “Taipei National University of the Arts”, puts into period 1, and it derives from the result of recognition-based query. Let tc and TH be, respectively, target county and preference themes, where TH = {theme k1, theme k2, …}. In period 2, given tc and TH, the system selects a seed POI (denoted as poi s ) which belongs to tc and TH with the highest social score. The selection of poi s follows the four conditions:

  1. (1)

    add(poi s ) ⊂ tc,

  2. (2)

    vp i (poi s ) = true,

  3. (3)

    ∃ theme k ∈ TH such that theme k (poi s ) = true,

  4. (4)

    ∀ poi x , poi x satisfies (1) ~ (3) ∧ poi x  ≠ poi s  → score(poi x ) ≤ score(poi s ).

Whenever poi s is selected, ts(poi s ) time slots will become occupied. For example, assuming that “Taipei Water Park” is the seed POI selected for period 2 and ts(“Taipei Water Park”) = 2. After selecting “Taipei Water Park” as the seed POI in period 2, the time slots slot (2,1) and slot (2,2) become occupied, and the values of rec (2,1) and rec (2,2) become “Taipei Water Park”.

The system continues selecting subsequent POIs for filling the schedule. As the last occupied time slot in the schedule of period i is slot(i,l), and rec(i,l) = poi a . The next POI, denoted as poi b , is selected according to the following conditions:

  1. (5)

    add(poi b ) ⊂ tc ∧ distance(poi a , poi b ) ≤ 3 km,

  2. (6)

    vp i (poi b ) = true,

  3. (7)

    ∃ theme k ∈ TH such that theme k (poi b ) = true,

  4. (8)

    l + ts(poi b ) ≤ soltnum(period i ),

  5. (9)

    ∀ other unselected poi x , poi x satisfies (5) ~ (8) ∧ poi x  ≠ poi b  → score(poi x ) ≤ score(poi b ).

The function distance(poi a , poi b ) is defined as the geographical distance between poi a and poi b . The condition (5) makes sure that the distance between two POIs is acceptable. The condition (8) ensures that the total number of time slots spent by all recommendation POIs will not exceed slotnum(period i ). The system continues selecting subsequent POIs until either all time slots in period i are occupied or there is no POI satisfying the conditions of (5) ~ (9).

4 The Resultants

The prototype application employed the Qualcomm Vuforia software development kit (SDK) to implement feature extraction and feature matching [15]. The APPs for Android and iOS can be downloaded from [16, 17], respectively. Three resultants were emphasized on and presented in this paper, including LBS-based query, recognition-based query, and trip scheduling. In what follows, the resultants are introduced in detail.

4.1 LBS-Based Query

Pressing the button of “Nearby” in the main frame, the application immediately presented the related information of landmarks, restaurants, and exhibitions around the user, which is realized based on LBS. Figure 4(a) shows the map marked the icons after pressing “Nearby”. Figure 4(b) and (c) depict the detailed information of two spots corresponded to the red icon and the blue icon in Fig. 4(a), respectively.

Fig. 4.
figure 4

LBS-based query: (a) the result map marked the icons of landmarks, (b) the information of Keelung corresponded to the red icon in (a), and (c) the information of Memorial Park corresponded to the blue icon in (a) (Color figure online).

Full size image

4.2 Recognition-Based Query

Pressing the button of “Scan” to shoot a picture, the application transmits the shot picture to cloud sever and then receives the related information after image recognition. Figure 5(a) and (b) show the query image and the corresponding information, respectively. Figure 5(a) is the image shot in the activity handbook. This function was practically verified through the activity of Taipei National University of the Arts in October 2014, and the activity called as “Kuandu Story Guidance”.

Fig. 5.
figure 5

Recognition-based query: (a) the image shot in the activity handbook of “Kuandu Story Guidance”, and (b) the activity information.

Full size image

4.3 Trip Scheduling

Given date, location, and user’s preference theme, the application planned a schedule of trip. Furthermore, the prototype designed to generate another trip schedule by shaking mobile device. In the case of Fig. 6(a), the user set a two-day trip in two locations, and he/she was interested in the theme of gourmet. Figure 6(b) shows two different schedules of trip under the same initial settings. It is obvious that the right-side schedule has one spot more than the left-side schedule has.

Fig. 6.
figure 6

Trip scheduling: (a) frame of initial setting, and (b) two schedules of trip under the same initial settings.

Full size image

5 Conclusions and Future Work

This paper introduces the framework of information query system, and it is realized by a prototype application. We propose a new travel information query scheme, which combines image acquisition device, image recognition, and recommendation technology. The resultants demonstrate that four functions run efficiently to provide user travel information, and those functions include passive information query, LBS-based query, recognition-based query, and trip scheduling. The future work will focus on how to actively provide user more appropriate information by analyzing user’s query behavior.