1 Introduction

Designing smart city services is a challenging and difficult process. First of all, there is the complexity of the urban context that exists out of an interplay of users, diverse types of technologies and other influencing urban factors, resulting in diverse usage contexts. These pose new challenges for designers in creating services and constructing interfaces that consider and simultaneously respond to this. Secondly, as these smart technologies evolve towards more personalized and interconnected tools, interactive modalities representing these novel functionalities in urban environments are needed and must be considered in order to fully explore the potential of new smart urban services [1].

This challenge has been gaining attention in (HCI) research, with various suggestions to tackle these gaps [2, 3]. The problem with existing research is that the solution is often seen in high-fidelity prototypes, due to their advanced interactive modalities and direct interactions. Hence, they demand quite some time in development and are implemented rather late in the design stage when most functionalities are already set and fixed. At this later stage, modifications would already imply extra costs in time and money and thus arrive with a high level of reluctance.

Next to this, the development and shaping of these high-fi prototypes is still preferably done in the safe and closed environment of the company or the lab environment. Opting for this design trajectory means often a trade-off against early user testing, currently performed by paper sketches or mock-ups. As more ‘smart’ and ‘personalized’ technologies are being used in the urban context, and city services are becoming smart, the involvement of these end-users is becoming of ever higher importance.

In this paper, we present an approach where we combine lo-fi prototyping with the philosophy of PTA, to support the early stage of the design process. This “live-prototyping proxy” seeks to enable researchers to study HCI not only in the very early design stage, but also to provide more realistic and contextual feedback, answering to the challenges referred to in the first two sections. Illustrated by a smart city pilot use case, we will describe the benefits of using live-prototyping proxies for HCI research in the development of complex urban IoT-related services.

We start by addressing current approaches that aim at capturing early and contextual feedback. We will emphasize on the importance and opportunities which early contextual feedback brings, and secondly describe how current early design approaches try to respond to this importance. Thereafter, we proceed by describing our live-prototyping methodology, providing the context and background details that motivated this approach, followed by a case study. Finally, results and our methodology impact on the design process will be discussed and future research directions will be delineated.

2 Early and Contextual User Feedback

In recent years, the importance of involving users in the design process of a technology has grown. In particular, the inclusion of users in product development processes is described as a well-suited approach to increase the “fit-to-market” qualities of an innovation [4, 5] and to ensure smooth integration into users’ habit and everyday lives [6, 7]. Different empirical studies have therefore proven the impact of user involvement on product success, and state that this is most successful in the early stages of the product development (idea generation and specification) [8,9,10,11].

Early user feedback not only allows effective means for defining user requirements, it also proves to be cost-effective as it lowers risk and unneeded effort in design [11]. At the same time, early user feedback generates early identification and resolution of usability problems [12]. This makes it also time-effective, as it provides researchers with something tangible to start with, and allows clear directions towards the further trajectory of the product. Moreover, it helps in finding and defining end-users of the device, steers engagement along the way and assures the understanding of the product [13].

However, this feedback will be most efficient and reflect reality when captured in the context of use. As Hastreiter et al. [14] point out, “focusing on early development stages, collecting functional requirements from prospective users is not sufficient to develop adequate design solutions. A detailed understanding of the user’s task, context, aims and workflow in which the product will be embedded is essential to adapt the design to his needs.”

Kjeldskov and Skov [15] indicated already in 2003 that we needed to break with the traditional lab context and techniques. Accordingly, they suggested to evolve towards an expansion of testing mobile technologies that focus on capturing and monitoring contextual data by means of what they call mobile laboratories [16, 17]. Today we see that HCI researchers have adopted this vision. More and more we see approaches where interactive prototyping tools are tested “in-the-wild”, in which the design and evaluation process goes beyond the lab environment and performs context-based, mobile interaction design [18]. Testing technologies, devices or services in the context of use provides the consideration of external influences and illustrates how devices and technologies influence users’ interaction and experiences, such as e.g. environmental effects [19]. These are valuable insights that cannot be measured in a closed, indoor setting or pre-scripted scenario testing. Also, taking the HCI research approach into an open, urban environment has the advantage to capture different contexts of use [20]. An important element here is serendipity. In normal lab conditions, it is impossible to draft each type of use case scenario and interaction upfront, live testing, in contrary, enables one to discover various, unforeseen types of use.

3 Current Approaches

In various user research domains, there is the growing need for adjusted methodologies and tools that can aid a multidisciplinary team of researchers and designers to collect this early realistic and contextual feedback. In order to support the latter and to overcome existing limitations, we have investigated whether the combination of early lo-fi prototyping with the philosophy of PTA allows for the implementation and testing of a tool, which can be used to inform further development and involve end-users of smart city services at early stages. This to offer insights to HCI research which will, if not already, be increasingly confronted with the difficult realities that smart urban technologies and services imply.

Different from high-fi, lo-fi prototypes imply raw material such as paper or other sketch-based prototypes and are focused on providing quick iterations and validations of a first drafted design. These are beneficial to capture early user feedback as they are easy to build, modify, use and detect errors with. Examples of such tools are DENIM, SketchiXML or UISKEI [21]. Nevertheless, main points of critique are their lack in interaction, attractiveness and “on-point functionalities”, hence missing out on real user experiences and context of use. Over the years this has evolved towards more interactive lo-fi mockups, often simulated by Wizard-Of-Oz techniques, where the underlying system is controlled by the researcher [22, 23]. Although they already offer some kind of interaction, they still remain controlled by the researcher and often imply a low-quality substitute of the real device, with no ability of testing non-functional requirements (NFRs) like privacy, reliability and operability.

When lo-fi prototypes are not yet available, one can use the Proxy Technology Assessment method to observe the interaction between human and technologies. This method might be less known and comes forth out of the domestication theory [24]. Within this method, (existing) off-the shelf technologies are being used in order to simulate future functionalities of the future device. These are introduced to the user in their daily setting for a certain period of time [25]. The latter derives from the philosophy that the introduction of a technology to an everyday setting is the start of a dynamic process wherein actors, technology and context are exposed to continuous change until they reach an equilibrium and the adoption is more or less stable. Proxy technologies have therefore been used to define social requirements and the place of these technologies in users’ household. By doing so, researchers can study the embedded practices these devices evoke, and capture changes in user behavior and social structures and motivations [26].

4 Live-Prototype Technologies

As we present it, the live-prototype tool aims to capture direct user feedback and gain insights on the technology development process. This is based on solid data deriving from the IoT-technologies incorporated in the device. Due to new low-threshold technologies such as 3D-printing, it has become quite fast and easy to develop a proxy as “an ideal proxy”. Not only can it already resemble specific design elements, it can also incorporate basic functionalities, combined with real-time monitoring and feedback mechanisms. This latter enables one to capture the contextual parameters in which the interaction occurs. By taking this prototype live in the wild, we can evoke, capture and evaluate the realistic experience of the end-user.

To develop such a live-prototype, some elements are required. First, there should be the ability to work in a multidisciplinary team with developers, designers and user researchers. Second, there ought to be context-aware technologies placed in the device to capture the contextual data, such as e.g. location tracking. Thirdly, users should have the option to provide direct feedback. Applying the experience sampling method is in that sense something that is within reach and favorable. Finally, the prototype should be flexible and adjustable so that an iterative design and testing process can take place.

These “live-prototype proxies”, should in the end cover an approach that tests usability, functionality, convenience and ease-of-use, while also covering a more “human actors” approach that evaluates aspects like social context, user experience, value and meaning. Furthermore, this “live-prototype proxy” distinguishes itself with the philosophy that testing in the field can already offer huge value even if there is no fixed idea of the future product. Such live-prototype proxy can be applied to research, to explore, design and shape new interaction devices to enhance products and services in an urban interactive context.

5 Live-Prototype Proxy – Citizen Bike

To demonstrate the use of live-prototyping proxies, we evaluate the Citizen Bike project, a pilot that is part of a larger smart city project in the City of Antwerp, BelgiumFootnote 1. This pilot was an experimental set-up that focused on exploring how the cycle experience within a city could be improved with the support of IoT-technologies. Instead of working with a predefined concept, we started from the user experience. How do citizens experience cycling in the city? Are there contextual elements that influence this experience, which are they? What needs do cyclists have? To answer those questions, insights on how and why citizens were cycling were needed. Along with insights to inform the design and shaping process of a potential cycling service. The targeted outcome of this pilot was namely to shape an idea by means of an ‘undefined’ live-prototype and by so move towards a first enriched and validated proof-of-concept.

For our “live-prototyping” methodology we used the process of Bleumers et al. [25] as a generic guideline, which are (1) the preparation phase, that involves both the preparation of the prototype as the recruitment of the participants (2) the distribution of the proxy device and last (3) the studying of the proxy technology use, via data triangulation. It is important to note that during this process we have made use of different philosophies deriving from both the HCI as the Living Lab world, among them the importance of iteration and the involvement of the quadruple helix.

5.1 The Live-Prototype Proxy

The first step of the preparation phase implied a desk research and an initial ideation session (with prospective end-users). These were set up to gain knowledge on the future technology, its characteristics, its users and the setting for which the technology aimed for. Within these sessions, we focused on defining cycling profiles, identifying existing urban cycling services and collecting insights on general cycling experiences related to the city of Antwerp. Based on these, a set of primary functionalities for the initial live-prototype were defined, with among them location tracking, a gyroscope, an accelerometer and noise sensors. This in order to capture objective data on the participants’ cycling context, such as routes, speed, location etc. For the user feedback, two interaction buttons were foreseen.

Instead of using a mixture of off the shelve technologies, as normally used in PTA, we developed our own, dedicated live-prototyping proxy. This resulted in a special 3D printed case, in which the abovementioned sensors were integrated, and which could be mounted on a bicycle. The prototype was designed in such a way that cyclists could use it for at least a week without additional charging. By means of LoRa technology, the data was not only available in real-time, but also no additional connection with the users’ smartphone nor the traditional cellular network was required. The latter was important to increase the battery lifespan and to avoid bad coverages.

The design itself was kept deliberately lean and simple so that first of all, these prototypes could be built as quick as possible and no time was lost in design steps (at least not in this stage of the pilot) and secondly, so that users could adequately provide feedback on what the proper design should be. For that reason, the interaction buttons we had foreseen (that provided the user with the ability to give any form of signal or feedback) were kept agnostic. They were labelled with * and # so that the meaning could be changed based on the users input. In line with the lean and agile approach and the objective of such live-prototype, only a limited set of devices were created. In total ten devices were produced of which six were distributed to a live test-panel, while the four others were used for internal testing.

5.2 Participants – Recruitment and Intake

For the initial test-pilot, six citizens were recruited with the profile of cycling commuter, varying in age and gender. These citizens were given the assignment to use the prototype for a period of two weeks and to participate in different user feedback sessions, such as an intake workshop, questionnaires and a focus group discussion. For this participation, each citizen received a coupon with the value of 20 euros as incentive.

During the first workshop, which we framed as an “intake session”, a twofold objective was obtained: first of all, the objective was to learn more about the participants’ cycling experience in the city. Second, was to capture the users first feedback on the live-prototyping proxies. For the first we elaborated on the participants’ cycling profiles by looking at their habits. We let them draw their city cycling routes on a map and let them evaluate their daily routes. For the second objective, we let the participants experiment themselves with the device, i.e. by turning it on and off, by pushing the buttons etc. First reactions, technical difficulties and additional questions were captured on the spot. This information was of high value as it taught us that e.g. privacy issues, the effect of weather conditions and possible theft of the device were of high concern. The last part of the session consisted of the mounting of the device onto the participants’ bicycles, as can be seen in Fig. 1. Being able to perform this with the live-prototypes, we could collect first impressions on the device in the context of use, such as the implications of different bicycle types, the users’ knowledge and skills and unforeseen design issues with the mounting kit (e.g. the clickable connection…).

Fig. 1.
figure 1

Live-prototype proxy mounted on a users’ bike

Full size image

5.3 The Pilot

After the intake, the users were free to use the live-prototype for two weeks in their regular daily life. Within PTA research, this is usually 4 weeks of testing in order to allow the users to get sufficiently acquainted with the technology before being able to optimally experience the different functionalities. In this case, we opted for a two-week testing period for several reasons: first of all, the live-prototype was limited in terms of direct user interaction and feasibility, secondly, an intense feedback interaction loop with the user was foreseen and third, there was the importance of the iterative process.

During these two weeks, the use of the live-prototyping proxy was monitored. This was done via automated logging deriving from the sensor data, which provided high accuracy on the routes cyclists were taking and the input they provided via the buttons. This offered us insights in daily use of the device over time and space, yet also allowed us to identify cycling patterns, with which we could combine other sensor data such as noise, and possibly identify relations as is shown in Fig. 2. In addition, the cyclists were also given the assignment to use the feedback buttons as a way to provide positive or negative experiences while cycling. Along with this, the device automatically registered time and locations.

Fig. 2.
figure 2

The live-prototypes provided data on users’ mobility (left) and users’ cycling experiences compared with different levels of noise in the city (right)

Full size image

Simultaneously, we used contextual inquiries to provide more qualitative insights on the cycling experiences. We provided a daily survey that questioned participants on the use of the interaction buttons and the underlying experience. This information was later mapped upon the automated location data, deriving from the device. By doing so, we were not only able to point out perceived positive and negative locations across the city, but could also provide additional context, such as a negative experience due to a traffic jam. In addition to this, we had direct interactions with the users through one-on-one phone calls. This in order to check if the assignment was going well and to confront users with their cycling data and ‘dropped’ experiences.

In the end, having the possibility of combining this information proved to be very valuable. By doing this, we could compare and reveal links with the objective data that we retrieved from the sensors. On one hand this helped us to stimulate users’ recollection. For instance, as soon as we confronted one participant with a certain push on the button, she remembered the whole route and could provide more information on why she pushed the button whilst reflecting on her general use. On the other hand, it allowed to obtain a better understanding on the objective data and helped to identify possible technical problems (i.e. if a sensor was not working well).

At the end of the two-week trial period, a closing focus group was held to reflect on the overall practice and experience. By using the logging data along with the information which we required from the survey and the phone calls, we were able to quickly move into specific issues and get more thorough and deeper insights from the discussions. We also used other probing techniques such as drawing the future trajectory of the application to have something more tangible on users’ feedback. This offered not only insights on what they liked and disliked but also on the place this technology could have within their cycling patterns.

6 Conclusion

In this paper, we have described and illustrated a new methodology called “live- prototyping”, to be used in the very early pre-elementary design stage. By combining elements of lo-fi prototyping with a PTA approach, “live-prototyping proxies” allow researchers to capture early HCI insights and design requirements (both technical as user-based) deriving from contextual user experiences. The latter has the objective to develop lo-fi prototypes in such a way that they can be used in a real-life environment and are able to capture both the contextual elements that might influence the interaction as the interaction and the usage itself. By applying this methodology, we gained knowledge in the contextual needs of the user, the HCI experience with the technology and the users expectations to it, with alongside a set of technical design and architectural insights for future development.

Based upon our experience, we do think that live-prototyping has potential in the early development stage. First of all, it takes the benefits that prototypes offer. But by immediately designing them in such a way that they are tangible with basic functionalities, and moreover can operate in a real-life environment, real experiences can be evoked. This leads to a set of both technology-related and contextual usage insights of the device or service such as the appropriation of the technology, the daily routines, social relations and even non-usage. Second, by capturing user data on a (near) real time basis, experience is being caught in the natural habitat and moment of use. Based upon this data, direct user feedback can be initiated and gathered. By so the issue of a recollection bias can be reduced and more in-depth research questions can be posed. Finally, an additional benefit is the bounding character this live-prototype sets upon developers and user researchers. It allows developers to start with the initial engineering, exploring various technological scenarios without a predefined and fixed track. While at the same time, user researchers can focus on capturing the user needs and insights to steer and input the further development track. By doing so, a solution to the paradox that often exist between those two is provided, along with a structured process that allows sufficient explorative freedom and flexibility.

Nevertheless, the methodology still needs to be further elaborated in order to benefit from its full potential. First, the objective of the live-prototype is above all a mean for collecting proper data and insights. Being able to collect different streams of real-time context and usage data in combination with user feedback, requires the proper analytical tools and processes. Second, the development of such initial live-prototype is a trade-off between what is technologically possible (in terms of functionalities), what is needed, the purpose and the available timing and resources. Therefore, a good framework to decide upon the basic functionalities and the technological options is beneficiary. Third, the live-prototypes still remain lo-fi prototypes with preliminary ideas of functionalities. As a researcher, it is important to know and acknowledge the limitation that comes along with it. Also, towards the user, it is even more important to frame this correctly and set the right expectations. The prototype remains a proxy, something that needs to be further shaped and therefore cannot be considered as a final product. Due to its lo-fi character and thus lack of maturity one should also consider the possibility that these devices are unstable or not performing well. Therefore, a good monitoring along with short testing periods is required with an importance for the iterative and agile development process.

As mentioned, nowadays the design process is being challenged. The context wherein new technologies tend to operate is constantly changing and influenced by actors and other surroundings that are part of this setting. New methods to capture the use-in-context are required. Live-prototyping wants to contribute to this by offering researchers and developers an approach that, enables enriched user interaction research through real-life, contextual inquiry and this at already the very early stages of the design and development process.