Changing Things so (Almost) Everything Stays the Same

2.1 Advisory Services as Rituals

To design adequate support for advisory services, one needs to understand what an advisory service is. Researchers have approached advisory services from numerous areas. Management scientists study them as dyadic decision situations – they aim at enhancing the overall quality of the final advice or decision through process innovation and expertise enhancement [45], [64]. Marketing research sees advisory services as a customer relationship channel. It increasingly advocates strategies oriented at long-term co-creative and mutually profitable relation rather than a one-time sale – it challenges the status quo and proposes new communication and listening routines to make the service appealing to the client [56], [68], [72], [82]. Literature concerned with supporting collaborative work frames advisory services as work – it approaches various collaborative practices and tries to improve the overall service by providing digital support for the existing or new practices [20], [27], [31]. Finally, sociologists and linguists see advisory services as a form of institutional talk and investigate how the sequential character of the services corresponds to the institutional identities of the participants – they contribute an understanding of the fundamental conversation processes in various advisory service or selling situations [37], [66], [81]. The latter perspective includes recent works on the material aspects of the interaction between advisees and advisors [19], [81]. Research offers multiple entry points to the study of advisory services and applies various lenses to understand and improve them.

We conceptualize advisory services as, simultaneously, physical, and ritual encounters between two individuals, an advisor and an advisee. This framing corresponds strongly to the institutional talk perspective [37], [66], [81] and the material lens [19], [81]. Institutional talk discourse outgrows, among others, Goffman’s research on rituals in encounters between humans [24]. Through explicitly referring to rituals, we want to stress that the performances during advisory services have a recurring, repeatable character and, at the same time, celebrate a specific order in which the participants act to establish or protect their reputation or, in Goffman’s terms, their face. For instance, the advisor protects her identity as a domain expert, a representative of an institution, and a customer-centric frontline employee [20].

On the other hand, by emphasizing the physical character, we point out that the space, human bodies, furniture, and materials belong to the ritual [22]. The design of the room in which the advisory takes place contributes to the typical advisory service experience and embodied interaction with the advisor. In advisory services as physical rituals, humans act according to obligations to others and expectations one has of others while employing symbolic acts, which involve material and space [38]. In particular, advisors and advisees distribute documents in space to illustrate complex matters and to exercise control over interaction space, use expressive gestures to introduce oneself, employ posture to signalize attention or disinterest, and deliberately take care of the surroundings to prepare an advisory service [19]. For instance, when presenting a business card to the client in a dignified manner, the advisor employs a range of verbal, bodily, and material means to express oneself and the institution [19]. In mundane actions, the physical and ritual nature of advisory services comes to the surface. Designing a system for advisory services requires careful handling of the ritualized physicality when trying to manipulate the process, enhance the expertise, change the communication style, or afford new collaborative practices.

2.2 Physical Interaction: From Rooms of Collaboration to Rooms of Advice

Since the rise of computer-supported cooperative work (CSCW) and group support systems, researcher have struggled to embed the newest technology into physical rooms to improve collaboration. A first generation of researchers embedded desktop computers into specially designed physical furniture to create digital lecture halls [61] and meeting rooms [54]. The rooms with the furniture were not just containers for the technology, but part of a comprehensive collaboration design [57]. As technology became more mobile, researchers moved beyond fixed rooms. Instead, the technology was designed to be adaptable to different needs and collaborative settings, e. g., workshops in different configurations [74] or informal collaboration spaces supported by ‘Roomware’ [80]. At the beginning of the 21st century, it became clear, that computers would not only become ubiquitous, but also should be hidden from the users in order to advance interaction [67]. This insight became the more important, the more intimate and delicate the collaboration is. One of the most intimate and delicate settings is advice giving.

Despite the importance of physical conduct in collaboration, research on the non-verbal conduct in advisory services has focused, primarily, on the table and what is on there [19] as well as the sitting position [35], as depicted in Figure 2 and Figure 3. It lacked a holistic view of the physical matter of an advisory service. However, banks employ building architecture and interior design alike redesign of their leaflets or process changes to transfer the desired picture of themselves as institutions [28]. Rooms dedicated to service provision influence the encounters too. Consequently, designing technology while focusing solely on the material used during advisory services, as it has been done previously [31], [35], [51], can be risky. The studies report on collaboration breakdowns attributed to changes in external conditions such as sitting position [35], [51]. While the changes might well be necessary to suit a specifically designed support system, they distorted the physical order of a ritual. They forced the participants to establish a new ritual. In short, they broke with participants’ expectations or obligations concerning the use of space, such that eye contact or effective conversation were hindered [35], [51].

Figure 1

Elements of physical configuration in advisory services.

The wide lens applied in the current study acknowledges the fact that material, the area accessible to the interactants, and the physical environment surrounding them impact the interaction. On the one hand, they afford specific performances while limiting others; on the other hand, they have been designed with specific routines and rituals in mind. This inherent relation between advisory services and the characteristics of the physical circumstance are related very tightly. The rituals’ occurring depends on the place (rooms, locations, buildings), conditions (light, furniture, sitting positions), and specific elements (documents, sheets of papers, pen) [19], [22], [81]. Figure 1 shows the physical configuration by illustrating the relationships between the physical areas. Note that the particular areas correspond to Kendon’s o-space (interaction area), p-space (access area), and r-space (environment) [46], [47]. However, to differentiate from Kendon, who studied how the spaces and new formations between individuals emerge, we prefer to stick to the more general term area, which also better expresses the static nature of physical configuration in advisory services [7]. In the current study, we treat the physical configuration of advisory services as a source of requirements for a system to be developed.

Figure 2

Physical organization of a traditional, pen-and-paper-based financial advisory service [19].

Apart from the spatial and physical configuration of advisory services, we also acknowledge that existing practices can be used as a source of knowledge about the advisory service ritual and provide requirements for a system as well. In particular, when attending to an advisory service, e. g., at a bank, advisees suppose not only a clean and tidy room and an advisor dressed in a specific manner, but they expect that the advisor will take notes, use tools to make calculations, and refer to leaflets and official documents [19]. Even those who never attended an advisory service at a bank rely on an intuition of what the experience may be like [40], [75]. The expectations from one side match the performances of the other side. Many bank advisors (and advisors in other commercial branches) rely on an advisory paradigm called pencil selling. It involves using empty sheets of paper and a pen to illustrate the main message transferred verbally with ad-hoc generated pictures, calculations, or text [25], [76], [77]. The range of particular practices employed in pencil selling is large. Still, they involve writing and drawing on paper, moving and freely positioning paper within the interaction area, as well as removing it from the interaction area and piling it on the side [19]. Overall, the pencil selling method – even though being often popularized as a specific marketing concept – involves traditional means of collaboration and explanation using pen and paper [19]. The pencil selling behaviors have become inherent elements of the current advisory ritual in many sales and advisory services.

Even though pencil selling has dominated advisory services for decades and thus shapes the expectations of many bank clients, this approach receives serious competition. First, after the crisis of 2008, banks are increasingly responsible for educating their clients and documenting this education effort. They want to involve them more in a shared decision-making process [64]. Second, financial products are increasingly complex, such that the drawings and explanations based on brochures cannot capture this complexity [63], [64]. Additionally, many advisors simply cannot draw, such that the pencil selling turns into a calculation exercise: the advisor calculates values in front of a waiting client [19]. Third, face-to-face advisory services receive competition in the form of more powerful online self-services and robo-advice [1].

Figure 3

Sitting positions and their relation with phases of an advisory service during collaboration with a table-top display [34].

If face-to-face advisory services survive in the future, they need to establish a competitive advantage beyond clever sales tactics. We argue that the advantage can emerge as a consequence of an intelligent combination of human and computer, such that the computer complements the human by contributing skills in which computers are strong (e. g., documentation, visualization, calculation) and vice versa [6]. Based on their experience with pencil selling, the advisors are strong in communicating and explaining complex products by using individualized and straightforward information [19]. They are also experts in rapport building [75]. And they established impression management tactics that involve the use of physical material and documents [19]. No computer can take over those tasks easily. However, previous research often disregarded the role of advisors’ skills by creating technology-oriented solutions: they imposed a specific order of actions by representing a step-wise logic of the software [62] or forced the advisor to explicate information about the advisee to enable for further processing [50], [52]. Primarily, however, the prior solutions took the essential tools of an advisor, pen and paper, away from them. The rituals were not possible to conduct, such that advisory services did not unfold in line with the expectations and obligations. This study aims at complementing advisors’ practices – especially the material routines – with IT and affording new performances compatible with the advisors’ embodied work practice. Accordingly, it relies on requirements we derive from the current physical configuration and practice. In the following, we identify those requirements while using Kendon’s [46], [47] perspective on the physical environment to structure them. The requirements are in italics and are assigned a specific number for further reference. We derive the requirements from previous literature on technological support of advisory services and from field observations collected in many years of research on financial advice.

Figure 4

A vertical display during a financial advisory service was shown to disturb the natural flow of a conversation.

The interaction area in advisory services is the space where participants exchange sheets of paper, collaborate on specific documents and position their materials. Physically, it is simply the part of the table between the advisor and the advisee. Documents and objects that enter the interaction area have a meaning that results from their content and their position in relation to other documents and the participants [19]. Two sheets of paper put parallel to each other are likely to represent alternatives or afford comparison, while a pile of documents lying next to the advisee includes, probably, his documents. This gives the client a chance to understand the ongoing encounter implicitly: they know what to focus on, what pertains to their specific case, and what should stay out of focus. Therefore, it is essential to allow for free positioning and moving of paper and other objects (DR1). At the same time, paper is a medium that keeps its semantics over time. This allows for recognizing previously used documents and easy reference to them [19]. The objects in the interaction area keep a stable, permanent identity. Thus, a solution needs to guarantee object permanence such that content is permanently linked with specific physical objects (DR2). Moreover, advisors rely on a pen to generate individualized content and make notes. This behavior signalizes interest in the client’s specific situation and gives the client a feeling of being treated uniquely [9], [49], [50]. Accordingly, the system needs to accommodate for unique and individualized input, output, and manipulation of data (DR3).

The access area in advisory services is the territory that is accessible to the advisor and the client but is not the focus of the interaction. The access area embraces the table client and advisor are sitting at and possibly some objects in reach like a bag or a folder positioned on a chair standing nearby. The table forms the physical basis for the pen-and-paper interaction: an interaction like described above would not be possible without a table underneath the interaction area. The client and the advisor require the table to distribute their documents across and provide physical support for their position: clients and advisors tend to lean forward towards the interaction space while putting their arms on the table. On the one hand, they put themselves in a comfortable position to focus on the content and the social interaction. On the other hand, they can signalize their interest in the content being presented by taking such a position [34], [51]. To enable those behaviors, a potential solution needs to support physical interaction on, at, and with the table (DR4). Additionally, the table (and, sometimes, surrounding chairs or other nearby furniture) takes the role of storage space for documents and objects which are out of focus at a specific moment. Frequently, the advisors distribute their material all over the table. This way, they do not forget things they wanted to talk about during the advisory service and signalize to the client what content remains to be addressed [19]. They also use the table to store documents they need for the service’s post-processing or files that need to be scanned or copied during the service (e. g., records provided by the client on paper during the conversation). They can stay in sight of the client without entering the interaction space. Accordingly, the developed solution should provide space for storing out-of-focus objects on the table (DR5).

The environment of advisory services embraces the room in which the conversation takes place. Our experience shows that the rooms are mostly very tidy, clean, and modern [19]. They provide space for two to four persons. The table stands in the middle of the room. Sometimes, another piece of furniture (e. g., a chest of drawers) stands at a side with some brochures displayed on it. Usually, there is no or one piece of art hanging on the wall. Technological devices like a printer, scanner, or monitor are located away from the interaction area. A monitor and a keyboard might be positioned at one of the sides of the table, although some banks do not have any monitors in the advisory service room. Other devices might be hidden in the drawer board, below the table, or are not even in the room, such that the advisor leaves the room to copy the documents. The way the rooms are furnished and decorated makes clear that the conversation between the advisor and the client is the core activity that happens inside. An unusual or dominating device like in Figure 4 might disturb this focus and negatively impact the conversation flow [51]. Therefore, a technology employed during financial consultation should interfuse with the physical surrounding (DR6) such that it disappears in the room and does not distract the participants. Yet, advisory service rooms share another characteristic: they are filled with light. Given that pen and paper play an important role during the interaction, bright light is necessary to see what is written on the paper. Consequently, the rooms feature large windows and bright, modern lights. Additionally, in modern branches, banks employ walls made of glass, which makes the rooms even brighter. Accordingly, a potential solution needs to be applicable in bright rooms (DR7).

2.3 Technology in Advisory Services

Overall, the listed requirements make clear that designing a system for use in advisory services is a challenge. Simultaneously, research [16], [18], [36], [62] and politics [63], [64] call for a more effective information transfer and higher transparency in advisory services, such that the advisor and the advisee can engage in collaborative problem-solving. IT can play a major role in this regard. Previous research on supporting financial advice has shown that dynamic manipulation of data can support client education, such that advisees can make more informed decisions and acquire an understanding of the core issues under consideration [35], [36]. It can also contribute towards informational and procedural transparency [62] and help the client share relevant information in a more structured and complete way [49], [52].

Figure 5

The use of a tablet during burglary prevention advisory services [11], [20], [30] to feature mobility during an advisory service.

Research on the application of IT in other advisory services, such as burglary prevention service (cf. Figure 5), travel planning services, or employment consultations, show further potentials for supporting advisory services. IT offers seamless documentation of the advisory process [30], empowers the advisors to provide more professional [32] and persuasive [11], [20] services, and creates a more engaging experience for the advisee [17], [58]. Overall, there is much evidence for a positive impact of technology during advisory services. Yet, the uptake of modern, multimodal, and interactive technologies in banks remains slow [19], [60]. The likely reason is that the tools, despite their advantages, compromise the quality of interaction between the advisor and the client.

Heinrich et al. [35] point to relationship building as a core value in an advisory service. Accordingly, a goal of an advisor is to establish a lasting and enjoyable interpersonal relationship with the client and vice versa. Simply put, the client and the advisor want to have a pleasant interaction with each other going beyond the fact-based formal interaction. Dolata and Schwabe [19] argue that material practices performed by the advisor aim to impress the client in many ways. For example, they should position the financial advisor as competent, orderly, and customer-oriented yet unique. They argue that technology might compromise any of those aspects and, especially, limit the advisor’s ability to express their unique personality. In a pen-and-paper-based service, they use their individual writing style or even a special pen of their choice for this [19]. With an interface imposed on all advisors, signaling an individual character might even be more demanding. Additionally, technology might introduce additional hassle for the client or the advisor: it might, for instance, force them into ineffective communication and collaboration practices [51], [52] or impose additional cognitive load, e. g., to make sure that the client does not see confidential or inappropriate content [14]. What emerges is a dilemma for the advisors and the responsible organizations: Are the potential improvements worth risking existing routines? Is best the enemy of good?

We claim that part of the problem is that the prototypes and systems employed in research and practice removed the existing practices and introduced a new physical configuration. At once, the focus of the conversations moved away from the interpersonal exchange in which the participants look at each other in a laid-back position to an encounter in which they bow over a (sometimes too small) screen in between them or look at a monitor filled with various information and loose shared focus [34], [51], [60]. Consequently, the real treasure of advisors’ process knowledge, practical experience, and embodied performance patterns seemed thrown overboard. Neither the client nor the advisor could rely on scripts and obligations that follow from the institutional character of the encounter [12], [19], [38]. None of the parties could draw on the collected experience and situational knowledge. In this manuscript, we present a system that preserves the physical, ritualized character of advisory services es expressed by the requirements listed above (DR1–DR7) and tries to complement them with the most natural advantages of a computer: quick calculations, dynamic presentation of data, and easy manipulation thereof.

2.4 Mixed Reality

Mixed reality (MR) responds to combining traditional interaction in a physical space with the potentials of modern computing. MR approaches, like spatial augmented reality [4] or tangible interfaces [43], [44], incorporate physical objects and surfaces into digital interaction to engage spatial perception and allow for touch as a sensory cue. This research promises to combine the material nature of human behavior with the powers of a computer. The relevant efforts flow into a range of mixed-reality discourses: Augmented reality (AR) studies how to overlay physical spaces with computer-generated content [2], [3]; spatial AR projects directly into the environment of the user [4], however, it requires powerful and high-fidelity overhead projectors. Organic user interfaces (OUI) focus on using non-planar spaces that function as means of input and output, such that users can interact with them through bending, folding, and manipulating the form – this includes using paper as a display [41], [42]. Tangible user interfaces (TUI) focus on enabling interaction, manipulation, and collaboration [73] with digital content through physical objects and space [44], [89]. Finally, pen-and-paper user interfaces (PPUI) try to bridge the gap between digital interaction and paper, remaining an essential tool in many human activities [55], [78], [79]. All those areas of research share the interest in extending the physical world with digital content and functionalities. To a certain extent, they want to turn the user’s focus and attention away from the computer and back to the physical world, thus allowing for calm design [84], i. e., one in which the technology provides information or enhances the experience, but does not compete for user’s attention and stays in the periphery until needed. Each discourse offers numerous ideas on how a support system for human-centered financial advisory encounters could look like. However, it remains unclear which suggestions are promising and how to design and develop a combination of them to support the advisory services.

The scientific discourses listed above focus, primarily, on single-user interaction and therefore approach the ease of use from this perspective. However, financial advisory services pose additional problems. Since it is a collaborative situation, the activities of one party need to be accountable to the other party, such that the action can be interpreted appropriately within the context, place, and the applied script or role [22]. Inability to interpret an action will lead to irritations that require attention to resolve [50]. For instance, if an advisor needs to tap three times on his face to launch a specific computation (even if this interaction is easy), the advisee may focus on resolving the meaning of the interaction rather than focusing on the content [22], [23]. With natural and visible data manipulation techniques, the advisee is more likely to understand the impact of specific actions on the financial situation or its representation [35]. Introducing manipulation of figures with a clear meaning along with TUI’s principles seems therefore promising [85]. Furthermore, since the financial advisors are concerned with their impression, any inappropriate or unprofessional action implemented in the system might result in a reluctance to use it. This issue also holds for environmental features: advisory services take place in light rooms at designer tables, such that designing a system for a dark room (as suggested by many spatial AR applications [2], [4]) would break with advisees’ and advisors’ expectations regarding advisory encounters. Additionally, the high-touch nature of advisory services leads to actions and behaviors with fuzzy, implicit meaning – often, the advisor makes notes that the advisee should not notice [11]. PPUI research encounters the same problem: many explicit activities (e. g., inking or drawing a link) have a varying conceptual meaning depending on the context, as well as previous and future activities [79].

Overall, even though the existing literature on MR rarely attends to professional, institutional collaboration, it includes indications for promising interaction mechanisms. However, it remains uncertain how to combine those singular implications into a coherent system for financial advisory services. Specifically, it is necessary to figure out how to overcome typical limitations of state-of-the-art MR systems like the dependency on low-light environment, recognition of relevant and irrelevant objects, or focus on a single-user scenario or collaboration between equal partners outside of the institutional context. Furthermore, the MR interfaces bring along new challenges: form and application of such a system is not a part of a typical advisory encounter script and might thus overwhelm the clients and the advisors who expect something different based on a situational script [12]. Additionally, MR might introduce usability problems leading to rejection of the system altogether. An unreliable touch input during a visit to an amusement park can generate a bit of disappointment and will be forgotten quickly, but an advisor who tries to launch an application for several seconds does not create a picture of a reliable person and the bank. Given the vital relevance of the ritualized character of physicality in advisory services and the importance of impression, there is less room for trial-and-error and less potential for testing early versions of the system in the field.

4.1 Environment: Room and Furniture

DR6 demands the system to interfuse with the physical environment. Accordingly, changes to the environment had to be as small as possible. Together with an interior architect, a ceiling-mounted unit (see Figure 7) was designed, which integrates a 4k projector (Optoma UHD60 (Optoma)), an MS Azure Kinect sensor, both pointing downwards, as well as the computer running the software. The bottom surface of the unit can be lit by LEDs, thus serving as a lamp and further concealing its purpose. The whole unit can be moved down and up to easy access to the specific devices. This solution was preferred to full integration in the ceiling for two reasons: (1) the sensors and the overhead projector work best if they are in a specific distance range to the projection surface (table), and regulating this distance might be necessary, (2) this construction can be easily replicated in other rooms or other branches of the bank without essential changes to the room itself. Yet, some cover elements (e. g., the color of the frame) can be adjusted to fit the room and make the impression that the unit belongs to the room.

Figure 7

LivePaper unit lowered from the ceiling during maintenance.

A flat surface underneath the unit, usually a table, acts as the screen for the projected image. While a homogenous, light gray, or white-colored table surface will increase the contrast of the projected image, the system is designed so that it also works on darker or textured surfaces. This, again, should support the applicability of the system across various physical settings. For instance, a wooden table is the projection surface in one of our three system installations. Projecting on darker surfaces works equally well because the essential information will be limited to sheets of white paper which can be arranged freely on the table. This solution was preferred to projecting directly on the table for several reasons: (1) interacting with paper feels more natural rather than interacting with projected content, (2) paper provides a constantly white background such that the projections are easier to see and read, (3) it is possible to write on the paper. Only auxiliary user interface elements are projected outside the paper boundary onto the table surface directly. Overall, the above choices respond to the DR4 such that the physical interaction can happen on, at, and with the table. Even moving or rotating the table, changing its size, or moving to a different sitting position is possible without any losses if the table is large enough to accommodate the projection. Each time the system starts, it conducts an automatic calibration sequence to adapt to smaller changes in the distance between the table and the projector and sensors.

Another challenge is maintaining contrast between the projected image and the reflected ambient light in a bright room. This is mitigated twofold: firstly, the 3000-lumen output of the projector is concentrated on a relatively small area (approximately 1.5 m²). Secondly, the visual design uses high-contrast colors and white-on-black text. This generates the impression that bright and crisp graphics or letters are presented on light-grey paper. This accommodates for DR6 such that the system can be effectively employed in bright rooms. The design team has tested various color schema, fonts, and sizes throughout the project and arrived at a combination of various light-green, light-blue, white, and yellow shadows as the optimal combination, which is well visible but at the same time not too aggressive. Blue, green, and white elements are used for feeling areas or text, while yellow is used for highlighting.

Figure 8

LivePaper system with three active papers, pen input, and tangibles.

4.2 Access Area: Paper Tracking and Movement

Since paper is one of the critical elements in this advisory setting, specially marked paper, so-called active paper, is used for interacting with dynamic digital content. The system recognizes it by a narrow bar code on its short edge, which is also used for distinguishing between active paper and other paper documents that may be part of the advisory setting, such as brochures or financial records. Active paper is a consumable that should retain all the properties of an ordinary sheet of paper, such as drawing or writing. Therefore, we provide PDF documents with barcodes that can be printed with conventional office laser printers for an inexpensive way to produce them. This solution was favored over the special paper to support paper rituals of an advisory service, which relies on an unlimited supply of paper.

Several approaches for active paper application development ranging from simply exposing an API to implementing a fat client have been evaluated. Finally, we settled for a Chromium Embedded Framework (CEF) based solution. CEF allows rendering web pages offscreen with the same technology as Google’s Chrome browser. By linking the URL of a web page to the bar code of an active paper, the digital content gets attached to a physical object. The system tracks active paper, and the web page’s content is rotated and translated according to the orientation and position of the corresponding active paper. This contributes towards free positioning and moving of paper (DR1) while at the same time assuring that the content is permanently linked with specific physical objects (DR2), in this case, a sheet of paper. Furthermore, it accommodates for storing out-of-focus objects on the table (DR5). Even if a sheet of paper is moved away from the interaction space and held somewhere on the table, its bar code will be recognized whenever it re-enters the space, and the belonging content will be presented.

In this context, tracking the movement and rotation of the paper turned out particularly challenging. Finding the borders of a white paper on a potentially white table surface is almost impossible due to the low dynamic range of the camera system. Therefore, we rely entirely on detecting the printed bar-code, which also encodes the orientation of the paper. The Kinect sensor features an HD RGB camera as well as an IR camera. While the RGB camera offers sufficient resolution to detect and decode the barcode, we observed that the high exposure time of the RGB camera blurs all objects in motion, making object following impractical. The IR camera has a much shorter exposure time and produces sharp images at a low resolution, such that only detection but not decoding of the barcode is possible. Consequently, we use the IR camera to detect potential barcode candidates, which then, in a second step, are verified and decoded using the high-resolution RGB camera as soon as the paper becomes stationary. We maintain a probabilistic model of the papers on the table while papers are in motion. This model gets updated and is corrected as necessary when papers are positively identified through the RGB camera. Redundancy in the barcode and the depth image of the Kinect sensor are used to deal with partial occlusions of the barcode, e. g., by hands reaching over the paper or when papers overlap.

In addition to the spatial limitations of the camera, we also had to address the inherent latency of the system. The infrared camera runs at 30 frames per second which introduces 33 ms of latency in the worst case. Our internal image processing can take up to 50 ms and the projector, independent of its settings, adds another 64 ms until the image is displayed. This results in an overall latency of approximately 150 ms. During our latency measurements through an independent high-speed camera (FL3-U3-13Y3M-C, Point Grey, 2016) observing the table, we were surprised by the speed at which users moved objects around. Velocities up to 1 m/sec were perceived during these tests. Therefore, the projected image was trailing several centimeters behind the physical object in motion due to the overall system latency. Three approaches were evaluated by a qualitative study (N = 9) to address the trailing image:

(1) Motion prediction: using a Kalman filter [33], motion data is used to forecast object position. While this technique works well for an object in constant motion, it tends to overshoot during deacceleration. The effect is similar to inertial scrolling found in today’s touch-based interfaces. Only two of the subjects liked the playful bounce effect of that implementation but still favored the other two approaches.

(2) Motion blurring: motion blur is the result of the limited temporal resolution of the human visual system [83] or the exposure time of a camera system. Details of objects in motion cannot be resolved as accurately as for static objects. Hence, we had the idea to artificially blur the projected image if the corresponding physical object is in motion to simulate the effect. The problem with this approach is that motion blur looks unnatural if the user is actively following the object in motion. In this case, everything else appears blurred due to physiological limitations. The object in motion, which should be sharp because it is followed by the user, is artificially blurred by the system resulting in visual inconsistency for the user. Only one subject preferred motion blur over the third approach.

(3) Fading: similar to motion blur, the visual appearance of the projected content is artificially altered while in motion. We adjust the transparency of the content by the velocity of the corresponding physical object. Static content is projected opaque while fast-moving content becomes fully transparent (fades out). Despite the fact, there is no real-world analogy of objects disappearing while in motion, eight subjects out of 9 preferred this implementation which in turn was selected for the final implementation.

The design decisions concerning the tracking and visualizing the movement of active paper across the table contribute towards enabling physical interaction on, at, and with the table (DR4).

Figure 9

Pen input with abbreviations.

4.3 Interaction Area: Pen Input and Tokens

Notably, the active paper should not only be a presentation surface but also allow for unique and individualized input, output, and manipulation of data (DR3). There are various aspects related to this requirement: providing new information (e. g., the name of the advisee), manipulating existing data including a choice of a data point to be changed (e. g., choosing and changing monthly interest rate), and presenting individual output (e. g., signalizing a recognized name). This all should be possible while accommodating the fact that the paper and other objects can be freely positioned and moved around the table (DR1) and keeping the link between specific content and a physical object permanent (DR2). In the following, we reflect on the various input and manipulation mechanisms featured in LivePaper.

In the spirit of supporting existing pen and paper techniques, handwriting input to the system was mandatory. The limited resolution of the camera system excluded an image-based approach. We use a Wacom Bamboo slate together with the Microsoft handwriting recognition engine (Microsoft) instead. This was a preferred decision compared to structured input or keyboard. During the contextual inquiry, it became apparent that paper forms were rarely used and would have an excessively structuring impact on the dialog with the customer. The keyboard was considered a foreign body on the otherwise clean and non-technological surface of the table. We also discovered that some advisors were using abbreviations for tagging specific numbers (e. g., annual income, house price, etc.) as a memory aid. Based on these observations, we adopted the practice and introduced a standard set of abbreviations for data input (e. g., “STR” for street, see Figure 9). This allows unstructured data input while still attaching the semantics to the data required for further processing by the system. The Wacom Bamboo Slate, which generates an electromagnetic field to record movements of a pen on paper, was preferred over solutions using dedicated paper and a sensor or camera in pen (e. g., Neo Smartpen or Livescribe pen) as well as IR-based solutions (e. g., Fujitsu Mobile NoteTaker). Whereas the Wacom Bamboo Slate (and similar solutions) can be hidden behind any paper block and deals as a pad, the other solutions are either visible such that a clip is added to the top side of the block (IR-based solutions) or require the user to write on specifically prepared paper (including a specific pattern). The latter would make it difficult to write on brochures or use the paper with the bank logo, while the former would again introduce a clearly visible and unusual technical artifact in the interaction space.

One of the key benefits of dynamic digital content over static printed content is its interactivity. Apart from adding new data, it might be necessary to interactively manipulate the projected data on the table. This might be data coming from the core banking system (e. g., the current interest rates) or defaults that require adjustments to the client preferences (e. g., the duration of a mortgage). However, before the user can even manipulate the data, they need to choose what item should be adjusted. Based on a solution proposed by Xiao et al. [86], we implemented finger touch detection. Touch events on an active paper are translated to mouse events for the corresponding web page. Even though pressing a button on paper is not a natural form of interaction, earlier research shows that touching and using deictic gestures with paper occurs naturally within an advisory service. We decided against using a dedicated pointing device (e. g., a special pen or wooden stick) to reduce the cognitive overload of the user: point and click with a finger was considered least obstructive.

Figure 10

Two tokens and a an active paper sheet.

Figure 11

The house with the map.

Having selected the specific item, the user can manipulate the data using tokens. We introduced several tokens with different functions. Specifically, we use a Microsoft Surface Dial for changing numerical values at various places in the application. The Dial is tracked by the IR camera together with the Kinect depth sensor. It is associated with an active paper by its proximity to the paper and a specific numerical item through the touch input. Turning the Dial results in mouse wheel events for the corresponding web page, which can use the events for animating content. The turning action in the physical environment, together with the animated content, helps to understand and interactively explore the underlying financial models.

Additional tangibles include a stylized house roughly the size of an espresso cup and several flat tokens representing different types of mortgages. The house serves as a tangible link for discussing the property and its surroundings. It is tracked by an infrared marker on the top together with the depth image of the Kinect 2 sensor. Similar to active paper, the house can be linked to an URL. Usually, we show a map centered around the property’s location in discussion (see Figure 11). With the same two-step process used for the LivePaper barcodes, tokens are tracked by the IR camera, and three different token types are identified through the RGB camera. Instead of barcodes, we used engraved text (e. g., “FIX” for a fixed-rate mortgage) for token identification.

Together with the tokens, we also introduced a novel interaction technique called “place and link.” In contrast to the static link between an URL and the barcode of an active paper, newly placed tokens on the surface can be linked interactively by a touch gesture to digital content. This link is then visualized by the system as a curved line (see Figure 10). The concept of “place and link” underlines the logical connection between two elements with interaction in the physical environment and is thus easier to comprehend for an observer than spontaneously appearing interface elements. Overall, the introduction of the tokens and actions associated with them provides means for manipulating the presentation’s content and enhances the overall feeling of interactivity. The actions associated with the tokens go beyond the traditional pen-and-paper interaction and, therefore, a new, physical form was chosen to make the intention clear to the user.

Figure 12

Using the Dial to change a selected value.

Figure 13

LivePaper used in an experimental advisory service.

4.4 Evaluation and Use

The system described above was evaluated in two design experiments (cf. Figure 13) and subsequently used in two branches of the partner bank. The experiments [9], [10] provided evidence that the system, in combination with appropriate practices, has the potential to significantly improve various aspects of financial advisory service. For instance, advisory services conducted with LivePaper outperform traditional services in terms of client’s satisfaction with the service, their assessment of the information transparency and interpersonal interaction, and the overall perception of bank service quality [10]. For detailed analyses, we refer the reader to the previously published articles [9], [10], [21]. Overall, the clients and the advisors preferred the LivePaper-supported advisory services over the traditional ones.

After the successful experimental evaluation, two bank branches were equipped with the system to be used in mortgage advisory services for four weeks. Two advisors were trained to use the prototypical system outside of an experimental setting. After four weeks, we conducted interviews with the advisors. On the one hand, one advisor used the system only sporadically, with few clients only, and explained that her reluctance to use the system was related to her insecurity and technical problems with starting up the system, as well as lack of “appropriate clients” who would understand that the system has some limited capabilities. Indeed, at that time, the system was not fully functional because it lacked integration in the core banking system – the simulations and orders made through LivePaper would need to be manually transferred to the banking system, and the organizational processes for doing so were not clear yet.

On the other hand, the second advisor developed long-term practices for using LivePaper and even trained his trainee to use the system regularly. He was reporting that “not every client is right for being advised with LivePaper” and “the understanding for who will enjoy it and who will be overwhelmed by it is important.” Still, he was assessing to use it roughly in 40 % of advisory sessions. He admitted that working with LivePaper still requires a significant time for starting up the system, but it became part of his routine: “In any case, you would want to see before an advisory session if the room is clean and tidy. So, I just went there a few minutes earlier and started the system. I can use the time to think about how I want to interact with the client. I launch the system and then position the house token in the middle. It makes a great impression when the client enters the room, and their house with the map is already on the table. I think it looks great, and clients are simply ‘wow’.” He explained that he uses the system primarily to explain the basics of the mortgage system in Switzerland, which he was previously explaining on paper while using hand-made drawings. Later, when the conversation moves towards a specific mortgage offer, he would turn off the LivePaper no to mislead the client by some invalid or out-of-date numbers or calculation models that were hardcoded in the system.

Apart from using the system for advisory services, the bank’s innovation department and management were frequently presenting the system to existing and potential partners. Given that the bank is active in a small region and possesses only 12 branches serving an area with about 500’000 inhabitants, it was looking for additional partners who would participate in further development of LivePaper and share the costs. Meanwhile, it partnered with another regional bank with more than 50 branches serving an area with more than 1’000’000 inhabitants. About one year after the pilot study with two branches, the banks decided to move the project towards a product. By now, in 2021, they have established a startup, which developed a second version of the system to address some technical issues (like refresh rate), update the technological standards, interface the system with core banking software, and add further functionalities. They use the second version of the system regularly in more than five branches and are successful at attracting interest from additional partners from Switzerland and beyond. Yet, the design and technical solutions presented in this chapter remain essentially unchanged.

5.1 Implications for Advisory Services

The research confirms the important role of physical rituals in financial advisory services. As opposed to some of the earlier systems [34], [50], [51], [52], [62], LivePaper was considered a natural yet fascinating supplement to a service rather than a disturbance which changes the character of the service. The research concerning the selection of features and technical solutions shows that stakeholders preferred a system that merges not only with the physical environment but also with the practices that happen in this physical setting. This confirms that people follow scripts about how institutional encounters should look like and establish expectations based on this [12], [19], [37], [38], [81]. The development of new practices which might lead to new shared rituals requires time and a gradual approach. This might be exemplified by the new approach developed by one of the advisors: making a strong point by using the system at the beginning but then returning to the more traditional, paper-based approach. We claim that many of the solutions developed for use in advisory services ignore the subtle yet important role of physical and institutional rituals and focus more on supporting procedures and processes of an (optimal) advisory service. They might provide the right functionalities, yet they will feel quite foreign and detached. They are redefining the whole encounter at once, without making space for existing practice and thus the feeling of continuity between what is old and expected and what is new and inspiring. This research shows that advisory services are very settled and ritualized encounters. Changing them will not happen through formalizing or enforcing any regulatory or technical requirements proposed in the literature or law [63], [64] but requires a longitudinal approach. Technology bears the potential to support this evolution to a certain extent by providing means to fathom what is acceptable for clients and advisors and what overwhelms them.

Whereas technology can support establishing new practices and incorporating them into the existing rituals, the diffusion of those practices requires effective training. Advisors are frequently lonely rangers, who learn about the practice of consulting clients from more experienced peers at the beginning of their career, but later only rarely receive uptraining and yet more seldom they receive feedback to their work as an advisor [11], [20]. This limits the natural propagation of effective and identification of ineffective advisory practices: each advisor establishes their unique practices and narratives. The introduction of a novel technology is one of the very few interventions which provide an opportunity to change the status quo. Yet, frequently, advisors are left alone with the system and develop not only different ways of dealing with the technology but also radically different attitudes, as presented above. We claim that if advisors were given a chance to learn from each other on how to use the system and share their experiences for real advisory sessions, rather than artificial training situations or experimental sessions, the diffusion of technology and of effective practices would be more successful. We thus call for a practice-oriented, iterative, and professionally facilitated sharing of expertise and experiences between advisors, especially when novel technology is being rolled out. A training session like this could involve the following: (1) discussion of positive and negative episodes from the encounters supported with the system, (2) identification of exceptionally effective and ineffective behavioral patterns based on those episodes, (3) practice sessions in which the advisors provide advisory services to each other in which they enact the identified practices such that they can experience how those practices “feel” from the perspective of the advisor and from the perspective of the advisee, (4) reflection about the practice sessions. Indeed, we implemented a similar approach towards training when preparing advisors for experiments with the difference that we provided them with imaginary scenarios about episodes rather than actual episodes. The advisors liked the training, and their effects were observed in the advisory services they provided during experiments, but a systematic evaluation of this training methodology is yet to come.

5.2 Implications for Technology in Advisory Services

Whereas some of the challenges experienced during the development of LivePaper result from the overall architecture and the decision to create an MR solution, many of them relate to the nature of advisory services. The latter challenges and implemented solutions are transferable to other systems for advisory services support, even those that rely on a different technological stack and use different interaction mechanisms. First, LivePaper combines two aspects of advisory services: marketing [19], [49] with information transfer [35], [62]. Marketing is addressed by elements helping with impression management, such as the token of the house and the map and personalized and visually appealing visualizations. Information transfer is supported by dynamic manipulation of graphics so the client can understand the implication of her decision as well as meaningful, easy-to-understand, and stable semantics of the tokens and papers on the table. The research on supporting advisory services has mostly treated those two aspects in separation, thus yielding systems that helped transfer information and educating the client, but were lacking improvements in marketing aspects and vice versa [10], [36]. LivePaper overcomes this divide and can improve marketing-relevant measures and pragmatic measures of a service encounter [9], [10]. This implies that technology for advisory services needs to acknowledge the fact that financial advisory service is simultaneously a selling encounter and education or problem-solving encounter [18]. Balancing the two is a challenge itself because selling and marketing frequently rely on information asymmetry, while information transfer tries to overcome this asymmetry. LivePaper thus deliberately focuses on making the right impression and inciting joy sparked by learning new things in an interactive manner. We are convinced that this approach is a promising and ethically valid way to improve advisory services not only in terms of their quality but also in terms of their economic value and, ultimately, sales.

Second, LivePaper positions the relationship and interaction between the advisor and the client at the core of the service. On the one hand, this results from hiding the technology and making it available and visible only when it is needed and actively used. On the other hand, the system inspires discussions by providing information previously unavailable in a service, like a map showing the location of the property and its surroundings. Consequently, discussions about the property under consideration emerge [9]. This gives the advisor a chance to show their interest in what is important to the client, and the client can express her motivations. We claim that technology should provide serendipitous information and so spark exchange between the advisor and the client. This might be achieved with as easy means as a map. Still, more advanced interventions are possible, where the content gets displayed or suggested based on the information about the conversation available to the system (e. g., the client’s address). This might function as an icebreaker and help the participants establish a relationship that relies on mutual recognition. The past literature already acknowledged the importance of a free conversation phase at the beginning of an advisory service. Specifically, it was shown that this phase is important to create an offering that fits the client’s implicit needs and values [49], [63]. We claim that role of the technology goes beyond storing and processing the collected information, but it might also inspire the discussion. Yet, the border between inducing a free conversation and forcing participants into ineffective and unpleasant practices (form filling) is very thin [52]. Thus, more research is needed on how free conversations can be effectively supported without creating a feeling of enforcement or coercion. LivePaper’s house token can inspire exploring other approaches.

LifePaper also advances traditional hardware supporting collaboration in rooms [61], [67], [74], [80]: Technology is hidden not only to the point, where only the interface is visible, but it completely disappears and only an augmented natural environment remains. Thus, it enables to maintain the delicate and intimate relationship between advisor and client and at the same time makes modern tools and information accessible during advisory service encounters. Such approaches could not only improve financial advice, but may also be applicable in settings like travel advice [59] or medical advice [26].

5.3 Implications for Mixed Reality

Finally, the design of the system and choices made along the way can inform the development of MR solutions and research related to them. It provides specific ideas on how to solve practical problems related to creating an immersive yet professional experience. First, it presents how the devices can be disguised to make them disappear for the user. Second, it suggests an approach to move objects and sheets of paper around the table, so they feel and seem natural and synchronized, despite latency. Third, it recommends specific colors to be used when projecting on paper and table surfaces, such that projection works and looks appealing on white and wood surfaces. Fourth, it provides an example of how PPUI can be combined with AR to create a coherent paper-based yet interactive and dynamic experience for the user. Fifth, it allows for multimodal yet unobtrusive input mechanisms. Sixth, it indicates how content and physical objects can be linked to each other, such that the links remain stable even if the physical objects are moved out of interaction space. Seventh, it establishes an overall framework that allows for dynamic input to projected content without making boundaries of the interaction area explicit. Based on the design of the system, we can identify the following design recommendations:

1. Use usual elements that people expect in the environment, like a lamp for hiding the technology.

2. Combine motion blurring with fading to create the impression of a synchronous move of a physical object like paper and the projection linked with this object.

3. Employ white and other bright colors for small elements and text. Use darker colors for filling areas and plains.

4. Provide simple sheets of paper as input and output surface yet make active sheets recognizable to the system by using individual bar codes.

5. Use pen input with optical character recognition for entering specific and precise information; use tokens for manipulating projected content in a dynamic way.

6. Associate each active sheet or active token with an URL that links to the website featuring the content connected to this item.

7. Allow each website linked with an item to appear on a black plane without visualizing the borders of this plane.

5.4 Digital Agents in Advisory Services

Over the last decade, the research on technology for advisory services points in one direction: less is more. Specifically, the solutions developed and evaluated over that time moved from clearly visible and dominating devices shown to impact the basic structure and rhythm of the conversation [16], [18], [52], [62], [75] to technologies which disappear and align with pre-existing practices and rituals while extending them only where necessary [9], [10], [21], [60]. This shift aligns to a certain extent with overall technological trends. Yet, it also is a symptom of accumulation of knowledge and evidence suggesting that invasive technology takes too much space while destroying the very subtle construction of institutional talk. We envision two alternative paths for the future development of this research area.

As argued before, LivePaper moves much responsibility to the advisor and provides very limited guidance. Advisors must sense the client and decide when and how they use the system or keep it invisible. According to the collected insights, three factors are key: (1) is the client at all open for technology in advisory services, (2) does the current stage of the conversation allow for meaningful use of technology, and (3) what functionality of the system should be used and how. One can imagine that decisions concerning (2) and (3) can be effectively supported with situated contextual awareness. Based on multimodal information about the ongoing conversation (voice pitch, transcription, keywords, gesticulation, exchanged documents, inputs made), the system could estimate the phase of an advisory encounter or derive what content is currently salient. It could then use the information to become more active, i. e., project relevant data or create notes in the background or disappear. This would reduce the cognitive overload on the advisor’s side, who could rely on guidance from the system. Thanks to the progress in machine learning and ambient intelligence, this idea could become a reality soon. It would make the technology disappear even more – not only for the client but also for the advisor – yet accessible when advantageous. This would keep many of the social and physical rituals intact and augment them in the traditional vision of institutional talk.

However, if we were able to make the technology smart enough to sense when it can offer a reasonable contribution, why not question the trends of hiding the technology explained before. One could assign conversational agency and institutional identity to the technology. In other words, the system would become a third partner equipped with specific abilities like quick calculation, note-taking, or overseeing the legal aspects of a specific transaction. Rather than disappearing and immersing with the background, it could possibly have a physical or visual representation and engage in conversation with the other two participants. While surely controversial, this vision finds support with some advisors [12]. Importantly, including a third party with its own identity and responsibilities, different from that of an advisor or a client, would require a deep rethinking of rituals and practices typical for an advisory service. Currently, the service encounters follow the ideal of institutional talk, such that appropriate and expected behaviors and contributions follow the institutional identities of the participants: the advisor acts as a selling person and a representative of the bank, while the client acts on behalf of her own and, sometimes, her family and seeks a product offering or a bit of adequate advice. The scripts of institutional talk do not foresee a third, independent party with very specific yet limited abilities. This bears challenges beyond visual and interaction design: What is the appropriate conversational behavior of a digital agent in an institutional conversation? What is its institutional identity? How can the identity be signalized to the other participants? How to make the behaviors consistent with an identity? How to engineer the collaboration between human and non-human participants? How to guarantee a natural flow of conversation? Given the rapid development of natural language processing and digital assistants, those questions require an answer and point towards new challenges in supporting advisory services.