Keywords

1 Introduction

Solid state lighting (SSL) provides possibilities not readily available from legacy lighting technologies and is disrupting the lighting industry: for example there are now many color temperature controllable, as well as color controllable, lighting products on the market. Lighting products are experienced in ways other appliances like toasters and dishwashers are not and now SSL is providing new lighting experience possibilities. This study was conducted as part of internal research by our Central Innovation application and design department within projects on user-experience and advanced natural user interfaces, with the primary goal of developing natural user interface solutions for advanced lighting control. The objectives and benefits of this project specifically were multifold. Systems Engineering and Human-centered design principles were used extensively to envision, re-search, and realize lighting solutions that control complex technology yet are compelling, engaging, easy to understand, and enjoyable for people to use. Such concepts may ultimately anticipate human needs and enhance human experience thereby pushing the boundaries of SSL solutions and applications. Natural language interfaces (NLU), voice control, and natural gesture interfaces now being explored for gaming, automotive, medical, and other applications can also be used in advanced lighting applications. A working prototype using Android-based mobile devices was architected and developed to control advanced lighting systems consisting of color tunable lighting and multiple sources.

The experimentation involved subjects controlling tunable lighting in a natural environment of a living room to set the lighting environment to their liking for common tasks within these applications. They controlled the light level and color of multiple lighting fixtures using voice commands, gestures, and a combination of both voice and gesture. They were intentionally not given instruction on what to say or how to gesture with the phone, but instead asked to just try whatever seemed natural to them. Care was taken not to provide language or example gestures for any of the tasks. To close the control loop, one or more “wizards” were employed to actuate the advanced lighting unbeknownst to the subjects.

Voice data and sensor data was recorded on the phone, video and audio was captured via a wall mounted camera, and observer notes were taken. The data was analyzed to extract the unique ways subjects attempted to control the lighting and reach their high level task goals. For example, natural ranges of acceleration data attempted by the subjects for dimming were statistically analyzed and compared to those attempted for shutting off the lights. The common and unique verbal commands and naming conventions were extracted. Mental models of the subjects were gleaned from think-out-loud methodology, as well as, moderator conducted interviews.

The subjects agreed to being recorded and to not disclose what they experienced outside of the experiment. All subjects were provided non-disclosure (NDA), consent, and release forms in advance. The light intensities and modulation levels were limited to well within the levels of normal lighting experience. No excessive glare or flicker was present. The study was approved by our internal review board.

2 Study Development

The approach taken to study natural ways people use to control lighting was to provide a well known and comfortable environment, to refine the instructional script and answer questions in such a way as to not influence the subjects’ behavior, to encourage experimentation by saying “just try it”, or asking the subjects to demonstrate how they would control the lighting for a given situation, and by employing expert trained personnel known as “wizards” to provide fast feedback. Internal subjects were used to refine the experimental procedure and to train “wizards”, moderators, and observers. It was then extended to external subjects recruited by an outside firm.

We recruited 20 participants to come to our facility to participate in a survey about natural language interfaces, voice control, and natural gesture interfaces used in advanced lighting applications. A broad demographic representation of male, female, 18-65 years old was selected but narrowed to those who would likely have or like to have home automation and use a smart phone regularly. We asked for them to be 1 h appointments Monday through Friday at 9:00, 10:30, 1:00, and 2:30. The participants were provided ahead of time and signed on site an NDA, an Informed Consent Form, and a Photo/Video Release.

During the experiments, the lighting was supervisorially or directly controlled by one or more unknown external operators called “wizards”. The knowledge that the lighting was being controlled by another person was not given to the subjects. This technique is called “Wizard of OZ” after the famous scene in the well-known old movie by that title, where the wizard is unknowingly controlling equipment from behind a curtain. This technique is well known in human centered interface design (Jacko 2012) (Salvendy 2006) but is to our knowledge novel to lighting and home automation.

Living Room Study Setup Details

The experiment was conducted in our internal laboratory. The study took place in a simulated living environment that is 12 ft. (3.8 m) by 17 ft. (5.2 m) with a 9 ft. (2.7 m) high open box ceiling designed for flexible lighting placement. It was furnished with two comfortable chairs facing a wall mounted TV. This allowed for left handed and right-handed subjects to have their hand movements captured by a wall mounted video camera and directed toward a table lamp on a small table between the chairs. The camera is barely visible as a blue dot above the TV in the photo (depicted in Fig. 1). There were two wall sconces, one on either side of the TV. There were two fake windows with birch trees and grass to give the feel of a lawn in an open neighborhood. These windows could be back illuminated with an edge-lit panel, or not, in order to represent day time or night. The windows remained illuminated throughout the study and the subjects all participated in the during daylight hours. Mounted in the ceiling were six independently controllable tunable fixtures that could produce a wide gamut of colors and variation in color temperature. A coffee table was placed in reach of the comfortable chairs with magazines for reading and the TV remote control. Some details such as artificial and real plants were added to complete the realistic living room environment shown in Fig. 1.

Fig. 1.
figure 1

Photo of living room

Full size image

The wizard control room was positioned behind the living room area behind a wall but the “wizards” could directly observe the subjects if need by peering around the wall while staying out of sight of the subjects but insight in the moderator and observer. The “wizards” could hear the subjects directly but had amplified audio and high resolution video from the wall mounted camera. The moderator conducted the experiment from a podium in the front and slightly to the right of the subjects, who were seated in one of the comfortable chairs. The observer was to the right of the subject and though peripherally, the observer could be seen when the subject turned their head and could on occasion add information, clarify points, or even converse with the subject.

Human Factors Experimental Procedure

To discover natural ways people use to control lighting, a procedure was developed whereby the subject was unaware of the control mechanisms and was not steered toward any particular vernacular or gestures. Internal subjects were used to refine the procedure and train the moderator, operators, and “wizards”. The moderators paid particular attention to making the script flow and sound natural. The observers listened closely for any moderator accidental prompts that would influence naming of light fixtures, colors, color temperature, or commands (trying not to say “save” or “store” or “setting” when we wanted the subject to save a preset was particularly challenging). The “wizards” were trained to try to quickly respond to the subjects actions. They practiced both doing what the subject wanted and intentionally not doing what the subject wanted in order to study subject response to failures, how they recover, and how it effects user experience. Both the moderators and the observers were trained in interview techniques for delving into thought processes behind the attempted actions and for discovering mental models behind those thought processes. Considerable effort was needed to coax the think-out-loud process to make sure the “wizards” could fully understand the intentions of the subjects. This was also necessary for the data analysis task of determining if the intent was met. Fortuitously natural naming conventions, light level and spectral changing commands, range and characteristics of sensor data and type of gestures, and the mental models employed were discovered, even in this preliminary phase and were verified in the external participant phase.

Each external participant was met in the lobby for briefing, signing and witnessing of forms, and then led to the simulated living room. In the living room the participant met the moderator and the observer (if different from the greeter), and were positioned in one of the comfy chairs according their handedness. The moderator read from a script (see Appendix A) that started with a brief description and introductions. Then again, verbal consent was obtained to start recording and the moderator asked the observer to start the recording even though the “wizards” were actually responsible and the observer pretended to start the recording with a remote control and confirmed it is recording. Part of the procedure was developed to intentionally deceive the subject into believing that there were no “wizards” and that the system was a highly intelligent controller. Then each high level task was introduced and instructions were given on how to proceed. The subjects were interviewed after completing all of the tasks, escorted to the lobby, thanked, and told the agency would pay them for their time.

3 Results

Observations and verbatims showed that most of the subjects were amazed at how well the system performed. Only one subject mentioned they thought that there may have been someone controlling the lighting. There were many commonalities in naming, commands, gestures, and choice of modality. Some common mental models emerged. Unique ways in which subjects did these things were studied in more detail for use in developing natural user interface prototypes.

One common way subjects named lights was by their proximity to other objects. Some examples are “lamp on the table”, “lights next to the TV”, and “ceiling lights”. Another common naming convention was by task, “reading light”. A third way is to name where or what the light from the source(s) illuminates. Examples of this are: “the light for the chairs”, “wall wash”, or commanding “light the room”.

Common commands that were intuitive such as “turn on” or just “on” outnumbered more obscure commands. “Make the ceiling blue” was one command that emerged that was not predicted ahead of time. The use of the commands “make” and “turn” was noted but not studied in detail.

Subjects showed dissatisfaction in unique ways. For example, when the lighting was too bright, the subjects would shield their eyes with their hand or arm in more of a dramatic gesture than to actually reduce the light level to their eyes. More common dissatisfaction reactions were saying things like “no, no, no”, or “not that” separately or in combination with waving the phone back and forth vigorously.

Nearly the same vertical motion was used to dim as to shut-off a light. The acceleration versus time was different. Figure 2 depicts a typical sequence for shutting off two different lights and dimming a third. The typical shape for both is two peaks one negative and one positive caused by the slight up and down motion at the ends of a mostly downward motion. By analyzing the video feeds that correspond to the accelerometer data and observer notes the subject’s intended actions could be categorized. The actions were categorized into three categories: dim, dim to off, and off. A statistically significant difference in peak to peak heights was found between off and the other categories, but not between dim and dim to off. This is graphically illustrated in Fig. 3.

Fig. 2.
figure 2

Vertical accelerometer data stream (Color figure online)

Full size image
Fig. 3.
figure 3

Statistical analysis of peak to peak vertical accelerometer data

Full size image

Cognitive models became most obvious when subjects were trying to use gestures to control color. Two major models were gleaned that correspond to peoples experience with color wheels and with color palettes. Some subjects would point the phone around a circle while observing the color change. Those subjects expected the direction or points on the circle to be arranged like color in a rainbow. Those same subjects described it as imagining a color wheel. Similarly some subjects had a color wheel in mind but used a faster circular motion to “spin the wheel” and a stopping or backing up motion to select the color they wanted.

The color pallet models were more complicated since the general understanding of additive and subtractive color space was not generally understood by the external participants. One subject explained in detail how the front corners of the living room were blue and red, and the rear of the room was yellow. Then by pointing between blue and yellow, green could be produced. Yet during this discussion, there was some confusion about how RGB displays mixed color. In additive color space, blue and yellow will produce white (for example the most typical white LED is made with blue LED light, some of which is converted to yellow using a phosphor). Some of the internal subjects (lighting experts) used the additive model consistently. Some subjects just waved the phone until the color happened to land where they wanted.

4 Conclusions

The results from this study confirm that people interact with lighting in ways that are unique compared to other devices. It is hypothesized that this is due to how central vision contributes to experience and to the nature of light itself in how it travels and reflects off of objects in the immediate world to provide people within this immediate environment with an understanding of how this environment is constructed. The results indicate that systems could be designed with better naming conventions, large vocabulary command sets, and color control based on cognitive color models.

The control system taking advantage of this invention would allow for multiple names and encourage multiple naming by these conventions. By prompting during the configuration phase, sometimes called commissioning, the system could encourage the programmer (commissioner) to consider giving multiple names for the light sources or groups of light sources by what they are next to, by what the light sources are used for, and by where the light source(s) illuminates. In this way, novice users of the system could refer to particular light by the natural naming convention.

When considering how to make a voice controlled system robust, adding more possibilities for commands must be balanced by increased misinterpretations of words from larger vocabularies.

Artificial lighting is critical in nearly every built environment and SSL is making advanced lighting products readily available while providing challenges in the application and control of these solutions (Boyce 2003). Future studies in other applications are under consideration.