Skip to main content
SpringerLink
Log in

Defining Trigger-Action Rules via Voice: A Novel Approach for End-User Development in the IoT

  • Conference paper
  • First Online:
End-User Development (IS-EUD 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13917))

Included in the following conference series:

Abstract

The possibility of personalizing devices and online services is important for end users living in smart environments, but existing End-User Development interfaces in this field often fail to provide users with the proper support, e.g., because they force users to deal with too many technological details. This paper explores novel approaches for personalizing IoT ecosystems via natural language and vocal interaction. We first conducted seven interviews to understand whether and how end users would converse with a conversational assistant to personalize their IoT ecosystems. Then, we designed and implemented two prototypes to define trigger-action rules through vocal and multimodal approaches. A usability study with 10 participants confirms the feasibility and effectiveness of personalizing the IoT via voice and opens the way to integrate personalization capabilities in smart speakers like Google Home and Amazon Echo.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Change history

  • 30 May 2023

    A correction has been published.

Notes

  1. 1.

    https://www.amazon.com/alexa-routines/b?ie=UTF8 &node=21442922011, last visited on February 16, 2023.

  2. 2.

    https://ifttt.com/, last visited on February 16, 2023.

  3. 3.

    https://zapier.com/, last visited on February 16, 2023.

  4. 4.

    https://cloud.google.com/dialogflow/docs/, last visited on February 21, 2023.

  5. 5.

    https://nodejs.org/en/, last visited on February 21, 2023.

  6. 6.

    https://developers.google.com/assistant/console, last visited on February 21, 2023.

References

  1. Ammari, T., Kaye, J., Tsai, J.Y., Bentley, F.: Music, search, and IoT: how people (really) use voice assistants. ACM Trans. Comput.-Hum. Interact. 26(3) (2019). https://doi.org/10.1145/3311956

  2. Barricelli, B.R., Casiraghi, E., Valtolina, S.: Virtual assistants for end-user development in the internet of things. In: Malizia, A., Valtolina, S., Morch, A., Serrano, A., Stratton, A. (eds.) IS-EUD 2019. LNCS, vol. 11553, pp. 209–216. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24781-2_17

    Chapter  Google Scholar 

  3. Barricelli, B.R., Fogli, D., Iemmolo, L., Locoro, A.: A multi-modal approach to creating routines for smart speakers. In: Proceedings of the 2022 International Conference on Advanced Visual Interfaces. AVI 2022, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3531073.3531168

  4. Barricelli, B.R., Valtolina, S.: Designing for end-user development in the internet of things. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 9–24. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18425-8_2

    Chapter  Google Scholar 

  5. Brich, J., Walch, M., Rietzler, M., Weber, M., Schaub, F.: Exploring end user programming needs in home automation. ACM Trans. Comput.-Hum. Interact. 24(2), 11:1-11:35 (2017). https://doi.org/10.1145/3057858

    Article  Google Scholar 

  6. Brooke, J.: SUS: A “quick and dirty” usability scale. In: Usability Evaluation in Industry, pp. 189–194. Taylor and Francis (1996). https://doi.org/10.1201/b15738-26

  7. Corno, F., De Russis, L., Monge Roffarello, A.: A high-level semantic approach to end-user development in the internet of things. Int. J. Hum.-Comput. Stud. 125(C), 41–54 (2019). https://doi.org/10.1016/j.ijhcs.2018.12.008

  8. Corno, F., De Russis, L., Monge Roffarello, A.: A high-level semantic approach to end-user development in the internet of things. Int. J. Hum.-Comput. Stud. 125, 41–54 (2019). https://doi.org/10.1016/j.ijhcs.2018.12.008

    Article  Google Scholar 

  9. Corno, F., De Russis, L., Monge Roffarello, A.: Recrules: recommending if-then rules for end-user development. ACM Trans. Intell. Syst. Technol. 10(5) (2019). https://doi.org/10.1145/3344211

  10. Corno, F., De Russis, L., Monge Roffarello, A.: HeyTAP: bridging the gaps between users’ needs and technology in IF-THEN rules via conversation. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3399715.3399905

  11. Corno, F., De Russis, L., Monge Roffarello, A.: TAPrec: supporting the composition of trigger-action rules through dynamic recommendations. In: Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 579–588. IUI 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3377325.3377499

  12. Corno, F., De Russis, L., Monge Roffarello, A.: Devices, information, and people: abstracting the internet of things for end-user personalization. In: Fogli, D., Tetteroo, D., Barricelli, B.R., Borsci, S., Markopoulos, P., Papadopoulos, G.A. (eds.) IS-EUD 2021. LNCS, vol. 12724, pp. 71–86. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-79840-6_5

    Chapter  Google Scholar 

  13. Corno, F., De Russis, L., Monge Roffarello, A.: From users’ intentions to if-then rules in the internet of things. ACM Trans. Inf. Syst. 39(4), 1–33 (2021). https://doi.org/10.1145/3447264

    Article  Google Scholar 

  14. Danado, J., Paternò, F.: Puzzle: a mobile application development environment using a jigsaw metaphor. J. Vis. Lang. Comput. 25(4), 297–315 (2014). https://doi.org/10.1016/j.jvlc.2014.03.005

    Article  Google Scholar 

  15. Daniel, F., Matera, M.: Mashups: Concepts. Models and Architectures. Springer, Cham (2014)

    Book  Google Scholar 

  16. Daniel, F., Matera, M., Pozzi, G.: Managing runtime adaptivity through active rules: the Bellerofonte framework. J. Web Eng. 7(3), 179–199 (2008)

    Google Scholar 

  17. De Russis, L., Corno, F.: Homerules: a tangible end-user programming interface for smart homes. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA 2015, ACM, New York, pp. 2109–2114 (2015). https://doi.org/10.1145/2702613.2732795

  18. De Russis, L., Monge Roffarello, A., Borsarelli, C.: Towards vocally-composed personalization rules in the IoT. In: Proceedings of the 2nd International Workshop on Empowering People in Dealing with Internet of Things Ecosystems (EMPATHY 2021) (2021). http://ceur-ws.org/Vol-3053/paper_1.pdf

  19. Desolda, G., Ardito, C., Matera, M.: Empowering end users to customize their smart environments: model, composition paradigms, and domain-specific tools. ACM Trans. Comput.-Hum. Interact. (TOCHI) 24(2), 121–1252 (2017). https://doi.org/10.1145/3057859

    Article  Google Scholar 

  20. Dey, A.K., Sohn, T., Streng, S., Kodama, J.: iCAP: interactive prototyping of context-aware applications. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) Pervasive 2006. LNCS, vol. 3968, pp. 254–271. Springer, Heidelberg (2006). https://doi.org/10.1007/11748625_16

    Chapter  Google Scholar 

  21. Gallo, S., Paterno, F.: A conversational agent for creating flexible daily automation. In Proceedings of the 2022 International Conference on Advanced Visual Interfaces, AVI 2022, Frascati, Rome, Italy, pp. 1-8. Association for Computing Machinery, New York (2022). ISBN: 9781450397193. https://doi.org/10.1145/3531073.3531090

  22. Huang, J., Cakmak, M.: Supporting mental model accuracy in trigger-action programming. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 215–225. UbiComp 2015, ACM, New York, NY, USA (2015). https://doi.org/10.1145/2750858.2805830

  23. Huang, T.H.K., Azaria, A., Bigham, J.P.: Instructablecrowd: creating if-then rules via conversations with the crowd. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1555–1562. CHI EA 2016, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2851581.2892502

  24. Le-Phuoc, D., Polleres, A., Hauswirth, M., Tummarello, G., Morbidoni, C.: Rapid prototyping of semantic mash-ups through semantic web pipes. In: Proceedings of the 18th International Conference on World Wide Web, pp. 581–590. WWW 2009, ACM, New York, NY, USA (2009). https://doi.org/10.1145/1526709.1526788

  25. Lieberman, H., Paternò, F., Klann, M., Wulf, V.: End User Development, chap. End-User Development: An Emerging Paradigm, pp. 1–8. Springer, Netherlands (2006). https://doi.org/10.1007/1-4020-5386-X_1

  26. Manca, M., Parvin, P., Paternò, F., Santoro, C.: Integrating Alexa in a rule-based personalization platform. In: Proceedings of the 6th EAI International Conference on Smart Objects and Technologies for Social Good, pp. 108–113. GoodTechs 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3411170.3411228

  27. Mattioli, A., Paternò, F.: A visual environment for end-user creation of IoT customization rules with recommendation support. In: Proceedings of the International Conference on Advanced Visual Interfaces. AVI 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3399715.3399833

  28. Munjin, D.: User Empowerment in the Internet of Things. Ph.D. thesis, Université de Genève (2013). http://archive-ouverte.unige.ch/unige:28951

  29. Srinivasan, V., Koehler, C., Jin, H.: Ruleselector: Selecting conditional action rules from user behavior patterns. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 2(1), 1–34 (2018). https://doi.org/10.1145/3191767

  30. Stolee, K.T., Elbaum, S.: Identification, impact, and refactoring of smells in pipe-like web mashups. IEEE Trans. Softw. Eng. 39(12), 1654–1679 (2013). https://doi.org/10.1109/TSE.2013.42

    Article  Google Scholar 

  31. Ur, B., McManus, E., Pak Yong Ho, M., Littman, M.L.: Practical trigger-action programming in the smart home. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 803–812. CHI 2014, ACM, New York, NY, USA (2014). https://doi.org/10.1145/2556288.2557420

  32. Ur, B., et al.: Trigger-action programming in the wild: an analysis of 200,000 ifttt recipes. In: Proceedings of the 34rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3227–3231. CHI 2016, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2858036.2858556

Download references

Acknowledgments

The authors want to thanks the 17 participants of the studies for their availability, and Carlo Borsarelli who helped with the creation of both prototypes as part of his M.S. thesis.

Author information

Authors and Affiliations

  1. Politecnico di Torino, Corso Duca degli Abruzzi, 24, 10129, Torino, Italy

    Alberto Monge Roffarello & Luigi De Russis

Authors
  1. Alberto Monge Roffarello
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luigi De Russis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alberto Monge Roffarello .

Editor information

Editors and Affiliations

  1. University of Cagliari, Cagliari, Italy

    Lucio Davide Spano

  2. LMU Munich, Munich, Germany

    Albrecht Schmidt

  3. ISTI-CNR, Pisa, Italy

    Carmen Santoro

  4. University of Glasgow, Glasgow, UK

    Simone Stumpf

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Monge Roffarello, A., De Russis, L. (2023). Defining Trigger-Action Rules via Voice: A Novel Approach for End-User Development in the IoT. In: Spano, L.D., Schmidt, A., Santoro, C., Stumpf, S. (eds) End-User Development. IS-EUD 2023. Lecture Notes in Computer Science, vol 13917. Springer, Cham. https://doi.org/10.1007/978-3-031-34433-6_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-34433-6_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-34432-9

  • Online ISBN: 978-3-031-34433-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation