skip to main content
10.1145/3176349.3176879acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
short-paper

How to Evaluate Humorous Response Generation, Seriously?

Published: 01 March 2018 Publication History

Abstract

Nowadays natural language user interfaces, such as chatbots and conversational agents, are very common. A desirable trait of such applications is a sense of humor. It is, therefore, important to be able to measure quality of humorous responses. However, humor evaluation is hard since humor is highly subjective. To address this problem, we conducted an online evaluation of 30 dialog jokes from different sources by almost 300 participants -- volunteers and Mechanical Turk workers. We collected joke ratings along with participants» age, gender, and language proficiency. Results show that demographics and joke topics can partly explain variation in humor judgments. We expect that these insights will aid humor evaluation and interpretation. The findings can also be of interest for humor generation methods in conversational systems.

References

[1]
Salvatore Attardo. 1994. Linguistic Theories of Humor. Walter de Gruyter.
[2]
Angelo Basile et al. 2017. N-GrAM: New Groningen Author-profiling Model. In CLEF'2017 Evaluation Labs and Workshop.
[3]
Nancy Bell and Salvatore Attardo. 2010. Failed humor: Issues in non-native speakers' appreciation and understanding of humor. Intercultural Pragmatics 7, 3 (2010), 423--447.
[4]
Jerome R Bellegarda. 2014. Spoken Language Understanding for Natural Interaction: The Siri Experience. In Natural Interaction with Robots, Knowbots and Smartphones. 3--14.
[5]
Kim Binsted. 1995. Using humour to make natural language interfaces more friendly. In AI, ALife and Entertainment Workshop.
[6]
Vladislav Blinov, Kirill Mishchenko, Valeria Bolotova, and Pavel Braslavski. 2017. A Pinch of Humor for Short-Text Conversation: an Information Retrieval Approach. In CLEF. 3--15.
[7]
Ken Goldberg, Theresa Roeder, Dhruv Gupta, and Chris Perkins. 2001. Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval 4, 2 (2001), 133--151.
[8]
Samuel D Gosling, Peter J Rentfrow, and William B Swann. 2003. A very brief measure of the Big-Five personality domains. J. Res. Pers. 37, 6 (2003), 504--528.
[9]
Bryan Anthony Hong and Ethel Ong. 2009. Automatically extracting word relationships as templates for pun generation. In CALC. 24--31.
[10]
Jiepu Jiang et al. 2015. Automatic online evaluation of intelligent assistants. In WWW. 506--516.
[11]
Peter Khooshabeh et al. 2011. Does it matter if a computer jokes. In CHI. 77--86.
[12]
Chia-Wei Liu et al. 2016. How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. In EMNLP. 2122--2132.
[13]
Rod A. Martin. 2007. The Psychology of Humor: An Integrative Approach. Elsevier.
[14]
Rada Mihalcea and Stephen Pulman. 2007. Characterizing Humour: An Exploration of Features in Humorous Texts. In CICLing. 337--347.
[15]
Rada Mihalcea and Carlo Strapparava. 2006. Learning to laugh (automatically): Computational models for humor recognition. Computational Intelligence 22, 2 (2006), 126--142.
[16]
Tavi Nathanson, Ephrat Bitton, and Ken Goldberg. 2007. Eigentaste 5.0: constanttime adaptability in a recommender system using item clustering. In RecSys. 149--152.
[17]
Peter Potash, Alexey Romanov, and Anna Rumshisky. 2017. SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor. In SemEval. 49--57.
[18]
Willibald Ruch. 1992. Assessment of appreciation of humor: Studies with the 3WD humor test. Advances in personality assessment 9 (1992), 27--75.
[19]
Dafna Shahaf, Eric Horvitz, and Robert Mankoff. 2015. Inside jokes: Identifying humorous cartoon captions. In PKDD. 1065--1074.
[20]
Oliviero Stock and Carlo Strapparava. 2005. HAHAcronym: a computational humor system. In ACL (demo). 113--116.
[21]
Alessandro Valitutti et al. 2013. 'Let Everything Turn Well in Your Wife': Generation of Adult Humor Using Lexical Constraints. In ACL (2). 243--248.
[22]
Miaomiao Wen et al. 2015. OMG UR Funny! Computer-Aided Humor with an Application to Chat. In ICCC. 86--93.
[23]
Diyi Yang, Alon Lavie, Chris Dyer, and Eduard Hovy. 2015. Humor Recognition and Humor Anchor Extraction. In EMNLP. 2367--2376.

Cited By

View all
  • (2024)The impact of smart voice assistants' humor styles on perceived moralityKorean Journal of Human Ecology10.5934/kjhe.2024.33.6.90333:6(903-914)Online publication date: 31-Dec-2024
  • (2023)Digitale Methoden in Bildungsforschung und BildungspraxisDatafizierung (in) der Bildung10.14361/9783839465820-006(81-102)Online publication date: 4-Dec-2023
  • (2023)Tickling Proactivity: Exploring the Use of Humor in Proactive Voice AssistantsProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627777(294-320)Online publication date: 3-Dec-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHIIR '18: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval
March 2018
402 pages
ISBN:9781450349253
DOI:10.1145/3176349
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 March 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. computational humor
  2. conversational systems
  3. crowdsourcing
  4. evaluation

Qualifiers

  • Short-paper

Conference

CHIIR '18
Sponsor:

Acceptance Rates

CHIIR '18 Paper Acceptance Rate 22 of 57 submissions, 39%;
Overall Acceptance Rate 55 of 163 submissions, 34%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)65
  • Downloads (Last 6 weeks)12
Reflects downloads up to 24 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)The impact of smart voice assistants' humor styles on perceived moralityKorean Journal of Human Ecology10.5934/kjhe.2024.33.6.90333:6(903-914)Online publication date: 31-Dec-2024
  • (2023)Digitale Methoden in Bildungsforschung und BildungspraxisDatafizierung (in) der Bildung10.14361/9783839465820-006(81-102)Online publication date: 4-Dec-2023
  • (2023)Tickling Proactivity: Exploring the Use of Humor in Proactive Voice AssistantsProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627777(294-320)Online publication date: 3-Dec-2023
  • (2023)Much Ado About GenderProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578316(269-279)Online publication date: 19-Mar-2023
  • (2023)“Funny How?” A Serious Look at Humor in Conversational AgentsProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3603761(1-7)Online publication date: 19-Jul-2023
  • (2023)Humorous Robotic Behavior as a New Approach to Mitigating Social AwkwardnessProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580821(1-16)Online publication date: 19-Apr-2023
  • (2023)The JOKER Corpus: English-French Parallel Data for Multilingual Wordplay RecognitionProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591885(2796-2806)Online publication date: 19-Jul-2023
  • (2023): A Multilingual Humor-aided Multiparty Dialogue Generation in multimodal conversational settingKnowledge-Based Systems10.1016/j.knosys.2023.110840278(110840)Online publication date: Oct-2023
  • (2022)Multidimensional Latent Semantic Networks for Text Humor RecognitionSensors10.3390/s2215550922:15(5509)Online publication date: 23-Jul-2022
  • (2022)“Alexa, Do You Want to Build a Snowman?” Characterizing Playful Requests to Conversational AgentsExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519870(1-7)Online publication date: 27-Apr-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media