Abstract
Emotions are inherent to any human activity, including human–computer interactions, and that is the reason why recognizing emotions expressed in natural language is becoming a key feature for the design of more natural user interfaces. In order to obtain useful corpora for this purpose, the manual classification of texts according to their emotional content has been the technique most commonly used by the research community. The use of corpora is widespread in Natural Language Processing, and the existing corpora annotated with emotions support the development, training and evaluation of systems using this type of data. In this paper we present the development of an annotated corpus oriented to the narrative domain, called EmoTales, which uses two different approaches to represent emotional states: emotional categories and emotional dimensions. The corpus consists of a collection of 1,389 English sentences from 18 different folk tales, annotated by 36 different people. Our model of the corpus development process includes a post-processing stage performed after the annotation of the corpus, in which a reference value for each sentence was chosen by taking into account the tags assigned by annotators and some general knowledge about emotions, which is codified in an ontology. The whole process is presented in detail, and revels significant results regarding the corpus such as inter-annotator agreement, while discussing topics such as how human annotators deal with emotional content when performing their work, and presenting some ideas for the application of this corpus that may inspire the research community to develop new ways to annotate corpora using a large set of emotional tags.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The term neutral will be used by annotators in those cases where they could not perceive any of the 119 proposed categories as a clearly identifiable emotion. For example, the sentence “the prince said” is usually annotated as neutral by most annotators.
References
Alm, C. O., & Sproat, R. (2005). Emotional sequencing and development in fairy tales. In First international conference on affective computing and intelligent interaction (pp. 668–674).
Alm, C. O., Roth, D., & Sproat, R. (2005). Emotions from text: Machine learning for text-based emotion prediction export find similar. In Proceedings of human language technology conference and conference on empirical methods in natural language processing (pp. 579–586).
Aman, S., & Szpakowicz, S. (2007). Identifying expressions of emotion in text. In V. Matousek, & P. Mautner (Eds.), Text, speech and dialogue. Lecture notes in computer science (Vol. 4629, pp. 196–205). Springer.
Aristotle (1960). Emotion and personality. New York: Columbia University Press.
Barreto, A. (2008). Non-intrusive physiological monitoring for affective sensing of computer users, human computer interaction: New developments. In K. Asai (Ed.).
Bestgen, Y. (1993). Can emotional valence be determined from words? Cognition and Emotion, 7, 21–36.
Blitzer, J., Dredze, M., & Pereira, F. (2007). Ciographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the Association for Computational Linguistics.
Busso, C., Bulut, M., Lee, C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008). Iemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42, 335–359.
Campbell, N. (2005). Getting to the heart of the matter: Speech as the expression of affect; rather than just text or language. Language Resources and Evaluation, 39, 109–118.
Carbonell, J. (1979). Subjective understanding: Computer models of belief systems. PhD thesis, Yale.
Cowie, R., & Cornelius, R. (2003). Describing the emotional states that are expressed in speech. Speech Communication Special Issue on Speech and Emotion, 40(1–2), 5–32.
Cowie, R., Douglas-Cowie, E., & Romano, A. (1999). Changing emotional tone in dialogue and its prosodic correlates. In Proceedings of the ESCA international workshop on dialogue and prosody, Veldhoven, The Netherlands (pp. 41–46).
Das, S., & Chen, M. (2001). Yahoo! for Amazon: Extracting market sentiment from stock message boards. In Proceedings of the Asia Pacific Finance Association annual conference.
Dave, K., Lawrence, S., & Pennock, D. M. (2003). Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of WWW (pp. 519–528).
Devillers, L., Vidrascu, L., & Lamel, L. (2005). Challenges in real-life emotion annotation and machine learning based detection. Neural Networks, 18(4), 407–422.
Douglas-Cowie, E., Campbell, N., Cowie, R., & Roach, P. (2003). Emotional speech: Towards a new generation of databases. Speech Communication, 40(1–2), 33–60.
Evens, M. (2002). New questions for circsim-tutor. In Symposium on natural language tutoring, University of Pittsburgh.
Ekman, P., Friesen, W. V., & Ellsworth, P. (1982). What emotion categories or dimensions can observers judge from facial behavior? In P. Ekman (Ed.), Emotion in the human face (pp. 39–55). New York: Cambridge University Press.
Fleiss, J. (1981). Statistical methods for rates and proportions. New York, USA: Wiley.
Fontaine, J., Scherer, K., Roesch, E., & Ellsworth, P. (2007). The world of emotion is not two-dimensional. Psychological Science, 18, 1050–1057.
Francisco, V., & Gervás, P. (2006). Automated mark up of affective information in english texts. In Text, speech and dialogue (pp. 375–382). Brno, Czech Republic: Springer.
Francisco, V., & Gervás, P. (2008). Ontology-supported automated mark up of affective information in texts. Special Issue of Language Forum on Computational Treatment of Language, 34(1), 23–36.
Francisco, V., & Hervás, R. (2007). Emotag: Automated mark up of affective information in texts. In EUROLAN 2007 Summer School Doctoral Consortium, Iasi, Romania (pp. 5–12).
Frijda, N. H. (1986). The emotions. New York: Cambridge University Press.
Grimm, M., & Kroschel, K. (2005). Evaluation of natural emotions using self assessment manikins. In IEEE automatic speech recognition and understanding workshop, San Juan, PuertoRico (pp. 381–385).
Heise, D. R. (1970). The semantic differential and attitude research. In G. F. Summers (Ed.), Attitude measurement (pp. 235–253). Chicago: Rand McNally.
Hoffman, T. (2008). Online reputation management is hot but is it ethical? Computer world.
Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. In Proceedings of the ACM SIGKDD conference on knowledge discovery and data mining (pp. 168–177).
Izard, C. E. (1971). The face of emotion. New York: Appleton-Century-Crofts.
Kready, L. F. (1916). A study of fairy tales. Houghton Mifflin Company. The Riverside Press. http://www.sacred-texts.com/etc/sft/index.htm.
Krenn, B., Pirker, H., Grice, M., Piwek, P., Deemter, K. V., Schröder, M., Klesen, M., & Gstrein, E. (2002). Generation of multimodal dialogue for net environments. In Proceedings of Konvens, Saarbrücken, Germany.
Landis, J., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
Lang, P. (1980). Behavioural treatment and bio-behavioural assessment: Computer applications. In J. B. Sidowski, J. H. Johnson, & T. A. Williams (Eds.), Technology in mental health care delivery systems (pp. 119–137). Norwood: Ablex Pub. Corp.
Liu, H., Lieberman, H., & Selker, T. (2003). A model of textual affect sensing using real-world knowledge. In Proceedings of intelligent user interfaces (IUI) (pp. 125–132).
Macdonald, C., & Ounis, I. (2006) The TREC Blogs06 collection: Creating and analyzing a blog test collection. Technical Report TR-2006-224, Department of Computer Science, University of Glasgow, 2006.
Merota, G. (2007). Emotional gestures in sport. Language Resources and Evaluation, 41, 233–254.
Ortony, A., Clore, G., & Collins, A. (1988). The cognitive structure of emotions. New York: Cambridge University Press.
Ortony, A., & Turner, T. J. (1990). What’s basic about basic emotions? Psychological Review, 97, 315–331.
Osgood, C. (1967). Contemporary bibliography of research related to the semantic differential technique. Urbana: University of Illinois Press.
Osgood, C. E., Suci, G., & Tannenbaum, P. (1957). The measurement of meaning. Urbana: University of Illinois Press.
Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Sentiment classification using machine learning techniques. In Conference on emprirical methods in natural language processing (pp. 79–86).
Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2), 1–135.
Parrott, W. (2001). Emotions in social psychology: Essential readings. Philadelphia, PA: Psychology Press.
Picard, R. (1997). Affective computing. Cambridge: MIT Press.
Picard, R. (2003). Affective computing: Challenges. International Journal of Human-Computer Studies, 59, 55–64.
Planet, S., Iriondo, I., Martínez, E., & Montero, J. (2008). TRUE: An online testing platform for multimedia evaluation. In Second international workshop on EMOTION: Corpora for research on emotion and affect at the 6th conference on language resources and evaluation.
Plutchik, R. (1980). A general psychoevolutionary theory of emotion. In R. Plutchik & H. Kellerman (Eds.), Emotion: Theory, research and experience: Vol 1. Theories of emotion (pp. 3–31). New York: Academic Press.
Popescu, A., & Etzioni, O. (2005). Extracting product features and opinions from reviews. In Proceedings of the conference on human language technology and empirical methods in natural language processing, Association for Computational Linguistics, Morristown, NJ, USA (pp. 339–346).
Quirk, R., Greenbaum, S., Leech, G., & Svartvik, J. (1985). A comprehensive grammar of the english language. Longman: New York.
Russell, J. (1980). A circumflex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178.
Sack, W. (1994). On the computation of point of view. In Proceedings of AAAI (p. 1488).
Scherer, K. R. (1984). On the nature and function of emotion: A component process approach (pp. 293–317).
Seki, Y., Kirk Evans, D., Ku, L. W., Chen, H. H., Kando, N., & Lin, C. Y. (2007). Overview of opinion analysis pilot task at NTCIR-6. In Proceedings of the references 131 workshop meeting of the National Institute of Informatics Test Collection for Information Retrieval Systems (pp. 265–278).
Shaver, P., Schwartz, J., Kirson, D., & O’Connor, C. (1987). Emotion knowledge: Further exploration of a prototype approach. Journal of Personality and Social Psychology, 52(6), 1061–1086.
Snyder, B., & Barzilay, R. (2007). Multiple aspect ranking using the good grief algorithm. In Proceedings of the joint human language technology/North American Chapter of the ACL conference (pp. 300–307).
Storm, C., & Storm, T. (1987). A taxonomic study of the vocabulary of emotions. Journal of Personality and Social Psychology, 53.
Strapparava, C., & Mihalcea, R. (2008). Learning to identify emotions in text. In Proceedings of the 2008 ACM symposium on applied computing (pp. 1556–1560). New York, NY, USA: ACM.
Tomkins, S. S. (1984). Affect theory. Approaches to emotion (pp. 163–195).
Turney, P. (2002). Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the Association for Computational Linguistics (pp. 417–424).
Turney, P., & Littman, M. (2003). Measuring praise and criticism: Interference of semantic orientation from association. ACM Transactions on Information Systems, 21(4), 315–346.
Wiebe, J. (1994). Tracking point of view in narrative. Computational Linguistics, 20, 233–287.
Wiebe, J., & Bruce, R. (1995). Probabilistic classifiers for tracking point of view. In Proceedings of the AAAI spring symposium on empirical methods in discourse interpretation and generation (pp. 181–187).
Wiebe, J., Bruce, R., & OHara, T. (1999). Development and use of a gold standard data set for subjectivity classifications. In Proceedings of the Association for Computational Linguistics (pp. 246–253).
Wiebe, J., & Rapaport, W. (1988). A computational theory of perspective and reference in narrative. In Proceedings of the Association for Computational Linguistics (pp. 131–138).
Wiebe, J., Wilson, T., & Cardie, C. (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2–3), 165–210.
Witt, A., Heid, U., Sasaki, F., & Sérasset, G. (2009). Multilingual language resources and interoperability. Language Resources and Evaluation, 43(1), 1–14.
Wright, A. (2009). Our sentiments, exactly. Communications of the ACM, 52(4), 14–15.
Yu, H., & Hatzivassiloglou, V. (2003). Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Proceedings of the conference on empirical methods in natural language processing.
Acknowledgments
This research is funded by the Spanish Ministry of Education and Science (TIN2006-14433-C02-01 project) and a joint research group grant (CCG08-UCM/TIC-4300) from the Universidad Complutense de Madrid and the Dirección General de Universidades e Investigación of the Comunidad Autónoma de Madrid.
We are very grateful to our annotators: Beatriz, Jesús, Lucia, Miguel, Susana, Alaukik, Juan Alvarado, Javier Arroyo, Cristina Arquiaga, Susana Bautista, María del Blanco, Pilar Bravo, Jorge Carrillo, Ana Casas, Alberto Díaz, Borja Foncillas, Ángela Francisco, Patricio Galera, David García, Pilar García, Silvia Garcia, Hector Gómez, Mónica Gónzalez, Francisco Guzman, Nuria Hernández, Jesús Herrera, Guillermo Jiménez, Carlos León, Álvaro Martín, Juanma Martín, Susana Martin, Gonzalo Méndez, Pablo Moreno, Laura Plaza, Celia Pérez, José Ramón Pérez, Patricia Sanz, Cristina Sobrados, Toñi Torreño and Miguel Vázquez. We would also like to thank Pablo Moreno for his useful insights and comments.
Author information
Authors and Affiliations
Corresponding author
Appendices: Instructions given to corpus annotators
Appendices: Instructions given to corpus annotators
Herein we show the instructions presented to the annotators. They were different depending on whether they were annotating the texts using emotional categories or emotional dimensions.
1.1 Appendix 1: Annotation using emotional categories
First of all thank you for your collaboration in this study about how people perceive the emotions that are conveyed in tales. Throughout the experiment, the different tales will appear divided into fragments. In each fragment the sentence you must mark at each moment will be highlighted in bold. You must label this sentence using one of the emotions that appear in the list of emotions that are available. In order to identify the beginning of a new story we have put the title of the story in red. Let’s try to make clear how this mark up must be done.
The intention is to identify emotions using emotional categories, that is, through the multiple words the language provides us for naming emotions. There are many such words, and a list of emotions is attached as an aid. If you think that a sentence does not convey any emotion you should mark it with the label “neutral”.
Please try to do it as fast as possible, and do not take much time thinking about each sentence. The purpose of this experiment is to find out how people interpret the feelings that a tale tries to convey in each sentence; therefore you should check what you interpret. You must use your first impression and not try to determine the “correct” answer or the answer that seems to make more sense because that answer does not exist. For example, the sentence “Cinderella’s mother died when she was a child” will be interpreted for some people as sad and the sentence will be marked with the category “sadness”. Others will interpret that the sentence conveys grief and they will mark it with the label “grief”. And there will be people who feel that this sentence does not attempt to convey anything and will mark it as neutral. None of these three answers is exclusively correct. All are equally valid. What you ought to think when annotating the tale is this: if I were a story-teller and I were reading this tale, what emotion would I give to each of the sentences to convey the corresponding emotional content to the listener?
Before starting we recommend that you carefully read the list of emotions that you will be provided to get an idea of the possibilities available and where they are located, thus it will be easier and faster to mark up each tale. The list is grouped “semantically”. For example, all words that identify sad emotions are in one group. This way it will be easier to find the emotional label you are seeking at any time. You can download this list here.
Keep in mind also that you do not have to mark all the tales at once. You can stop when you want and the point where you have stopped will be registered so you can resume the annotation whenever you want from the point where you stopped last time.
Thank you very much again for your collaboration.
1.2 Appendix 2: Annotation using emotional dimensions
First of all thank you for your collaboration in this study about how people perceive the emotions that are conveyed in tales. The aim of this study is to identify emotions by using so-called emotional dimensions. Throughout the experiment the different tales will appear divided into fragments. In each fragment the sentence you must mark at each moment will be highlighted in bold. In order to identify the beginning of a new story we have put the title of the story in red. Let’s try to make clear how this mark up must be done.
Each of the three emotional dimensions aims to identify a different type of feeling: joy vs. sadness (evaluation), excitation vs. calm (activation) and control vs. lack of control (power). Your job is to identify the emotion to be conveyed in each sentence in the tale through these three dimensions. If you were a storyteller reading these sentences, would you give the sentence a positive or a negative evaluation? More or less activation? Would you try to transmit control of the situation? In order to help you in this task you will be provided with the SAM scale of figures where you can see how the three emotional dimensions are represented and how this scale is numbered.
The first row of figures represents evaluation, ranging from a smiling figure to a figure with a sad frown. The left end of the scale conveys joy, happiness, satisfaction or optimism. If you feel that the sentence transmits joy it ought to be identified with this part of the scale (9). The other end represents emotions such as sadness, annoyance, dissatisfaction, melancholy, despair or boredom. You can indicate that a sentence attempts to convey sad feelings using the right side of the scale (1). The figures also allow you to describe feelings that are neither entirely happy nor entirely sad by using the figures between the extremes (4, 5, 6, …). If you think that the feeling a sentence is transmitting is between two figures, use the score that appears between them (8, 6, 4, 2) as shown in Fig. 7. This is a scale with a total of nine points and you must choose one of them to identify the evaluation conveyed in each sentence.
Intermediate values in the SAM scale
The second row of figures represents activation. The right side of the scale indicates that the sentence transmits stimulation, excitement or nervousness. When the sentence suggests nervousness you must select a point on the right (9). If we look at the left side of the scale we have the completely opposite feeling, as it conveys relaxation, calm, laziness or sleepiness. To indicate that a sentence conveys calm we will select this extreme of the scale (1). As in the previous scale for evaluation, average levels of arousal or calm can be represented through the figures in the middle of the scale (4, 5, 6, …). Similarly, if you think the excitement or calm transmitted by the sentence is between two figures you must use the numbering between them (8, 6, 4, 2). This is a scale with a total of nine points and you must choose one of them to identify the activation conveyed in each sentence.
The last row represents power. The left side of the scale conveys the feeling of being controlled, guided, intimidated or submissive. To indicate a feeling of total submission this part of the scale (1) will be chosen. The other side conveys control, influence, importance, or self-mastery. Keep in mind that a larger figure represents control and a smaller one represents submission. If you consider that a sentence neither represents control nor total submission you must use the midpoints of the scale (4, 5, 6, …). Again, you can use the scores between two figures if appropriate (8, 6, 4, 2). This is a scale with a total of nine points and you must choose one of them to identify the power conveyed by each sentences.
For example, how can the sentence “I’m angry so leave me alone!” be marked?
-
Evaluation: the speaker is angry. Anger involves a negative emotion so we can mark this sentence with the right end of the scale: 1.
-
Activation: the anger seems to be a situation that is very active, so we could mark up the sentence with the right side of the scale: 9.
-
Power: the speaker is trying to make someone leave him in peace, which leads us to believe that he is trying to dominate someone. So we could mark up the sentence with the right side of the power scale: 9.
To summarize, the annotation for the previous sentence could be the following (although this is not the only possibility; simply one of the options):
-
Evaluation = 1
-
Activation = 9
-
Power = 9
Please try to do it as fast as possible, and do not take too much time thinking about each sentence. The purpose of this experiment is to find out how people interpret the feelings that the tale conveys in each sentence; therefore you should choose your own interpretation of the emotion. You must use the first impression and not try to determine the “correct” answer or the answer that seems to make more sense because that answer does not exist.
Before starting with the first tale, we recommend that you become familiar with SAM; that way it will be easier and faster to mark each of the sentences.
Please also take into account that you do not have to mark all the tales at once. You can stop when you want and the point where you have stopped will be registered so you can resume the annotation whenever you want from the point where you stopped last time.
Again, thank you very much for your collaboration.
Rights and permissions
About this article
Cite this article
Francisco, V., Hervás, R., Peinado, F. et al. EmoTales: creating a corpus of folk tales with emotional annotations. Lang Resources & Evaluation 46, 341–381 (2012). https://doi.org/10.1007/s10579-011-9140-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10579-011-9140-5