skip to main content
research-article

Photo Sequences of Varying Emotion: Optimization with a Valence-Arousal Annotated Dataset

Published: 21 July 2021 Publication History

Abstract

Synthesizing photo products such as photo strips and slideshows using a database of images is a time-consuming and tedious process that requires significant manual work. To overcome this limitation, we developed a method that automatically synthesizes photo sequences based on several design parameters. Our method considers the valence and arousal ratings of images in conjunction with parameters related to both the visual consistency of the synthesized photo sequence and the progression of valence and arousal throughout the photo sequence. Our method encodes valence, arousal, and visual consistency parameters as cost terms into a total cost function while applying a Markov chain Monte Carlo optimization techniques called simulated annealing to synthesize the photo sequence based on user-defined target objectives in a few seconds. As our method was developed for the synthesis of photo sequences using the valence-arousal emotional model, a user study was conducted to evaluate the efficacy of the synthesized photo sequences in triggering valence-arousal ratings as expected. Our results indicate that the proposed method synthesizes photo sequences in which valence and arousal dimensions are perceived as expected by participants; however, valence may be more appropriately perceived than arousal.

References

[1]
Edoardo Ardizzone, Marco La Cascia, and Filippo Vella. 2008. A novel approach to personal photo album representation and management. In Multimedia Content Access: Algorithms and Systems II, Vol. 6820. International Society for Optics and Photonics, 682007.
[2]
Kristopher Blom and Steffi Beckhaus. 2005. Emotional storytelling. In Proceedings of the IEEE Virtual Reality Conference. 23–27.
[3]
L. Carretié, M. Tapia, S. López-Martín, and J. Albert. 2019. EmoMadrid: An emotional pictures database for affect research. Motiv. Emot. 43, 6 (2019), 929–939.
[4]
Tony F. Chan, Patrick Ciarlet Jr., and W. K. Szeto. 1997. On the optimality of the median cut spectral bisection graph partitioning method. SIAM J. Sci. Comput. 18, 3 (1997), 943–948.
[5]
Seah Chang, Chai-Youn Kim, and Yang Seok Cho. 2017. Sequential effects in preference decision: Prior preference assimilates current preference. PloS One 12, 8 (2017).
[6]
Jiajian Chen, Jun Xiao, and Yuli Gao. 2010. iSlideShow: a content-aware slideshow system. In Proceedings of the 15th International Conference on Intelligent User Interfaces. 293–296.
[7]
Jun-Cheng Chen, Wei-Ta Chu, Jin-Hau Kuo, Chung-Yi Weng, and Ja-Ling Wu. 2006. Tiling slideshow. In Proceedings of the 14th ACM International Conference on Multimedia. 25–34.
[8]
Siddhartha Chib and Edward Greenberg. 1995. Understanding the metropolis-hastings algorithm. Amer. Stat. 49, 4 (1995), 327–335.
[9]
Wei-Ta Chu and Chia-Hung Lin. 2009. Automatic summarization of travel photos using near-duplication detection and feature filtering. In Proceedings of the 17th ACM International Conference on Multimedia. 1129–1130.
[10]
Tammara T. A. Combs and Benjamin B. Bederson. 1999. Does zooming improve image browsing? In Proceedings of the 4th ACM Conference on Digital Libraries. 130–137.
[11]
Jeffrey Dalton, James Allan, and Pranav Mirajkar. 2013. Zero-shot video retrieval using content and concepts. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management. 1857–1860.
[12]
Andrew J. Elliot, Mark D. Fairchild, and Anna Franklin. 2015. Handbook of Color Psychology. Cambridge University Press.
[13]
Lisa A. Feldman. 1995. Valence focus and arousal focus: Individual differences in the structure of affective experience.J. Personal. Soc. Psychol. 69, 1 (1995), 153.
[14]
Robert Fergus, Li Fei-Fei, Pietro Perona, and Andrew Zisserman. 2005. Learning object categories from google’s image search. In Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05), Vol. 2. IEEE, 1816–1823.
[15]
Nico H. Frijda et al. 1986. The Emotions. Cambridge University Press.
[16]
Adrian Furnham. 1986. Response bias, social desirability and dissimulation. Personal. Individ. Differ. 7, 3 (1986), 385–400.
[17]
Yuan Gan, Yan Zhang, Zhengxing Sun, and Hao Zhang. 2020. Qualitative photo collage by quartet analysis and active learning. Comput. Graph. 38 (2020), 35–44.
[18]
Yuli Gao, Clayton Brian Atkins, Phil Cheatle, Jun Xiao, Xuemei Zhang, Hui Chao, Peng Wu, Daniel Tretter, David Slatter, Andrew Carter et al. 2009. MagicPhotobook: Designer inspired, user perfected photo albums. In Proceedings of the 17th ACM international conference on Multimedia. 979–980.
[19]
Joe Geigel and Alexander C. P. Loui. 2000. Automatic page layout using genetic algorithms for electronic albuming. In Internet Imaging II, Vol. 4311. International Society for Optics and Photonics, 79–90.
[20]
Arjan Gijsenij, Theo Gevers, and Joost Van De Weijer. 2011. Computational color constancy: Survey and experiments. IEEE Trans. Image Process. 20, 9 (2011), 2475–2489.
[21]
Gerardo Gonzalez Garcia and Rudy Lapeer. 2009. An evaluation of photo-consistency for intra-operative registration in an image enhanced surgical navigation (IESN) System. Proceedings of Medical Image Understanding and Analysis (MIUA’09). 229–233.
[22]
Paul Heckbert. 1982. Color image quantization for frame buffer display. ACM SIGGRAPH Comput. Graph. 16, 3 (1982), 297–307.
[23]
Winston H. Hsu, Lyndon S. Kennedy, and Shih-Fu Chang. 2007. Reranking methods for visual search. IEEE MultiMedia 14, 3 (2007), 14–22.
[24]
Jun Huang, Xiaokang Yang, Xiangzhong Fang, Weiyao Lin, and Rui Zhang. 2011. Integrating visual saliency and consistency for re-ranking image search results. IEEE Trans. Multimedia 13, 4 (2011), 653–661.
[25]
Ronald Hübner and Martin G. Fillinger. 2016. Comparison of objective measures for predicting perceptual balance and visual aesthetic preference. Front. Psychol. 7 (2016), 335.
[26]
Alejandro Jaimes and Shih-Fu Chang. 1998. Model-based classification of visual information for content-based retrieval. In Storage and Retrieval for Image and Video Databases VII, Vol. 3656. International Society for Optics and Photonics, 402–414.
[27]
Yushi Jing and Shumeet Baluja. 2008. Visualrank: Applying pagerank to large-scale image search. IEEE Trans. Pattern Anal. Mach. Intell. 30, 11 (2008), 1877–1890.
[28]
David S. Johnson, Cecilia R. Aragon, Lyle A. McGeoch, and Catherine Schevon. 1989. Optimization by simulated annealing: An experimental evaluation; part I, graph partitioning. Operat. Res. 37, 6 (1989), 865–892.
[29]
Kolbeinn Karlsson, Wei Jiang, and Dong-Qing Zhang. 2014. Mobile photo album management with multiscale timeline. In Proceedings of the 22nd ACM International Conference on Multimedia. 1061–1064.
[30]
Mel W. Khaw and David Freedberg. 2018. Continuous aesthetic judgment of image sequences. Acta Psychol. 188 (2018), 213–219.
[31]
Jinho Kim, Suan Lee, Ji-Seop Won, and Yang-Sae Moon. 2011. Photo cube: an automatic management and search for photos using mobile smartphones. In Proceedings of the IEEE 9th International Conference on Dependable, Autonomic and Secure Computing. IEEE, 1228–1234.
[32]
Kwanghwi Kim, Sora Kim, and Hwan-Gue Cho. 2012. A compact photo browser for smartphone imaging system with content-sensitive overlapping layout. In Proceedings of the 6th International Conference on Ubiquitous Information Management and Communication. 1–8.
[33]
Serkan Kiranyaz, Stefan Uhlmann, and Moncef Gabbouj. 2009. Dominant color extraction based on dynamic clustering by multi-dimensional particle swarm optimization. In Proceedings of the International Workshop on Content-Based Multimedia Indexing. IEEE, 181–188.
[34]
Scott Kirkpatrick, C. Daniel Gelatt, and Mario P. Vecchi. 1983. Optimization by simulated annealing. Science 220, 4598 (1983), 671–680.
[35]
Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. 2016. Photo aesthetics ranking network with attributes and content adaptation. In Proceedings of the European Conference on Computer Vision. Springer, 662–679.
[36]
Jean Kossaifi, Georgios Tzimiropoulos, Sinisa Todorovic, and Maja Pantic. 2017. AFEW-VA database for valence and arousal estimation in-the-wild. Image Vision Comput. 65 (2017), 23–36.
[37]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. MIT Press, 1097–1105.
[38]
Peter Kuppens, Francis Tuerlinckx, James A. Russell, and Lisa Feldman Barrett. 2013. The relation between valence and arousal in subjective experience.Psychol. Bull. 139, 4 (2013), 917.
[39]
Benedek Kurdi, Shayn Lozano, and Mahzarin R. Banaji. 2017. Introducing the open affective standardized image set (OASIS). Behav. Res. Methods 49, 2 (2017), 457–470.
[40]
Dmitry Kuzovkin, Tania Pouli, Rémi Cozot, Olivier Le Meur, Jonathan Kervec, and Kadi Bouatouch. 2018. Image selection in photo albums. In Proceedings of the ACM on International Conference on Multimedia Retrieval. 397–404.
[41]
Marco La Cascia, Marco Morana, and Salvatore Sorce. 2010. Mobile interface for content-based image management. In Proceedings of the International Conference on Complex, Intelligent and Software Intensive Systems. IEEE, 718–723.
[42]
Dong Liu, Xian-Sheng Hua, Linjun Yang, Meng Wang, and Hong-Jiang Zhang. 2009. Tag ranking. In Proceedings of the 18th International Conference on World Wide Web. 351–360.
[43]
Guang-Hai Liu, Zuo-Yong Li, Lei Zhang, and Yong Xu. 2011. Image retrieval based on micro-structure descriptor. Pattern Recogn. 44, 9 (2011), 2123–2133.
[44]
Paul J. Locher, Pieter Jan Stappers, and Kees Overbeeke. 1998. The role of balance as an organizing design principle underlying adults’ compositional strategies for creating visual displays. Acta Psychol. 99, 2 (1998), 141–161.
[45]
Hugo Lövheim. 2012. A new three-dimensional model for emotions and monoamine neurotransmitters. Med. Hypoth. 78, 2 (2012), 341–348.
[46]
Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z. Wang. 2014. Rapid: Rating pictorial aesthetics using deep learning. In Proceedings of the 22nd ACM international conference on Multimedia. 457–466.
[47]
Shuang Ma, Jing Liu, and Chang Wen Chen. 2017. A-lamp: Adaptive layout-aware multi-patch deep convolutional neural network for photo aesthetic assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4535–4544.
[48]
Albert Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Curr. Psychol. 14, 4 (1996), 261–292.
[49]
Albert Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Curr. Psychol. 14, 4 (1996), 261–292.
[50]
Ali Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor. 2017. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10, 1 (2017), 18–31.
[51]
Natali Moyal, Avishai Henik, and Gideon E. Anholt. 2018. Categorized affective pictures database (CAP-D). J. Cogn. 1, 1 (2018).
[52]
Wolfgang Nejdl and Claudia Niederee. 2015. Photos to remember, photos to forget. IEEE MultiMedia 22, 1 (2015), 6–11.
[53]
David Chek Ling Ngo, Azman Samsudin, and Rosni Abdullah. 2000. Aesthetic measures for assessing graphic screens. J. Info. Sci. Eng. 16, 1 (2000), 97–116.
[54]
Pere Obrador, Rodrigo De Oliveira, and Nuria Oliver. 2010. Supporting personal photo storytelling for social albums. In Proceedings of the 18th ACM international conference on Multimedia. 561–570.
[55]
Teresa K. Pegors, Marcelo G. Mattar, Peter B. Bryan, and Russell A. Epstein. 2015. Simultaneous perceptual and response biases on sequential face attractiveness judgments.J. Exper. Psychol.: Gen. 144, 3 (2015), 664.
[56]
Sang Phan, Duy-Dinh Le, and Shin’ichi Satoh. 2015. Multimedia event detection using event-driven multiple instance learning. In Proceedings of the 23rd ACM international conference on Multimedia. 1255–1258.
[57]
Robert Plutchik. 2001. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Amer. Sci. 89, 4 (2001), 344–350.
[58]
Mohamad Rabbath, Philipp Sandhaus, and Susanne Boll. 2011. Multimedia retrieval in social networks for photo book creation. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval. 1–2.
[59]
Rose M. Rider. 2010. Color psychology and graphic design applications. Senior Honors Theses 111. Liberty University. Retrieved From https://digitalcommons.liberty.edu/honors/111.
[60]
Kerry Rodden, Wojciech Basalaj, David Sinclair, and Kenneth Wood. 2001. Does organisation by similarity assist image browsing? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 190–197.
[61]
James A. Russell. 1980. A circumplex model of affect.J. Personal. Soc. Psychol. 39, 6 (1980), 1161.
[62]
Rob A. Rutenbar. 1989. Simulated annealing algorithms: An overview. IEEE Circ. Devices Mag. 5, 1 (1989), 19–26.
[63]
Fereshteh Sadeghi, J. Rafael Tena, Ali Farhadi, and Leonid Sigal. 2015. Learning to select and order vacation photographs. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. IEEE, 510–517.
[64]
Mukesh Kumar Saini, Fatimah Al-Zamzami, and Abdulmotaleb El Saddik. 2014. Towards storytelling by extracting social information from OSN photo’s metadata. In Proceedings of the 1st International Workshop on Internet-Scale Multimedia Management. 15–20.
[65]
Carl Emil Seashore. 1908. Elementary Experiments in Psychology. Holt.
[66]
Feng Shao, Mei Yu, and Gangyi Jiang. 2007. Dominant color extraction based color correction for multi-view images. Chinese Optics Lett. 5, 8 (2007), 449–451.
[67]
Pinaki Sinha, Hamed Pirsiavash, and Ramesh Jain. 2009. Personal photo album summarization. In Proceedings of the 17th ACM international conference on Multimedia. 1131–1132.
[68]
Terry Lee Stone, Sean Adams, and Noreen Morioka. 2008. Color Design Workbook: A Real world Guide to Using Color in Graphic Design. Rockport Pub.
[69]
Pablo P. L. Tinio and Helmut Leder. 2009. Just how stable are stable aesthetic features? Symmetry, complexity, and the jaws of massive familiarization. Acta Psychol. 130, 3 (2009), 241–250.
[70]
Cody Tousignant and Glen E. Bodner. 2014. Context effects on beauty ratings of photos: Building contrast effects that erode but cannot be knocked down.Psychol. Aesthet. Creat. Arts 8, 1 (2014), 81.
[71]
Cody Tousignant and Glen E. Bodner. 2018. Context effects on beauty ratings of abstract paintings: Contrast, contrast, everywhere!Psychol. Aesthet. Creat. Arts 12, 3 (2018), 369.
[72]
Tiberio Uricchio, Marco Bertini, Lorenzo Seidenari, and Alberto Bimbo. 2015. Fisher encoded convolutional bag-of-windows for efficient image retrieval and social image tagging. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 9–15.
[73]
Lujin Wang, Joachim Giesen, Kevin T. McDonnell, Peter Zolliker, and Klaus Mueller. 2008. Color design for illustrative visualization. IEEE Trans. Visual. Comput. Graph. 14, 6 (2008), 1739–1754.
[74]
Mark D. Wood. 2008. Exploiting semantics for personalized story creation. In Proceedings of the IEEE International Conference on Semantic Computing. IEEE, 402–409.
[75]
Mark D. Wood, Madirakshi Das, Peter O. Stubler, and Alexander C. Loui. 2016. Event-enabled intelligent asset selection and grouping for photobook creation. Image Vision Comput. 53 (2016), 57–67.
[76]
Xiaolin Wu. 1991. Efficient statistical computations for optimal color quantization. In Graphics Gems II. Elsevier, 126–133.
[77]
Yuanjun Xiong, Kai Zhu, Dahua Lin, and Xiaoou Tang. 2015. Recognize complex events from static images by fusing deep channels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1600–1609.
[78]
Shao-Fu Xue. 2015. Aesthetics of photographs, photobooks, and magazine covers: Tools for autonomous quality evaluation and content creation. Ph.D. Dissertation. Purdue University.
[79]
Nai-Chung Yang, Wei-Han Chang, Chung-Ming Kuo, and Tsia-Hsing Li. 2008. A fast MPEG-7 dominant color extraction with new similarity measure for image retrieval. J. Vis. Commun. Image Represent. 19, 2 (2008), 92–105.
[80]
Seungji Yang, Sihyoung Lee, Yong Man Ro, and Sang-Kyun Kim. 2007. Semantic photo album based on MPEG-4 compatible application format. In Proceedings of the International Conference on Consumer Electronics. IEEE, 1–2.
[81]
Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, and Shipeng Li. 2016. Automatic generation of visual-textual presentation layout. ACM Trans. Multimedia Comput. Commun. Appl. 12, 2 (2016), 1–22.
[82]
Jun Yu, Xiaokang Yang, Fei Gao, and Dacheng Tao. 2016. Deep multimodal distance metric learning using click constraints for image ranking. IEEE Trans. Cybernet. 47, 12 (2016), 4014–4024.
[83]
Lei Zhang, Le Chen, Feng Jing, Kefeng Deng, and Wei-Ying Ma. 2006. EnjoyPhoto: A vertical image search engine for enjoying high-quality photos. In Proceedings of the 14th ACM international conference on Multimedia. 367–376.

Cited By

View all
  • (2023)Synthesizing Game Levels for Collaborative Gameplay in a Shared Virtual EnvironmentACM Transactions on Interactive Intelligent Systems10.1145/355877313:1(1-36)Online publication date: 9-Mar-2023
  • (2022)Affective Image Sequence Viewing in Virtual Reality Theater Environment: Frontal Alpha Asymmetry Responses From Mobile EEGFrontiers in Virtual Reality10.3389/frvir.2022.8954873Online publication date: 19-Jul-2022
  • (2022)Procedural Game Level Design to Trigger Spatial ExplorationProceedings of the 17th International Conference on the Foundations of Digital Games10.1145/3555858.3563272(1-11)Online publication date: 5-Sep-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems
ACM Transactions on Interactive Intelligent Systems  Volume 11, Issue 2
June 2021
267 pages
ISSN:2160-6455
EISSN:2160-6463
DOI:10.1145/3465444
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 July 2021
Accepted: 01 March 2021
Revised: 01 November 2020
Received: 01 April 2020
Published in TIIS Volume 11, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Valence
  2. arousal
  3. photo sequence
  4. visual consistency
  5. optimization
  6. simulated annealing

Qualifiers

  • Research-article
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)31
  • Downloads (Last 6 weeks)2
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Synthesizing Game Levels for Collaborative Gameplay in a Shared Virtual EnvironmentACM Transactions on Interactive Intelligent Systems10.1145/355877313:1(1-36)Online publication date: 9-Mar-2023
  • (2022)Affective Image Sequence Viewing in Virtual Reality Theater Environment: Frontal Alpha Asymmetry Responses From Mobile EEGFrontiers in Virtual Reality10.3389/frvir.2022.8954873Online publication date: 19-Jul-2022
  • (2022)Procedural Game Level Design to Trigger Spatial ExplorationProceedings of the 17th International Conference on the Foundations of Digital Games10.1145/3555858.3563272(1-11)Online publication date: 5-Sep-2022

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media