Tan Tang
Skip Abstract Section
Abstract
Videos are well-received methods for storytellers to communicate various narratives. To further engage viewers, we introduce a novel visual medium where data visualizations are embedded into videos to present data insights. However, creating such data-driven videos requires professional video editing skills, data visualization knowledge, and even design talents. To ease the difficulty, we propose an optimization method and develop SmartShots, which facilitates the automatic integration of in-video visualizations. For its development, we first collaborated with experts from different backgrounds, including information visualization, design, and video production. Our discussions led to a design space that summarizes crucial design considerations along three dimensions: visualization, embedded layout, and rhythm. Based on that, we formulated an optimization problem that aims to address two challenges: (1) embedding visualizations while considering both contextual relevance and aesthetic principles and (2) generating videos by assembling multi-media materials. We show how SmartShots solves this optimization problem and demonstrate its usage in three cases. Finally, we report the results of semi-structured interviews with experts and amateur users on the usability of SmartShots.
- [1] . 2013. How Consumers Shop with Video: Based on a 4Q 2012 Research Study of 1000 Consumers.
Technical Report .Google Scholar
- [2] . 2019. G2. Retrieved March 31, 2019 from https://github.com/antvis/g2.Google Scholar
- [3] . 2019. OpenCV. Retrieved March 31, 2019 from https://opencv.org/.Google Scholar
- [4] . 2019. React. Retrieved March 31, 2019 from https://reactjs.org/.Google Scholar
- [5] . 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation. 265–283.Google Scholar
- [6] . 2005. Amazon Mechanical Turk. Retrieved March 31, 2019 from https://www.mturk.com/.Google Scholar
- [7] . 2015. Understanding data videos: Looking at narrative visualization through the cinematography lens. In Proceedings of ACM Conference on Human Factors in Computing Systems. 1459–1468.Google ScholarDigital Library
- [8] . 2018. Hooked on data videos: Assessing the effect of animation and pictographs on viewer engagement. In Proceedings of the Working Conference on Advanced Visual Interfaces. 1–9.Google Scholar
- [9] . 2017. Authoring data-driven videos with dataclips. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017), 501–510.Google ScholarDigital Library
- [10] . 2009. Aesthetic measure of alignment and regularity. In Proceedings of ACM Symposium on Document Engineering. 56–65.Google ScholarDigital Library
- [11] . 1985. Video composition method and apparatus.
US Patent 4,538,188. Google Scholar - [12] . 2014. Understanding typescript. In Proceedings of the European Conference on Object-Oriented Programming. 257–281.Google ScholarDigital Library
- [13] . 2011. D\( ^3 \): Data-driven documents. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2301–2309.Google ScholarDigital Library
- [14] . 2003. Using Video in Teacher Education. Emerald Group Publishing Limited.Google ScholarCross Ref
- [15] . 2020. MARVisT: Authoring glyph-based visualization in mobile augmented reality. IEEE Transactions on Visualization and Computer Graphics 26, 8 (2020), 2645–2658.Google ScholarDigital Library
- [16] . 2003. Automatic video summarization by graph modeling. In Proceedings of IEEE Conference on Computer Vision. 104–109.Google Scholar
- [17] . 2020. Infomages: Embedding data into thematic images. Computer Graphics Forum 39, 3 (2020), 593–606.Google ScholarCross Ref
- [18] . 2006. Color harmonization. ACM Transactions on Graphics 25, 3 (2006), 624–630.Google ScholarDigital Library
- [19] . 2009. Introduction to Algorithms (3rd ed.). MIT Press, Cambridge, MA.Google Scholar
- [20] . 2018. The State of World’s Forests. Retrieved March 31, 2019 from http://www.fao.org/state-of-forests/en.Google Scholar
- [21] . 2020. Canis: A high-level language for data-driven chart animations. Computer Graphics Forum 39, 3 (2020), 607–617.Google ScholarCross Ref
- [22] . 2009. AdOn: An intelligent overlay video advertising system. In Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval. 628–629.Google ScholarDigital Library
- [23] . 2019. China and India Help Make Planet Leafier. Retrieved March 31, 2019 from https://www.bbc.com/news/science-environment-47210849.Google Scholar
- [24] . 2003. ColorBrewer.org: An online tool for selecting colour schemes for maps. Cartographic Journal 40, 1 (2003), 27–37.Google Scholar
- [25] . 2007. Animated transitions in statistical data graphics. IEEE Transactions on Visualization and Computer Graphics 13, 6 (2007), 1240–1247.Google ScholarDigital Library
- [26] . 2020. SmartOverlays: A visual saliency driven label placement for intelligent human-computer interfaces. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 1121–1130.Google ScholarCross Ref
- [27] . 2015. Speaker-following video subtitles. ACM Transactions on Multimedia Computing, Communications, and Applications 11, 2 (2015), 1–17.Google ScholarDigital Library
- [28] . 2017. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7310–7311.Google ScholarCross Ref
- [29] . 2013. Storytelling: The next step for visualization. IEEE Computer 46, 5 (2013), 44–50.Google ScholarDigital Library
- [30] . 2020. A view on the viewer: Gaze-adaptive captions for videos. In Proceedings of ACM Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- [31] . 2015. More than telling a story: Transforming data into visually shared stories. IEEE Computer Graphics and Applications 35, 5 (2015), 84–90.Google ScholarDigital Library
- [32] . 2010. Proximity. In Universal Principles of Design. Rockport Publishers, Beverly, MA, 196–197.Google Scholar
- [33] . 2004. Evaluation of visual balance for automated layout. In Proceedings of the ACM Conference on Intelligent User Interfaces. 101–108.Google ScholarDigital Library
- [34] . 2020. Illustrating changes in time-series data with data video. IEEE Computer Graphics and Applications 40, 2 (2020), 18–31.Google ScholarDigital Library
- [35] . 2015. librosa: Audio and music signal analysis in Python. In Proceedings of the Python in Science Conference. 18–25.Google Scholar
- [36] . 2019. Information placement in virtual reality. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces. IEEE, Los Alamitos, CA, 1765–1769.Google Scholar
- [37] . 2007. VideoSense: Towards effective online video advertising. In Proceedings of the ACM International Conference on Multimedia. 1075–1084.Google ScholarDigital Library
- [38] . 2014. Visualization Analysis and Design. CRC Press, Boca Raton, FL.Google ScholarCross Ref
- [39] . 2014. Learning layouts for single-page graphic designs. IEEE Transactions on Visualization and Computer Graphics 20, 8 (2014), 1200–1213.Google ScholarDigital Library
- [40] . 2015. DesignScape: Design with interactive layout suggestions. In Proceedings of the ACM Conference on Human Factors in Computing Systems. 1221–1224.Google ScholarDigital Library
- [41] . 2000. The impact of technology on journalism. Journalism Studies 1, 2 (2000), 229–237.Google ScholarCross Ref
- [42] . 2003. Creating personalized documents: An optimization approach. In Proceedings of the ACM Symposium on Document Engineering. 68–77.Google ScholarDigital Library
- [43] . 2017. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 6 (2017), 1137–1149.Google ScholarDigital Library
- [44] . 2009. Gapminder. Gapminder Foundation. Retrieved March 31, 2019 from http://www.gapminder.org.Google Scholar
- [45] . 2010. Narrative visualization: Telling stories with data. IEEE Transactions on Visualization and Computer Graphics 16, 6 (2010), 1139–1148.Google ScholarDigital Library
- [46] . 2019. iStoryline: Effective convergence to hand-drawn storylines. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 769–778.Google ScholarDigital Library
- [47] . 2020. Design guidelines for augmenting short-form videos using animated data visualizations. Journal of Visualization 23 (2020), 707–720.Google ScholarDigital Library
- [48] . 2020. SmartShots: Enabling automatic generation of videos with data visualizations embedded. In Proceedings of the 28th ACM International Conference on Multimedia. 4509–4511.Google ScholarDigital Library
- [49] . 2009. VideoSense: A contextual in-video advertising system. IEEE Transactions on Circuits and Systems for Video Technology 19, 12 (2009), 1866–1879.Google ScholarDigital Library
- [50] . 2008. Gestalt principles. Scholarpedia 3, 12 (2008), 5345.Google ScholarCross Ref
- [51] . 2002. Color design support system considering color harmony. In Proceedings of the IEEE Conference on Fuzzy Systems. 378–383.Google ScholarCross Ref
- [52] . 2010. A survey of augmented reality: Technologies, applications, and limitations. International Journal of Virtual Reality 9, 2 (2010), 1.Google ScholarCross Ref
- [53] . 2006. Fully and semi-automatic music sports video composition. In Proceedings of the IEEE Conference on Multimedia and Expo. 1897–1900.Google ScholarCross Ref
- [54] . 2005. Automatic generation of personalized music sports video. In Proceedings of the ACM Conference on Multimedia. 735–744.Google ScholarDigital Library
- [55] . 2019. Optimizing color assignment for perception of class separability in multiclass scatterplots. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 820–829.Google ScholarDigital Library
- [56] . 2018. Line graph or scatter plot? Automatic selection of methods for visualizing trends in time series. IEEE Transactions on Visualization and Computer Graphics 24, 2 (2018), 1141–1154.Google ScholarCross Ref
- [57] . 2019. Vlog. Retrieved March 31, 2019 from https://en.wikipedia.org/wiki/Vlog.Google Scholar
- [58] . 2017. Embedded data representations. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017), 461–470.Google ScholarDigital Library
- [59] . 2014. CAVVA: Computational affective video-in-video advertising. IEEE Transactions on Multimedia 16, 1 (2014), 15–23.Google ScholarCross Ref
- [60] . 2016. Automatic generation of visual-textual presentation layout. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 2 (2016), 1–22.Google ScholarDigital Library
- [61] . 2017. Computational aesthetic evaluation of logos. ACM Transactions on Applied Perception 14, 3 (2017), 1–21.Google ScholarDigital Library
- [62] . 2020. DataQuilt: Extracting visual elements from images to craft pictorial visualizations. In Proceedings of the ACM Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- [63] . 2017. Layout style modeling for automating banner design. In Proceedings of the ACM Conference on Multimedia Thematic Workshops. 451–459.Google ScholarDigital Library
- [64] . 2021. Preserving minority structures in graph sampling. IEEE Transactions on Visualization and Computer Graphics 27 (2021), 1698–1708.Google ScholarCross Ref
Index Terms
- SmartShots: An Optimization Approach for Generating Videos with Data Visualizations Embedded
Recommendations
SmartShots: Enabling Automatic Generation of Videos with Data Visualizations Embedded
MM '20: Proceedings of the 28th ACM International Conference on MultimediaVideos become prevalent for storytellers to inspire viewers' interests. To further enhance narrations, visualizations are integrated into videos to present data-driven insights. However, manually crafting such data-driven videos is difficult and time-...
Deconstructing and restyling D3 visualizations
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technologyThe D3 JavaScript library has become a ubiquitous tool for developing visualizations on the Web. Yet, once a D3 visualization is published online its visual style is difficult to change. We present a pair of tools for deconstructing and restyling ...
How Information Visualization Novices Construct Visualizations
It remains challenging for information visualization novices to rapidly construct visualizations during exploratory data analysis. We conducted an exploratory laboratory study in which information visualization novices explored fictitious sales data by ...
Comments