Fast content-aware resizing of multi-layer information visualization via adaptive triangulation

Abstract

Visual graphics and image-based content have become the pervasive modes of interaction with the digital information flow. With the immense proliferation of display systems and devices, visual content representation has become increasingly challenging. Classical static image resizing algorithms are not directly suitable for the current dynamic information visualization of streaming data flows and processes because most of the visual content often consists of superimposed, multi-layered, multi-scale structure. In this paper, we propose a new adaptive method for content-aware resizing of visual information flow. Scaling is performed by deforming a hierarchical triangle mesh that matches the visual saliency map (VSM) of the streaming data. The VSM is generated automatically based on a series of predefined rules operating on a triangular mesh representation of visual features. We present a linear energy function to minimize distortions of the triangular deformations to perceptually preserve informative content. Through multiple experiments on real datasets, we show that the method has both high performance as well as high robustness in the presence of large differences in the visual aspect ratios between target displays.

Introduction

Content-aware resizing is an adaptive technique in image processing that filters out less important content and retains more important ones. This technique has also become a useful tool for information visualization because the diversity of displays for hardware is increasing. In addition, virtual displays of arbitrary size or aspect ratio require content-aware resizing techniques. Although artists, web designers, and programmers can design several available layouts for different scenarios, the task is time-consuming and costly.

The main objective of this work is to perceptively adjust the data output to any size or aspect ratio of a target display in the context of multi-layered information visualization applications. The basic resizing techniques such as cropping and linear scaling often result in information loss and visual distortions. Cropping (Fig. 1b) is the simplest operation for visualization resizing. Cropping can be used to adapt to different types of displays. However, cropping often removes important content. Linear scaling (Fig. 1c) is another approach. However, distortions often appear when more important content regions have the same scaling rate as less important regions. Missing content and added distortions in visualizations quickly lead to loss of attention, or worse, the complete misinterpretation of the information presented. Hence, it is paramount to adopt a robust approach that not only resizes the content appropriately but also retains the important content in information visualization.

Existing approaches for content-aware resizing mainly focus on natural images such as portraits, landscapes, and buildings. On the other hand, in information visualization, images normally consist of abstract mathematical representations such as vectors, points, lines, icons and geometrical shapes. The important content often represents the main subject in visual content. Fig. 1(e) shows an example of content-aware resizing of visual information produced by our method.

Most of the existing image resizing approaches are not entirely suitable for information visualization. Grid-based methods [1], as shown in Fig. 1(d), have been used for resizing such images as geometric distortions and are easily identified. Pixel-based seam carving, Avidan et al. [2], is normally used for image resizing. This technique cannot be easily extended to account for layout adjustments of geographical scatterplots and social network graphs. In addition, the criterion for significant regions in visualization is different from natural images because their color schemes are different. For example, a blue sky in a natural scene is normally classified as background. Therefore, it would often be considered less important than a person in the foreground. Unlike natural images, a blue region rendered by a visualization system may be regarded as an important region.

Information visualization often consists of multiple information layers. For example, a geographical application would normally contain several layers such as water, continents, and various location markers. If we ignore major regions such as continents in resizing, then the results will suffer from distortions. Multi-layered based resizing is rarely discussed in the previous work either on image resizing or on information visualization resizing. The resizing framework of Wu et al. [3] assumed the information visualization layer is single such as scatter-plot, network, and word cloud. However, often there are many abstract layers in information visualization designs such as a scatter-plot on a map and graph with group shapes. Therefore, it is necessary to revisit the multi-layer approaches to detect and preserve the different layers in information visualization. When the resizing content is complex and the canvas become larger, the time performance is becoming more important. Prior work such as Wu et al. [3] requires adjustment to fast resize the information visualization.

Hence, based on the resizing pipeline of Wu et al. [3], we present a different visualization resizing approach in three aspects. First, we define a visualization-related saliency map. Second, we consider the classes of information to be segregated into multi-layers for visualization. Third, the controlling mesh for resizing in our approach is adaptive so that users can emphasize the content of the visualization with fewer distortions in a shorter period of time. The contributions of our work are:

• an abstract multi-layer model for the resizing problem of information visualization. Our model can be used to resize the output from a visualization system to automatically match the native aspect ratio of any external target display;

• a set of criteria called the visual saliency map (or VSM) to describe the features of information visualizations in different saliency layers;

• a triangle mesh-based energy optimization method to achieve better visual distribution of information features after resizing. We present the results of our experiments on different genres of multi-layered visualizations to demonstrate the performance of our approach.