Smoothness in frame reduction

Abstract

A general framework for smoothing a finite sequential set of two-dimensional frames is proposed in this paper. It is intuitive that the more the number of frames is, the smoother the frames will be. However, the storage space required for handling the frames will also be increased. Given the desired size of the frames, finding an optimal frame set is facilitated by the notion of smoothness. The smoothness of a frame set is measured in terms of the energy embedded in the frame set, on the analogy of measuring the smoothness of motion data. A frame set with lower energy value is considered to be smoother. Then, the problem of finding a frame set of given frame size which gives the smoothest measure is formulated as an optimization problem that seeks to minimize the weighted sum of the frame energy and the sum of the squared distance errors.

Scope and purpose

This paper considers the problem of downsampling a sequence of two-dimensional frames subject to the frame size requirement. A smoothness metric is proposed that measures the presentation quality of selected frames from the energy viewpoint. The downsampling problem is formulated as a combinatorial optimization problem that aims to minimize the weighted sum of the frame energy and the sum of the squared distance errors, and then solved by deploying heuristic techniques.

Introduction

Data smoothing plays a vital role in computer graphics, virtual reality (VR), and computer-aided design (CAD). In image processing, for instance, noise reduction in the image data enhances the quality of data presentation, aiding to visualize and differentiate the image content [1], [2], [3]. In a VR environment, the motion data that is sampled from a moving creature via high resolution sensors requires smoothing due to stationary noises, contributed by these sensors, and non-stationary noises, caused by the system delay [4], [5], [6]. In the context of design, the manipulated data are smoothed in accordance with aesthetic and functional requirements [7], [8], [9], [10], [11].

There are in general two means of measuring the smoothness of data: geometric approach and filtering approach. It is presumed for the geometric approach that if the data is smooth, the rate of the change of the neighboring velocities of the data shall be small. Thus, in order to smooth the data the geometric approach aims to reduce the magnitude of the curvatures of the data, which leads to strain-energy minimization [5], [6], [10], [12]. Yet, as solely minimizing the strain energy would straighten or flatten the data, distance metrics are imposed to prevent the resulting data from being far dissimilar to the original data, hence the preservation of the geometric shape of the data. For the filtering approach, which is commonly used in the fields of signal processing and image processing, it is assumed that the jerkiness of the data is due to the noise embedded in the data. Thus, the filtering approach achieves noise reduction by detecting and eliminating the noisy components in the frequency domain. For example, when noises are modeled as high frequency components, low-pass filtering schemes will be deployed [4], [13], [14]. In this paper, the geometric approach is adopted.

In smoothing geometric data such as position data, orientation data and motion data of a rigid body, ordering among data exists, hence the data representation in vector form. The trajectory of a rigid body in three-dimensional Euclidean space $\text{R}{}^3$, for instance, can be described as a position vector $\text{p}\text{(s)∈}\text{R}{}^3$, where t is in some time interval. In a discrete domain, $\text{p}\text{(s)}$ becomes $\text{P}{}_{\mathrm{n}}\text{=[}\text{p}{}_0\text{⋯}\text{p}{}_{\mathrm{n-1}}\text{]}{}^{\mathrm{T}}\text{,}\text{p}{}_{\mathrm{i}}\text{∈}\text{R}{}^3$, where n is the number of instances sampled from $\text{p}\text{(s)}$. To smooth this position vector, it is hoped that each position data point can be fine-tuned so that the overall appearance of the position vector is smooth and the shape of the position vector is preserved. If the orientation of the rigid body is also taken into account, the motion $\text{m}\text{(s)}$ of the rigid body is established: $\text{m}\text{(s)=[}\text{p}\text{(s)}\text{r}\text{(s)]}$, where $\text{p}\text{(s)}$ is the position vector and $\text{r}\text{(s)}$ is the orientation vector, represented by a four-component vector $\text{q}\text{(s)=[}\text{w}\text{̂}\text{(s)}\text{x}\text{̂}\text{(s)}\text{y}\text{̂}\text{(s)}\text{z}\text{̂}\text{(s)]}$ on a four-dimensional unit sphere [15], [16]. In this paper, a more general data representation is considered in which each motion data point is associated with a multi-dimensional frame and the content of the frame depends upon the position and orientation of the data point in space. As an example, consider fly-through in a VR environment in which what we visualize on screen, or on the head-mounted display (HMD), at some time instant is determined by the position we are at and the direction in which we are looking. Here, the content on screen is termed as a frame, the underlying motion of which consists of the corresponding position and direction. The on-screen images transmitted from a remote robot traveling on a rough terrain is another example. In these two examples, each frame is a two-dimensional image. Although it is possible that the dimensionality of a frame could be greater than two (range data immediately comes in mind [17], [18], [19], [20]), this study focuses mainly on 2D frames.

With a large amount of frames within some time interval (as with high sampling rate), the sequence of frames appears to be under smooth transition.¹ The problem arises, however, in considerations of storage space and user interactivity. The storage space grows as the number of frames increases. Transmitting a large frame set through the limited bandwidth of the media is less feasible. Also, dynamic manipulation of the frame motion from users requires rapid swapping of the frames in real time. When there are a huge number of frames to be handled, lag incurs. With a small number of frames (as with low sampling frequency), smooth transition, nevertheless, is not guaranteed. It is thus motivated in this study that a smoothness measure is proposed to facilitate the finding of a better frame set from the optimality perspective that meets the storage space requirement. On the analogy of measuring the smoothness of motion data, the smoothness of a frame set is measured by the frame energy that is the magnitude of the change ratio of the differences of neighboring frames. Then, given a relatively large number of frames, it is inquired that which subset of given frame size gives the smoothest measure. Such frame smoothing problem is formulated as an optimization problem which aims to minimize the weighted sum of the frame energy and the sum of the squared distance errors.

The frame smoothing problem formulated in this study differs from the frame morphing problem [21] in which each intermediate frame is created from adjacent frames. Such an interpolated frame may not exactly resemble the actual one at the same position and in the same orientation. Frame smoothing, on the other hand, presumes no interpolation and frames to be chosen are a subset of the original frame set. Also, frame smoothing differs from frame compression. Compression aims to remove the redundancy among frames and keep the differences, or all the differences if lossless compression is encountered, of frames, whereas frame smoothing considers the smooth transition of the differences of frames so as to find a smoother frame set. It is plausible, however, to combine frame smoothing and frame compression as a two-step process: (1) obtain a smoothed frame set by frame smoothing, and (2) compress the smoothed frame set to obtain a more compact frame set. The remainder of the paper proceeds as follows. Section 2 gives an introduction to the definitions and notations used in the paper, followed by the presentation of the smoothing formulation in Section 3. Two algorithms, based on the local search method and genetic algorithms, are developed. In Section 4, an example of smoothing a cyclic frame set is illustrated. Finally, Section 5 concludes with some remarks.