Video thumbnail extraction for HEVC

https://doi.org/10.1016/j.image.2017.08.007Get rights and content

Abstract

This paper proposes a thumbnail quality enhancement algorithm that uses a predefined weight table. Conventional thumbnail extraction algorithms in high efficiency video coding use a simple downsampling method to produce thumbnail images with low complexity, resulting in thumbnail quality deterioration. The proposed algorithm estimates the original average values of each thumbnail pixel using a weighted average value of several pixels, based on intra-mode direction. The proposed method improves the visual quality of thumbnail images while maintaining low complexity.

Introduction

The extraction of thumbnail images from an encoded video bitstream requires a highly complex computational process, including full decoding followed by downsampling. To solve this problem, several thumbnail extraction algorithms have been proposed. In MPEG-2 [1], a thumbnail image can be generated directly by extracting the DC value for each macroblock in the DCT domain; this is because the DC value in the DCT domain is the average value in the pixel domain. Intra prediction was added in H.264/AVC to obtain a high compression ratio [2]. The MPEG-2 thumbnail extraction algorithm cannot be used with H.264/AVC because the DC value in the DCT domain is the residual DC value caused by the intra prediction method; this has led to the development of H.264/AVC thumbnail extraction algorithms. Two previously developed algorithms each define a generalized equation for intra prediction and generation of thumbnail images using a lookup table (LUT) in the DCT domain [3], [4]. These methods can be used to quickly extract thumbnail images. However, to apply the previously proposed thumbnail extraction algorithm in H.264/AVC, a total of 264 two-dimensional LUTs are required, and these are generated by combining the block sizes and the number of modes. In addition, the DC value, which is the average value of the block, is extracted directly from the DCT domain, and only one DC value is extracted regardless of the block size. If an 8 × 8 DCT is performed, one DC value is extracted and shared by the neighboring 4 × 4 positions (Normally, the thumbnail is 1/16 of the original image size, and four average values are needed in an 8 × 8 block). Recently, the High Efficiency Video Coding (HEVC) standard has provided a very high compression ratio, with higher complexity than H.264/AVC [5]. A thumbnail extraction algorithm with low complexity is required for use with HEVC. However, the H.264/AVC thumbnail extraction method cannot be applied to HEVC because of the new intra prediction mode, various intra mode directions, and the larger block sizes used to achieve its high compression. The problems with the methods described above can cause severe image quality degradation in HEVC with a larger block size. In addition, planar mode – which is the new intra prediction mode – has been added in HEVC, and previous algorithms must be able to define an LUT for this mode. To be defined as an LUT, this LUT requires a very large memory (i.e., 264 two-dimensional LUTs), as described above. Applying this to HEVC requires consideration of additional intra prediction modes and various block sizes; it is therefore necessary to store a total of 1680 two-dimensional LUTs. For these reasons, it is not possible to apply previous methods to HEVC. To generate a thumbnail image from an encoded HEVC bitstream, the HEVC decoder sequentially performs entropy decoding, dequantization, intra prediction, inverse transform, in-loop filtering, and downsampling [5]. In contrast, the proposed method for thumbnail extraction reduces the computational complexity by decoding the 4 × 4 boundaries using partial decoding processes only, skipping the in-loop filters. These partial decoding processes are performed on the transform unit and the prediction unit. The proposed algorithm defines equations for the partial decoding processes. Additionally, to reduce aliasing artifacts, the proposed method for generating thumbnail images includes the use of a predefined weight table.

Section snippets

Partial decoding for transform and prediction units

The N × N inverse discrete cosine transform of HEVC can be described as follows: rN×N=AN×NT×RN×N×AN×N,where N and A denote the block size and the transform matrix, and rN×N and RN×N denote residual blocks in the pixel domain and the transform domain, respectively. To generate a thumbnail image for HEVC video, the inverse discrete cosine transform should be performed using Eq. (1) [5]. Whole residual values in the residual block should be reconstructed to generate a complete intra-frame for the

Downsampling methods for thumbnail generation

To generate a thumbnail image, downsampling needs be performed because a thumbnail requires a reduced image size. In general, a thumbnail can be generated using a subsampling method, which extracts one pixel per 4 × 4 area. However, this approach may cause excessive aliasing [6]. A thumbnail image can also be generated using the average value of all the pixels in the 4 × 4 block; this method can reduce aliasing, but has high computational complexity. A previously reported method generates the

Experimental results

The proposed algorithm was tested using HEVC reference software (HM 13 [8]). All HEVC test sequences except Class C and D were used in the experiments. In general, thumbnails are generated to 1/16 of the size of the original frame. In the case of Class C and Class D, resolution of the sequence is too small as 832 × 480 and 416 × 240, respectively. Because the thumbnail size is too small to recognize, we excluded these sequences from the experiments. All frames were I-frames. Table 2 compares

Conclusions

To reduce the computational complexity of thumbnail generation, a method has been proposed that employs partial decoding and a predefined weight table to enhance the visual quality of the resulting thumbnail image. The proposed algorithm improves the visual quality of a thumbnail image using weight values based on results from the intra prediction mode. The proposed method also effectively removes aliasing artifacts without noticeably increasing the computational complexity of the overall

Acknowledgment

This work was supported by the research fund of Signal Intelligence Research Center supervised by Defense Acquisition Program Administration and Agency for Defense Development of Korea .

References (8)

  • ITU-T and ISO/IEC JTC 1, ITU-T Recommendation H.262 and ISO/IEC 13818-2 MPEG-2: Generic Coding of Moving Pictures and...
  • WiegandT. et al.

    Overview of the H264/AVC video coding standard

    IEEE Trans. Circuits Syst. Video Technol.

    (2003)
  • KimE.-S. et al.

    A fast thumbnail extraction method in H264/AVC video streams

    IEEE Trans. Consumer Electron.

    (2009)
  • KimM. et al.

    Fast thumbnail generation in integer DCT domain for H264/AVC

    IEEE Trans. Consumer Electron.

    (2011)
There are more references available in the full text version of this article.

Cited by (1)

View full text
© 2017 Elsevier B.V. All rights reserved.