A design-of-experiment based statistical technique for detection of key-frames

Abstract

In this paper decision variables for the key-frame detection problem in a video are evaluated using statistical tools derived from the theory of design of experiments. The pixel-by-pixel intensity difference of consecutive video frames is used as the factor or decision variable for designing an experiment for key-frame detection. The determination of a key-frame is correlated with the different values of the factor. A novel concept of meaningfulness of a video key-frame is also introduced to select the representative key-frame from a set of possible key-frames. The use of the concepts of design of experiments and the meaningfulness property to summarize a video is tested using a number of videos taken from MUSCLE-VCD-2007 dataset. The performance of the proposed approach in detecting key-frames is found to be superior in comparison to the competing approaches like PME based method (Liu et al., IEEE Trans Circuits Syst Video Technol 13(10):1006–1013, 2003; Mukherjee et al., IEEE Trans Circuits Syst Video Technol 17(5):612–620, 2007; Panagiotakis et al., IEEE Trans Circuits Syst Video Technol 19(3):447–451, 2009).

Appendices

Appendix A: Obtaining (13) from [7]

Hoeffding’s inequality [ (7)] In our problem, m _i is the number of frames in the ith unit. Then we can formulate the problem by a sequence of i.i.d. random variables $\{X_q\}_{q=1,2,3,...,m_i}$, such that 0 ≤ X _q  ≤ 1. Let us define X _q as,

$$ X_q =\left\{ \begin{array}{ll} 1 & \mathrm{when}~~ l_q < \eta\\ 0 & \mathrm{otherwise}, \end{array}\right. $$

(19)

for a given η, where l _q is the l-ratio value of the qth frame of the ith unit. We set $S_{m_i}=\sum_{q=1}^{m_i}X_q$ (i.e., the number of frames of ith unit having l-ratio greater than η) and $\nu m_i=E\left[S_{m_i}\right]$. Then for νm _i  < t < m _i (since ν is a probability value less than 1), putting $\sigma=\frac{t}{m_i}$ as in [5], according to Hoeffding’s inequality,

$$P_i^{\eta}=P\left(S_{m_i}\geq t\right)\leq e^{-m_i\left(\sigma \log {\frac{\sigma}{\nu}}+(1-\sigma) \log {\frac{1-\sigma}{1-\nu}}\right)}. $$

(20)

In addition, the right hand term of this inequality satisfies,

$$ e^{-m_i\left(\sigma \log {\frac{\sigma}{\nu}}+(1-\sigma) \log {\frac{1-\sigma}{1-\nu}}\right)}\leq e^{-m_i\left(\sigma-\nu\right)^2H(\nu)}\leq e^{-2m_i\left(\sigma-\nu\right)^2}, $$

(21)

where

$$ H(\nu) =\left\{ \begin{array}{ll} \dfrac{1}{1-2\nu}\log {\dfrac{1-\nu}{\nu}} & \mathrm{when}~~ 0<\nu<\dfrac{1}{2}\\\\ \dfrac{1}{2\nu(1-\nu)} & \mathrm{when}~~ \dfrac{1}{2}\leq \nu<1 \end{array}\right. $$

(22)

This is Hoeffding’s inequality. We then apply this for finding the sufficient condition of ϵ-meaningfulness. If $t\geq \nu m_i+\sqrt{\frac{\log {\lambda} - \log {\epsilon}}{H(\nu)}}\sqrt{m_i}$, then using (20) and (21) and putting $\sigma=\frac{t}{m_i}$ we get

$$ m_i\left(\sigma-\nu\right)^2 \geq \frac{\log {\lambda}-\log {\epsilon}}{H(\nu)}. $$

(23)

Then using (20) and (23) we get,

$$ P_i^{\eta}\leq e^{-m_i(\sigma-\nu)^2H(\nu)}\leq e^{-\log {\lambda}+\log{\epsilon}}=\frac{\epsilon}{\lambda}. $$

(24)

This means by definition of meaningfulness, the cut-off η is meaningful (according to (11)).

Since for ν in (0,1), H(ν) ≥ 2 (according to (22)) so from (24) we get the sufficient condition of meaningfulness as (13).

Appendix B: Algorithm of the proposed approach

• (1) Input the video sequence with speed X fps.

• (2) Find the Euclidean distance of color values of each pair of consecutive frames.

• (3) For γ = 1 to R (R is the maximum bound of color values) do

  • (a) Give a binary value to each pixel using (2).

  • (b) Find the matrix β _d for each frame d.

  • (c) Calculate p-ratio p _d using (3).

  • (d) For κ = 0 to 1 step δ _κ do

    • (i) Find all the frames with p _d  > κ.

    • (ii) If (any selected frame f _i is less than X frame apart from f _i + 1) do

      • (A) Find the temporal distance between f _i + 1 and f _i + 2, f _i + 2 and f _i + 3, and so on until the temporal distance between f _i + m and f _{i + m + 1} is greater than X.

      • (B) Take the f _i + m frame as the boundary of the group starting at frame f _i − 1.

    • (iii) End if

    • (iv) Find F-ratio using (4)–(6).

  • (e) End for κ

• (4) End for γ

• (5) Find F _max  =  max (F-ratio) and corresponding value of γ and κ.

• (6) If F _max  < F _critical

  • (a) Consider the set of frames having p-ratio greater than κ as unit boundaries. else

  • (b) Consider whole video as a single unit.

• (7) End if

• (8) Find l-ratio of each frame by (8).

• (9) For each unit i do

  • (a) For $\eta=0 ~ \mathrm{to} ~ 1 ~ \mathrm{step} ~ \frac{1}{\lambda}$ do

    • (i) Find probability ν using (9).

    • (ii) Find probability $P_{i}^{\eta}$ using (10).

    • (iii) Find NFA using (11).

    • (iv) If $\emph{NFA}<(\epsilon = 1)$,

      • (A) η ^′ = η.

      • (B) Exit from the for loop of η. else

      • (C) Continue the for loop of η.

    • (v) End if

  • (b) End for η

  • (c) For $\xi=\eta^{\prime} ~ \mathrm{to} ~ 1 ~ \mathrm{step} ~ \frac{1}{\lambda}$ do

    • (i) Find $r_{i}(\eta^{\prime})$ using (14).

    • (ii) Find c-value using (15).

  • (d) End for ξ

  • (e) Find the ξ satisfying (16).

  • (f) Select the frames having l-ratio greater than ξ as key-frames.

• (10) End for unit

• (11) Display all the selected frames as key-frames.