Motion synthesis through 1D affine matching

Abstract

We present the study of a data-driven motion synthesis approach based on a 1D affine image-matching equation. We start by deriving the relevant properties of the exact matching operator, such as the existence of a singular point. Next, we approximate such operator by the Green’s function of a second-order differential equation, finding that it leads to a more compelling motion impression, due to the incorporation of blur. We then proceed to show that, by judicious choice of the matching parameters, the 1D affine Green’s filter allows the simulation of a broad class of effects, such as zoom-in and zoom-out, and of complex nonrigid motions such as that of a pulsating heart.

Appendix: Numerical validation of the experiments

As a means of validating the experiments in Section 4, we have used a software for motion estimation—kindly provided to us by Professor Michael Black—which is based on affine regression [34]. In it, the 2D affine model is expressed as

$$ \left[\begin{array}{c}{\tilde{U}}\\{\tilde{V}}\end{array}\right] = \left[\begin{array}{c}{\tilde{u_0}}\\ -{\tilde{v_0}}\end{array}\right] + \left[\begin{array}{cc} {\tilde{u_1}}&{\tilde{u_2}}\\-{\tilde{v_1}}& {\tilde{v_2}}\end{array}\right] \cdot \left[\begin{array}{c}{x - c_x}\\{y - c_y}\end{array}\right], $$

(23)

where (c _x , c _y )^T denotes the coordinates of the central image point. Comparing the above with Eq. (16), we find the relations

$$ \left\{\begin{array}{l} u_0 = \tilde {u_0} - \tilde{u_1} c_x - \tilde{u_2} c_y\equiv {u^{*}_0} \\ u_1 = \tilde{u_1} \\ u_2 = \tilde{u_2} \\ v_0 = -\tilde{v_0} + \tilde{v_1} c_x - \tilde{v_2} c_y\equiv {v^{*}_0} \\ v_1 = -\tilde{v_1} \\ v_2 = \tilde{v_2} \end{array}\right. $$

(24)

Taking the above into account, we have thus employed Michael Black’s program to estimate the affine motion components, in order to compare them with the input parameters of the Green’s filter. In each considered sequence, the input image and a synthesized one have been used for this purpose. It should be noted that, since we have restricted ourselves here to a separable 2D affine model, it is expected that we should find $\tilde{u_2}\approx\tilde{v_1}\approx 0,$ in all the experiments. Below, we present the validation results only for a subset of the more complex simulated sequences, namely those illustrated in Figs. 12–16.

Zoom-out. Table 1 presents the optical flow parameters yielded by [34], along with those used as input to the Green’s filter. The third frame in Fig. 13 has been used.

Table 1 Zoom-out (Fig. 13)

We see that a very good correspondence is obtained in this case, for all the parameters.

Zoom-in. Again, the estimated and input parameters are very consistent, as shown by Table 2. The second frame in Fig. 14 has been used.

Table 2 Zoom-in (Fig. 14)

Funny eye. The second frame in Fig. 16 has been used. Table 3 shows the estimated and input parameters. Again, the correspondence is fairly good.

Table 3 Funny eye (Fig. 16)

Next, we discuss two examples where the validation through Michael Black’s program has not been possible, those of the pulsating heart and the deforming ball simulations.

Pulsating heart. Table 4 shows the input parameters and those estimated from the third frame of Fig. 12. The data are inconsistent.

Table 4 Pulsating heart (Fig. 12)

A similar situation occurs with the deforming ball, as shown below:

Deforming ball. Table 5 shows the input parameters and those estimated from the second frame of Fig. 15. Again, the data are not consistent, although the errors are somewhat smaller than in the pulsating heart experiment.

Table 5 Deforming ball (Fig. 15)

We conjecture that the problem, in the above simulations, may arise from the fact that, in both cases, the input images consist of the superposition of a central object over a dark background, what could somehow induce errors in the estimation process. In order to check such hypothesis, we performed an additional test based on an image where such a clear-cut figure/background segmentation is not present. For this purpose, we chose the input image to the zoom experiments, applying over it a Green’s filter with parameters (u ₀, u ₁, x _U ) =  (2, −0.031, 64), in order to simulate only horizontal motion, as in the deforming ball and pulsating heart examples. The generated pair appears in Fig. 17.

Fig. 17

Additional test: a Original image. b G-filtered image, for parameters (u ₀, u ₁, x _U ) =  (2, −0.031, 64)

Table 6, below, shows the estimated and input parameters, which, in this case, prove fairly consistent.

Table 6 Additional test (Fig. 17)

From the foregoing discussion, we may conclude that, except in the case of images with the characteristics of Figs. 12 and 15—i.e., with a sharp figure/background separation—the Green’s function simulations can be numerically validated by the motion estimation algorithm of [34].