In this paper, amplify-and-forward (AF)-based cooperation in a multiple-antenna-based wireless communication system is studied. In particular, we address the problem of hop-by-hop beamforming and combining in a dual-hop AF multiple-input–multiple-output (MIMO) relay network with two antennas at each node over Nakagami- $m$ fading channels for any integer values of $m$. The beamforming and combining vectors are calculated by using a maximum-eigenvalue-based criterion. A moment-generating-function (mgf)-based approach is used for the analysis. We derive a closed-form expression of mgf of the instantaneous received signal-to-noise ratio (SNR) of the considered cooperative scheme. In addition, diversity order analysis of the beamforming-and-combining-based AF MIMO relay system is performed for three cases. It is deduced on the basis of this paper that the diversity order of the hop-by-hop beamforming and combining in an AF MIMO relay system linearly depends upon the minimum of the Nakagami parameters of both hops.