The biological hypothesis of spinal engine states that locomotion is mainly achieved by the spine, while the legs may serve as assistance. Inspired by this hypothesis, a compliant, multiple degree-of-freedom, biologically-inspired spine has been embedded into a quadruped robot, named Kitty, which has no actuation on the legs. In this paper, we demonstrate how versatile behaviors (bounding, trotting, and turning) can be generated exclusively by the spine's movements through dynamical interaction between the controller, the body, and the environment, known as embodiment. Moreover, we introduce information theoretic approach to quantitatively study the spine internal dynamics and its effect on the bounding gait based on three spinal morphologies. These three morphologies differ in the position of virtual spinal joint where the spine is easier to get bent. The experimental results reveal that locomotion can be enhanced by using the spine featuring a rear virtual spinal joint, which offers more freedom for the rear legs to move forward. In addition, the information theoretic analysis shows that, according to the morphological differences of the spine, the information structure changes. The relationship between the observed behavior of the robot and the corresponding information structure is discussed in detail.