With the rise of advanced driver assistance systems and highly automated driving, the research of highly reliable and accurate tracking algorithms received special attention within the last years. While most of the investigation treats the objects to be tracked as single points, the appropriate handling of uncertainties, when also the object's extent is of interest, has been ignored so far. Classical approaches do not capture the properties of the probability density function, which in fact resembles a banana-shaped distribution, when handling the object's state and the measurements in the Euclidean space. In this paper, we tackle this problem, called extended object tracking, by tracking the object's state on the affine group which belongs to the matrix Lie groups. By incorporating the second-order dynamics of the movement, we model the state space as the direct product Aff(2) ×\mathbbR³ and handle the uncertainties on the associated Lie algebra to establish a proper representation. We derive the necessary equations to use the extended Kalman filter on Lie groups including an appropriately modeled measurement space. We show the advantage of our proposed method by comparing it to a classical approach, which treats both the state and the measurement space in the Euclidean space. The results of the simulations prove that, by encapsulating the state and measurement space in the appropriate manifold, we obtain higher stability and realibility with smaller state estimation errors.