Recently, the optimal transceiver design in multi-user multiple-input-multiple-output (MIMO) relay systems has attracted great attention. Unlike most existing works, we take the channel correlation and estimation errors into account, and jointly optimize the relay and receiver matrix in multiuser MIMO relay systems. First, the mean-square error (MSE) expression at receiver is derived, and a minimum mean-square error (MMSE)-based optimization problem is formulated. Then, a Bi-step iterative algorithm is developed to search for the optimal numerical solution. Besides, in order to solve the original problem more efficiently, we further transform it to a sub-optimal problem that minimizes the upper bound of the MSE. Then, we propose a structurized algorithm, where the closed-form expressions of the sub-optimal relay matrix and receiver matrix are given. Finally, simulation results are presented to illustrate the effectiveness of the proposed algorithms.