Efficient detection schemes for an overloaded multiple-input multiple-output (MIMO) system have been investigated recently. The literature shows that trellis coded modulation (TCM) is able to enhance a system's capability to separate signal streams in the detection process of MIMO systems. However, the computational complexity remains high as a maximum likelihood detection (MLD) algorithm is used in the scheme. Thus, a sphere decoding (SD) algorithm with a pseudo distance (PD) is proposed in this paper. The PD maintains the coding gain advantage of the TCM by keeping some potential paths connected unlike conventional SD which truncates them. It is shown that the proposed scheme can reduce the number of distance calculations by about 98% for the transmission of 3 signal streams. In addition, the proposed scheme improves the performance by about 2dB at the bit error rate of 10^-2.