Efficient detection schemes for an overloaded multiple-input multiple-output (MIMO) system has been investigated recently. In previous literature, it has been shown that trellis coded modulation (TCM) can enhance system capability to separate signal streams at detection process in MIMO systems. However, the computational complexity remains high as a maximum likelihood detection (MLD) algorithm is used in the scheme. Thus, a sphere decoding (SD) algorithm with a pseudo distance (PD) is proposed in this paper. It is shown that the proposed scheme can reduce the number of distance calculations by about 98% for the transmission of 3 signal streams. In addition, the proposed scheme improves the bit error performance by about 2dB at the bit error rate of 10^-2.