Most spatial modulation (SM) schemes in the literature use hard-output maximumlikelihood (ML) antenna index detection. In trellis coded spatial modulation (TCSM) scheme, the detection should be also performed for the transmitted symbol, while the sequence decoding is performed using a hard-output Viterbi algorithm. We modified the hard-output ML-SM detector to determine a soft estimate of the transmit antenna index and a hard estimate of the transmitted symbol, and used it in conjunction with the logarithmic maximum a posteriori probability (log-MAP) sequence decoding algorithm. If at least four receive antennas are used, the soft-output and the less complex hybrid detectors offer a coding gain, over the hard-output detector, of at least 4 dB in Rician fading and 2 dB in spatially correlated (SC) fading channels.

Keywords: Spatial modulation, Trellis coded spatial modulation (TCSM), Soft-output maximum-likelihood (ML) detectors.