This letter presents a theoretical analysis of the security of two over-complete-dictionary-based improved spread spectrum (ISS) watermarking systems, i.e., subspace-ISS (sub-ISS) and random matching pursuit (RMP)-ISS, in comparison with the conventional orthogonal-transform-based ISS system. Considering Kerckhoffs' principle and the watermarked-only attack (WOA) scenario, we derive the associated decoding regions under brute force attack in which the adversary randomly draws probing keys until getting access to the watermarking channel. We first reveal that the sub-ISS system is as equally secure as the conventional ISS system in terms of message decoding, but the sensed key could provide computational security of the spreading key. More importantly, the RMP-ISS system is shown to be securer than the conventional ISS system for it being able to survive brute force attack. Based on the findings, we further propose an empirical design mechanism that optimizes the security of the RMP-ISS scheme.