This paper describes an analytical study of performance of a proposed signal detection scheme that will allow coexistence of an additional radio communication system (generally, secondary system) in the service area where the existing communication system (primary system) is operated. Its performance characteristics are derived by an analytical method based on stochastic theory, which is subsequently validated by software simulation. The main purpose of the detection scheme is to protect the primary system from the secondary system. In such a situation, the signals of the primary system and secondary system may be simultaneously received in the signal detector. One application of such a scheme is D-to-D (Device-to-Device) communication, whose system concept including the detection scheme is briefly introduced. For improved secondary signal detection, we propose the signal cancellation method of the primary system and the feature detection method of the secondary system signal. We evaluate the performance characteristics of the detection scheme in terms of “probability of correct detection”. We reveal that an undesired random component is produced in the feature detection procedure when two different signals are simultaneously received, which degrades the detection performance. Such undesired component is included in the analytical equations. We also clarify that the cancellation scheme improves the performance, when the power ratio of the primary signal to secondary signal is higher than 20-22dB.