The problem of detecting a subspace signal is studied in colored Gaussian noise with an unknown covariance matrix. In the subspace model, the target signal belongs to a known subspace, but with unknown coordinates. We first present a new derivation of the Rao test based on the subspace model, and then propose a modified Rao test (MRT) by introducing a tunable parameter. The MRT is more general, which includes the Rao test and the generalized likelihood ratio test as special cases. Moreover, closed-form expressions for the probabilities of false alarm and detection of the MRT are derived, which show that the MRT bears a constant false alarm rate property against the noise covariance matrix. Numerical results demonstrate that the MRT can offer the flexibility of being adjustable in the mismatched case where the target signal deviates from the presumed signal subspace. In particular, the MRT provides better mismatch rejection capacities as the tunable parameter increases.