In this paper, we consider the pilot design based on the mutual incoherence property (MIP) for sparse channel estimation in orthogonal frequency-division multiplexing (OFDM) systems. With respect to the length of channel impulse response (CIR), we first derive a sufficient condition for the optimal pilot pattern generated from the cyclic different set (CDS). Since the CDS does not exist for most practical OFDM systems, we propose three pilot design schemes to obtain a near-optimal pilot pattern. The first two schemes, including stochastic sequential search (SSS) and stochastic parallel search (SPS), are based on the stochastic search. The third scheme called iterative group shrinkage (IGS) employs a tree-based searching structure and removes rows in a group instead of removing a single row at each step. We later extend our work to multiple-input–multiple-output (MIMO) systems and propose two schemes, i.e., sequential design scheme and joint design scheme. We also combine them to design the multiple orthogonal pilot patterns, i.e., using the sequential scheme for the first several transmit antennas and using the joint scheme to design the pilot pattern for the remaining transmit antennas. Simulation results show that the proposed SSS, SPS, and IGS converge much faster than the cross-entropy optimization and the exhaustive search and are thus more efficient. Moreover, SSS and SPS outperform IGS in terms of channel estimation performance.