The inter-system handover is one of the key features in third generation cellular systems. The compressed mode, with variable transmission gaps and power levels, is standardized to support inter-frequency/inter-system handover. To optimize the system resources while maintaining the border-cell handover quality, a capacity-based compressed mode algorithm is proposed. Considering the tradeoff between the capacity and the connection quality, the algorithm can adaptively manage the compressed mode operation based on the potential impacts on the capacity and the efficiency of the compressed mode measurement. The performance is studied based on UMTS simulation platform