In this paper, the leader-following consensus problem of multi-agent systems with general linear dynamics and arbitrary switching topologies is considered. The multi-agent system with arbitrary switching topologies is formulated as a switched system. Then the leader-following consensus problem is transformed to a stability problem of the switched system. A necessary and sufficient consensus condition is derived. And this condition is also extended to the multi-agent system with time-varying delays. Numerical examples are provided to illustrate the effectiveness of the proposed results.