The consensus problem of linear multi-agent systems with directed switching topologies is investigated. An observer-based consensus protocol is designed, under which the final consensus value may be nontrivial if the agent is stabilizable and observable. An algorithm is presented to construct the consensus protocol to achieve the joint design of the final consensus value and the prescribed convergence rate for arbitrarily switching topologies having a spanning tree. A numerical example indicates the effectiveness and advantages of the algorithm.