This paper focuses on the controllability of multi-agent systems with fixed topology based on agreement protocols. We analyze three models of agents: single integrator, double integrator and high-order integrator. For a group of single-integrator agents, controllability is studied in a unified framework for both networks with leader-following structure and networks with undirected graph. Some new necessary/sufficient conditions for controllability of networks of single-integrator agents are established. For networks of double-integrator agents, we prove that controllability of the networks is equivalent to that of networks of single-integrator agents under the same topology and same prescribed leaders. This result is further extended to the case of networks of high-order-integrator agents. Moreover, two influencing factors of controllability of networks are investigated, that is, the selection of leaders and the link weights of graphs.