Iterative SpMV (ISpMV) is a key operation in many graph-based data mining algorithms and machine learning algorithms. Along with the development of big data, the matrices can be so large, perhaps billion-scale, that the SpMV can not be implemented in a single computer. Therefore, it is a challenging issue to implement and optimize SpMV for large-scale data sets. In this paper, we used an in-memory heterogeneous CPU-GPU cluster computing platforms (IMHCPs) to efficiently solve billion-scale SpMV problem. A dedicated and efficient hierarchy partitioning strategy for sparse matrices and the vector is proposed. The partitioning strategy contains partitioning sparse matrices among workers in the cluster and among GPUs in one worker. More, the performance of the IMHCPs-based SpMV is evaluated from the aspects of computation efficiency and scalability.