The peer-review process is the most widely accepted standard for validating products of researchers within the scientific community. It is also adopted by funding agencies. An essential component of peer-review is to find a certain number of experts to review a research paper or a grant proposal. Previous work mainly focuses on finding experts with the necessary expertise relevant to the paper or proposal while ignoring the diversity in the selected reviewers, which potentially leads to the conflict of interest (COI). In this paper, we propose a novel and unified framework that takes three major key factors into account for reviewer assignment: importance, diversity and expertise coverage of a group of reviewers. Our framework selects a panel of reviewers that not only cover all topics of a submission but also reduce various potential COIs. The proposed framework effectively integrates probabilistic topic model and activation spread model in the presence of a social network of researchers. To the best of our knowledge, this is the first work to study the diversity of reviewers and leverage its effect in the reviewer assignment. We conduct extensive experiments to evaluate the performance of our proposed framework for reviewer assignment. The experimental results show that our approach is very effective in finding panels of relevant, authoritative and diverse reviewers for given submissions to review.