Device-to-device (D2D) multicast communications have become a promising technology to satisfy the increasing demands for popular content sharing among users. However, most of the existing works mainly focus on the benefits resulting from constituting the clusters, while ignoring the ever-present costs. Facing the coexistence of the benefits and costs, the two issues should be tackled, i.e., i) how to motivate the users to form clusters, and ii) how to achieve a balance between them. In our work, a social-monetary-aware incentive model based on the social ties and the content fee is proposed to motivate both the cluster head (CH) and content requesters (CRs) to form clusters. Then, we formulate the cluster formation problem as an incentive-based coalition formation game with nontransferable utility, where the utility takes into account the benefits in terms of the content fee and the costs in terms of the transmission delay. An incentive-based cluster formation algorithm based on the merge-and-split rules and the Pareto order is proposed. Its performance, e.g., the convergence, the stability, is discussed, and is also evaluated by extensive simulations based on the comparisons with other algorithm.