Device-to-device (D2D) communications with non- orthogonal multiple access (NOMA) have been proved in enhancing the network capacity and reducing the burden of the traditional cellular network. Due to reusing the same resources, the underlying D2D communication may cause high interference to cellular users, and thus reducing the network throughput. To alleviate this issue, we consider moving D2D users to the unlicensed band. In this case, D2D users can reuse the resources in cellular networks or share the unlicensed band with WiFi users. However, a critical question is how to effectively perform NOMA user clustering and determine D2D pairs that should use unlicensed band and how to guarantee the quality service of users at both networks. Our objective is to maximize the network throughput by investigating a joint resource allocation problem of user clustering, power control, and D2D mode selection. Since the formulated problem is a mixed-integer nonlinear programming problem, finding an optimal solution is a great challenge. To solve this problem, we propose to use a swarm intelligence approach, namely whale optimization algorithm in solving our optimization problem with an efficient solution. Through simulation results, we show that our proposed scheme attains a better performance than other alternative schemes.