In this paper, we propose admission control and power optimization algorithms for Device-to-Device (D2D) communication which share uplink resources with cellular system in underlay mode. In the presence of quality of service (QoS) protection (both for D2D and cellular users) and transmit power limit, we propose a set-based admission control (SAC) algorithm in which some novel strategies of sets adjustment are presented to handle the violations of constraints. By using SAC, we get a set which contains admitted D2D links (DLs) as many as possible, and other DLs are rejected, but the D2D system capacity is not optimal. Then, we propose a distributed power optimization (DPO) algorithm to maximize the D2D system capacity after implementation of SAC. In DPO, QoS protection is also provided, each DL uses gradient descent method to update its transmit power, and DLs only need to exchange “price” messages which indicate the effect of interference. Through the numerical results, we demonstrate the feasibility and satisfying performance of proposed algorithms compared to traditional algorithms.