To exploit the potential of content caching and device-to-device (D2D) communication, we propose a user-centric joint device caching and channel assignment (DCA) policy to facilitate content exchanges between user equipments (UEs). The objective is to minimize the average content delivery delay by effectively leveraging D2D communications using as few channels as possible, subject to the UEs' cache capacities and availability of D2D links. This joint design problem is formulated as a nonlinear combinatorial optimization problem which is NP-hard. We first analyze the optimal DCA policy in two special cases. Then, a low-complexity heuristic algorithm is proposed for general cases which alternatively performs greedy device caching and graphcoloring based channel allocating. Simulation results show that the proposed DCA policy can reduce the average content delivery delay by more than half, in contrast to baseline schemes with locally popular caching.