Device-to-device (D2D) communication is an emerging technique to enhance the throughout and reduce energy consumption of future cellular networks that are expected to serve a tremendous amount of wireless devices. With the D2D communication functionality, the data link between two users in the proximity can be established directly without routing through the base station (BS). The optimal mode selection policy is to choose the communication route (i.e., either direct or through the BS) that offers the higher throughput, but its implementation, which requires network wide information about the interference level of every user, is far from trivial. We propose a simple yet effective mode selection scheme that uses an upper bound in replacement of the exact interference level to choose the communication mode. We prove in which conditions that the proposed scheme performs as good as the optimal mode selection scheme. Simulations are performed to evaluate the performance of the proposed scheme. The proposed scheme can achieve the comparable performance to the optimal scheme in terms of sum throughput but with much lower implementation complexity.