Buffer-aided relay networks provide more reliability and coverage in future wireless communications. Therefore, this paper investigates a buffer-aided cooperative relaying system with $K$ relays and a direct link from the source to the destination, providing a general scenario different from other existing state-of-the-art techniques. In particular, we propose a novel link selection scheme, which adaptively coordinates the selection priorities of the direct and cooperative relay link according to the instantaneous buffer state. The performance of the proposed link selection scheme is analyzed, in terms of outage probability, average packet delay (APD) and diversity order by providing closed-form expressions. For asymptotic analysis, a theoretical framework is presented by dividing all buffer states into different sets, which verifies that the minimum buffer size is just two for achieving the full diversity order of $2K+1$ . We also provide the relationship between the asymptotic APD and diversity order by adjusting predefined target queue lengths, which shows that the diversity order ranges from $K+1$ to $2K+1$ as the asymptotic APD ranges from 0 to $K$ time slots per packet. Both theoretical and simulation results demonstrate that direct transmission significantly improves the outage and delay performance simultaneously.