The opportunistic spectrum access in cognitive radio networks has always played a key role in improving the performance of secondary users. In this paper, we propose an optimal spectrum access strategy for multi-channel access of secondary users under multiple physical considerations. Firstly we integrate path loss and Nakagami-m fading of primary and secondary links with imperfect spectrum sensing into our model to obtain the probability that data packet is successfully transmitted. Then we introduce the M/M/1/K queueing to deduce the average packet delay under the assumption of finite-size buffer. In order to minimize the total average packet delay, the optimal probability vector for spectrum access is confirmed by genetic algorithm. Result proves that our proposed spectrum access strategy outperforms the strategies of equal probability and inverse ratio. Furthermore, according to the optimal probability vector, the total average packet loss rate is derived with one more consideration that the number of transmission attempts for automatic repeat request is fixed. Finally, numerical results illustrate the effects of referred lower-layer parameters on total average packet delay and loss rate.