We study the use of reinforcement learning to model Dynamic Spectrum Access in a realistic multi-channel environment. Three different approaches from the literature on the multi-armed bandit problem are compared on a set of realistic channel access models - two are based on stochastic models of the channel occupancy, while a third assumes an adversarial model. The algorithms are experimentally tested on channels occupied by primary users that behave according to a simple fair scheduler and a semi-Markov model based on WLAN traffic measurements; models that generate more realistic channel occupancy patterns than allowed by fixed i.i.d. probability models. The experiments show that the UCB1 algorithm of Auer et. al. [1] outperforms the other algorithms, and we support these findings using some simple theoretical results.