We consider the optimal routing problem in a discrete-time system with a job dispatcher connected to $M$ parallel servers. At every time slot, the job dispatcher sends the incoming jobs to a server for execution, with each server having a queue that stores the jobs. The arrival process of incoming jobs, and the service processes of the servers are stochastic with unknown and possibly heterogeneous rates. Each server $s_{m}$ is associated with an underlying utility $v_{m}$ that is initially unknown. Whenever server $s_{m}$ completes a job, a utility of $v_{m}$ is obtained and a noisy observation of $v_{m}$ is received. The goal is to design a policy that makes routing decisions to maximize the total utility obtained by the end of a finite time horizon $T$ . The performance of policies is measured in terms of regret, which is the additive difference between the expected total utility obtained by the policy and the supremum of the expected total utility over all the policies. The optimal routing problem can be interpreted as a problem of multi-armed bandit with queues where each server is viewed as an arm and the completion of a job is viewed as a pull of an arm. The key distinction between the optimal routing problem and traditional multi-armed bandit problems is in the queueing dynamics at the server, which arises due to the stochastic nature of the arrival and service processes. Our results combine techniques from control of stochastic queueing systems and stochastic multi-armed bandits to provide insights to the design and analysis of policies for the optimal routing problem. We first present analytical bounds that link the regret to the utilization and queue length of servers. Next, we start by assuming that the ordering of the underlying utilities is known and introduce the Priority- $K$ routing policy which makes priority-based routing decisions that send the incoming jobs to the server of the highest underlying utility with queue length no larger ...