Budget allocation for multi-channel in advertising deals with distributing different sub-budgets to different channels under a fixed budget periodically. However, the issue of sequential decision making, with the goal of maximizing total benefits accrued over a period of time instead of immediate benefits, has rarely been addressed. Besides, there is a lack of explicit linking between the advertising actions taken in one channel and the responses obtained in another. What's more, the budget constraint restricts the feasible space of various optimal strategies. In this paper, we resolved these challenges by invoking a novel integrated algorithm based on both the Reinforcement Learning (RL) and Multi-Choice Knapsack Problem (MCKP), termed as Q-MCKP. Besides, we proposed some improvements such as a discretization method of discretizing the costs so as to decrease the complexity of the model. Moreover, the reward function of Q-learning was rebuilt by concerning an additional impact factor among channels. We conducted experiments using approximately two years of daily practical advertising data. Comparing to the state-of-arts, our experimental results demonstrated more effective in two angles.