Driven by the surging mobile data traffic demands, the cloud radio access network (Cloud-RAN) of small cells has attracted intense interest in both academia and industry. In practice, the capacity-limited backhaul may eventually restrict the performance potential of cooperative multipoint transmission in Cloud-RAN. In this paper, we consider the delay-aware cooperative beamforming control for delay-sensitive traffic with limited backhaul data transfer in Cloud-RAN of small cells. Followed by the proposed simple threshold-based MS-centric cooperative clustering scheme, we formulate the delay-aware cooperative beamforming control problem as an infinite horizon average cost Markov decision process (MDP) and introduce the discrete time Bellman equation which embraces both the queue state information (QSI) and channel state information (CSI). To tackle with the emerging slow convergence of value functions and curse of dimensionality, the continuous time Bellman equation is derived and the suboptimal solution is obtained using the linear approximation of value functions. Since the formulated problem is non-convex, we derive the conservative reformulation of it and further transform it into convex problem using semidefinite relaxation (SDR) technique. Finally, an alternating optimization algorithm is proposed to effectively solve the SDR problem. Simulation results validate the performance gain of the proposed delay-aware cooperative beamforming control with limited backhaul consumption.