Based on the Gauss–Markov channel model, we investigate the stochastic feedback control for transmit beamforming in multiple-input–single-output systems and design practical implementation algorithms leveraging techniques in dynamic programming and reinforcement learning. We first validate the Markov decision process formulation of the underlying feedback control problem with a $4R$-variable ( $4R$-V) state, where $R$ is the number of the transmit antennas. Due to the high complexity of finding an optimal feedback policy under the $4R$-V state, we consider a reduced 2-V state. As opposed to a previous study that assumes the feedback problem under such a 2-V state remaining an MDP formulation, our analysis indicates that the underlying problem is no longer an MDP. Nonetheless, the approximation as an MDP is shown to be justifiable and efficient. Based on the quantized 2-V state and the MDP approximation, we propose practical implementation algorithms for feedback control with unknown state transition probabilities. In particular, we provide model-based offline and online learning algorithms, as well as a model-free learning algorithm. We investigate and compare these algorithms through extensive simulations and provide their efficiency analysis. According to these results, the application rule of these algorithms is established under both statistically stable and unstable channels.