This paper presents a new approach to the general-purpose self-improving controller based on Q-learning control strategies. The previous approach was based on a three-dimensional Q-matrix, significantly slowing down the learning process and limiting the ability of practical implementation in industrial practice. The proposed new algorithm solves this problem by reducing the size of the matrix and the number of tuning parameters of the method without sacrificing its accuracy and learning capabilities. The algorithm is validated by simulation on the two second-order dynamics models and the results show significant improvement compared to the previous version of the developed method.