Hybrid precoding based on the sub-connected structure is one of promising technologies for millimeter-wave communications. Whereas, the existing hybrid designs of the sub-connected structure are all based on the linear precoding, which may suffer from a noise enhancement and performance deterioration. To tackle this issue, this letter proposes a novel nonlinear hybrid Tomlinson-Harashima (TH) precoding algorithm. In the proposed algorithm, a column-wise refinement procedure is firstly developed to update the radio frequency (RF) precoding and combining matrices column by column via Schur complement. Such a refinement procedure can effectively obtain the optimal solutions of each columns of the RF precoding and combining matrices, however, it requires matrix inversions with high complexity. Thus a recurrence relation of the matrix inversion is further established, which can reduce the complexity of the aforementioned refinement procedure greatly. Finally, the digital precoding and combining matrices are given by QR decomposition. Simulations show that the proposed algorithm can effectively improve the spectral efficiency with lower complexity.