In this paper, a non-linear precoding algorithm with low out-of-band (OOB) radiation is proposed for massive multiple-input multiple-output (MIMO) systems. Massive MIMO sets more than one hundred antennas at each base station to achieve higher spectral efficiency and throughput. Full digital massive MIMO may constrain the resolution of digital-to-analog converters (DACs) since each DAC consumes a large amount of power. In massive MIMO systems with low resolution DACs, designing methods of DAC output signals by nonlinear processing are being investigated. The conventional scheme focuses only on a sum rate or errors in the received signals and so triggers large OOB radiation. This paper proposes an optimization criterion that takes OOB radiation power into account. Gibbs sampling is used as an algorithm to find sub-optimal solutions given this criterion. Numerical results obtained through computer simulation show that the proposed criterion reduces mean OOB radiation power by a factor of 10 as compared with the conventional criterion. The proposed criterion also reduces OOB radiation while increasing the average sum rate by optimizing the weight factor for the OOB radiation. As a result, the proposed criterion achieves approximately 1.3 times higher average sum rates than an error-based criterion. On the other hand, as compared with a sum rate based criterion, the throughput on each subcarrier shows less variation which reduces the number of link adaptation options needed although the average sum rate of the proposed criterion is smaller.