Orthonormality constraints, in which parameter sets are constrained to be perpendicular to each other and of unit length, are important for many estimation, detection, and classification tasks. Such constraints are not appropriate in all practical scenarios, however. In this paper, we describe simple adaptive algorithms that adjust a matrix so that its rows are close to orthonormality after adaptation, as specified by user-selectable bounds on pairwise inner products and squared vector lengths. The algorithms have rapid convergence. Applications to independent component analysis and deep learning system training show the benefits of the approach.