Canonical polyadic (CP) tensor decomposition is an important task in many applications. Many times, the true tensor rank is not known, or noise is present, and in such situations, different existing CP decomposition algorithms provide very different results. In this letter, we introduce a notion of sensitivity of CP decomposition and suggest to use it as a side criterion (besides the fitting error) to evaluate different CP decomposition results. Next, we propose a novel variant of a Krylov-Levenberg-Marquardt CP decomposition algorithm which may serve for CP decomposition with a constraint on the sensitivity. In simulations, we decompose order-4 tensors that come from convolutional neural networks. We show that it is useful to combine the CP decomposition algorithms with an error-preserving correction.