In general, hyperspectral images (HSIs) are degraded by a mixture of complicated noise (i.e., mixture of Gaussian and sparse noise), and how to precisely model HSI noise plays a vital role in the task of HSI denoising. The most popular choices for encoding the noise distribution are Gaussian, Laplacian, and the mixture of Gaussians, but they are always incompatible with real-world HSI noise. By investigating histograms of the error map, we first explore that asymmetry is a typical and general feature of HSI noise. Inspired by this discovery, we find that a bandwise asymmetric Laplacian (AL) distribution can be finely used to model this type of noise. Equipped with the low-rank matrix factorization (LRMF) framework, we formulate a novel model by the maximum likelihood estimation (MLE) principle, which can be efficiently solved using the iterative optimization algorithm. Extensive experimental results on synthetic and real datasets demonstrate that the proposed model outperforms other counterparts. It is also found that scale and asymmetry parameters in the AL distribution can well interpret the pattern of real-world HSI noise.