Currently, most of the human detection methods are based on low-level features. In this paper, we proposed a middle-level feature generation method based on non-negative matrix factorization (NMF) for human detection. We also proposed an improvement scheme to guarantee that a better middle-level feature can be achieved. The proposed scheme can be applied to a complex background and the experimental results are better than those when only the low-level feature is involved.