This paper presents a wildfire smoke detection method based on a spatiotemporal bag-of-features (BoF) and a random forest classifier. First, candidate blocks are detected using key-frame differences and non-parametric color models to reduce the computation time. Subsequently, spatiotemporal three-dimensional (3D) volumes are built by combining the candidate blocks in the current key-frame and the corresponding blocks in previous frames. A histogram of gradient (HOG) is extracted as a spatial feature, and a histogram of optical flow (HOF) is extracted as a temporal feature based on the fact that the diffusion direction of smoke is upward owing to thermal convection. Using these spatiotemporal features, a codebook and a BoF histogram are generated from training data. For smoke verification, a random forest classifier is built during the training phase by using the BoF histogram. The random forest with BoF histogram can increase the detection accuracy and allow smoke detection to be carried out in near real-time.