This article designs a visual surveillance framework for human fall detection. In order to solve the conventional issues in fall detection, such as unsatisfactory feature generalization, low recall rates, and large computational time, we design a model that incorporates the deep convolutional neural network and the aggregated heuristic visual features in detecting the occurrence of falls. First, the convolutional neural network (Openpose model) is utilized to extract human skeleton in the image. Second, the hand-crafted spatial features, such as the angle of human shank inclination, are aggregated to determine the fall presence. It should be noticed that our fall detection method has been integrated to healthcare Internet of Things (IoT) video surveillance architecture, which has multiple graphic processing unit groups to perform real-time monitoring and alarming for the elderly in need. The experimental results prove that our method is able to accurately distinguish fall and nonfall activities with a competitive false-alarm rate.