Background modeling is challenging due to background dynamism. Most background modeling methods fail in the presence of intensity changes, because the model cannot handle sudden changes. A solution to this problem is to use intensity-robust features. Despite the changes of an edge's shape and position among frames, edges are less sensitive than a pixel's intensity to illumination changes. Furthermore, background models in the presence of moving objects produce ghosts in the detected output, because high quality models require ideal backgrounds. In this paper, we propose a robust statistical edge-segment-based method for background modeling of non-ideal sequences. The proposed method learns the structure of the scene using the edges' behaviors through the use of kernel-density distributions. Moreover, it uses segment features to overcome the shape and position variations of the edges. Hence, the use of segments gives us local information of the scene, and that helps us to predict the objects and background precisely. Furthermore, we accumulate segments to build edge distributions, which allow us to perform unconstrained training and to overcome the ghost effect. In addition, the proposed method uses adaptive thresholding (in the segments) to detect the moving objects. Therefore, this approach increases the accuracy over previous methods, which use fixed thresholds.