Recently, Vulnerable Traffic Participants (VTPs) trajectory prediction has got some attention, which can help autonomous vehicles better understand complex traffic environments. This paper systematically investigates VTPs (pedestrians, cyclists, and electric cyclists) trajectory prediction based on a fresh data-driven stacking model. Firstly, a micro-dynamic Modified Social Force Model (MSFM) is proposed, pedestrian-cyclist-electric cyclist-dynamic vehicle interactions, the effect of zebra crossing and pedestrian heterogeneity (age and gender) are taken into account. Then an Attention-Long Short Term Memory Network (AN-LSTM) is developed, AN is used to achieve the influence weights of pedestrian heterogeneity and road users. Finally, a fresh stacking model by combining the MSFM and AN-LSTM is proposed for VTPs trajectory prediction. The stacking model is evaluated with the current state-of-the-art models. The results indicate that the proposed model achieves an increased accuracy of more than 12%. Moreover, a real-time online integrated vision system which combines target detection, multi-target tracking, pedestrian heterogeneity recognition and distance measurement is developed. The proposed model is tested with the integrated vision system, and high accuracy and good real-time performance (0.030 seconds) are achieved, which can give us great confidence to use the proposed model in the autonomous vehicles for improving VTPs safety and transportation efficiency.