Drivers can be either human beings or artificial drivers in future intelligent transportation systems (ITSs). It is important to learn how people drive so that artificial drivers can be programmed to drive consistently with them. This will lead to future ITSs that are safe and efficient. In this article, we propose a new, fully end-to-end decision-making method, namely the pyramid pooling convolutional neural network with long short-time memory (PPC-LSTM), for multitask (longitudinal and lateral) decision inference in future ITSs. In this method, the features were extracted from multiscale red, green, blue images, depth images, and historical driving sequences. Our multitask loss for learning was designed by comprehensively considering the homoscedastic uncertainty of each task. Finally, experimental evaluations were conducted on a data set collected from CAR Learning to Act and the BDD100K naturalistic driving data set to examine the effectiveness of our proposed method. The results show that the prediction accuracies of driving speed and steering angle by our proposed PPC-LSTM are 89.97% and 84.67%, respectively. This is an improvement over state-of-the-art-methods by at least 2.52% and 2.67%, respectively, which demonstrates the method’s promising applications in future ITSs.