The detection of tactical maneuvers performed by other traffic participants is a key component for modern autonomous driving systems. We present a novel vision-based approach to detect tactical maneuvers for vehicles, pedestrians, and bicycles. Our approach uses neural networks with temporal convolutions to incorporate temporal information into the detection. The training and evaluation of the presented architecture is performed on a video dataset obtained in a city and industrial environment spanning 3.5 hours. Our approach detects nine distinct maneuver classes with an average accuracy of 54.21%, with single detection accuracies of up to 88.17%.