Streaming video analytics focuses on the real-time analysis of streaming video data from multiple resources, such as security cameras, and IoT devices with video capabilities. It involves applications of various techniques to extract valuable information from live video streams. Edge computing and cloud computing facilitate video stream analytics by utilizing computation resources across both ends, enabling both high accuracy and low latency. However, video streaming behaviours are dynamic and constantly evolving across the edge and the cloud. The network conditions, computing resources, and video content can change rapidly, making it crucial to continuously adjust the analytics methods to provide accurate results. Previous works both based on deep neural networks (DNNs) or heuristic algorithms learn a suitable deployment plan for streaming video analytics applications from historical data or synthetic data and therefore are not able to capture the dynamics. Hence, we propose reinforcement learning-based methods that can adapt to ongoing changes in video streaming behaviours. To ensure the scalability of video analytics in distributed environments, we implement OsmoticGate2, a distributed streaming video analytics system that features optimized processing pipelines and multi-agent RL-based controllers for fast adapting the system configurations across the edge and the cloud. Experiments on a real testbed show that our method outperforms baselines, assuring real-time video analysis and high accuracy in dynamic and distributed environments.