Crowdsourced live streaming services (CLS) present significant challenges due to massive data size and dynamic user behavior. Service providers must accommodate personalized QoE requests, while managing computational burdens on edge servers. Existing CLS approaches use a single edge server for both transcoding and user service, potentially overwhelming the selected node with high computational demands. In response to these challenges, we propose the Reinforcement Learning-based-Collaborative Edge-Assisted Live Streaming (RL-CEALS) framework. This innovative approach fosters collaboration between edge servers, maintaining QoE demands and distributing computational burden cost-effectively. By sharing tasks across multiple edge servers, RL-CEALS makes smart decisions, efficiently scheduling serving and transcoding of CLS. The design aims to minimize the streaming delay, the bitrate mismatch, and the computational and bandwidth costs. Simulation results reveal substantial improvements in the performance of RL-CEALS compared to recent works and baselines, paving the way for a lower cost and higher quality of live streaming experience.