Urban greening trees play a significant role in optimizing the environment, and the segmentation of branches can effectively help researchers assess the growth status of trees. This paper proposed an innovative RGB image-based model to segment tree branches within urban landscapes. The proposed model employed a dual-encoder architecture, including a basic encoder and an edge encoder. These components were designed with specialized blocks adept at extracting critical semantic features and edge information, enhancing the model's ability to segment fine branches. A graph reasoning decoder block with attention-based feature fusion was proposed to capture the semantic associations between regions incorporating edge information. Moreover, the elastic interaction-based loss function, a groundbreaking loss function, was introduced to ensure that the segmentation of the fine branches should be achieved smoothly and consistently. Upon evaluation against a public urban street tree dataset, the precision, recall, IoU, and accuracy of tree branch segmentation are 94.39%, 93.16%, 88.27%, and 98.78%, respectively, achieving the best result among all the tested models. This performance demonstrates the impact of deep learning on enhancing urban greening and sustainable development in effective tree management.