In this study, we address the challenge of traversability analysis for autonomous vehicles in diverse environments, leveraging LiDAR sensors. We propose the Transformer-Voxel-Bird’s eye view (BEV) Network (TVBNet), a novel dual-branch framework designed to increase the accuracy and versatility of such analyses in both urban and off-road conditions. TVBNet first preprocesses raw point cloud data through voxelization and the generation of a BEV. It incorporates a Transformer network with a rotational attention mechanism to aggregate features from multiple point cloud frames, capturing long-range correlations both within and between point clouds. Additionally, a Swin Transformer extracts the relative positional relationships in the BEV projection, facilitating a comprehensive understanding of the scene. The fusion of data from both branches via a multisource feature fusion module, which employs a context aggregation mechanism based on a residual structure, allows for robust local to global contextual understanding. This approach not only improves the extraction of correlation features between 2D BEV and 3D voxel data but also demonstrates superior performance on the challenging off-road dataset RELLIS-3D and the urban dataset SemanticKITTI.