In recent years, the issue of predicting metro ridership has gained traction within the intelligent transportation systems community, due to its potential advantages for the metro network system such as improving the service quality and making informed decisions about infrastructure investments. When it comes to metro station-level ridership forecasting, in the literature this is often tackled by using recurrent neural network (RNN)-based approaches. While RNNs have shown promising results in providing station-level metro ridership predictions over short-term prediction horizons, they are still challenged when it comes to long-term prediction horizons. Thus, in this work, we are introducing a novel approach, the Inter-Station-Aware Transformer Networks framework, for efficient and scalable station-level metro ridership forecasting over both short and long-term prediction horizons. Our proposed approach models and fuses both the temporal historical ridership data and the metro network topology using an encoder-decoder framework based on the transformer network architecture. The proposed approach has been evaluated on two publicly available datasets and compared against a number of baseline approaches. We achieved superior results when it comes to longer-prediction horizons when compared with state-of-the-art methods from the literature, while we proved it is also three times more efficient in terms of the number of model parameters required.