Baggage screening for identifying concealed threat items has become inevitable for maintaining public security at high-risk locations, including airports and border checkpoints. However, manual screening needs both expertise and experience, in addition to being cumbersome and prone to errors, encouraging researchers to invest in developing autonomous baggage screening systems. However, these approaches based on CNNs prioritize localized interactions due to their solid inductive bias, restricting their ability to model object-level and image-wide context. Hence, in this paper, we explore Transformers for baggage threat recognition to exploit their ability to model global features to capture concealed threat items within cluttered and tightly packed baggage scans and thereby learn enhanced representations to identify the abnormal scans. Further, the property of visual transformers to prioritize shape over textural information render them a suitable candidate for threat recognition from baggage scans since they lack texture and have low contrast. We also explore the potential of visual transformers in heavily imbalanced settings. Further, we have also implemented a weakly supervised localization approach to identify the input regions contributing to the abnormality classification. The proposed approach surpasses the state-of-art methods achieving 0.979 on Compass-XP, and 0.873 on SIXray, in terms of F1 score.