In the context of deep learning models for semantic matching problems, we propose a novel Multi-View Progressive Attention (MV-PA) mechanism general enough to operate on various linguistic structures of text. More importantly, we study the interaction effect between explicit linguistic structures (e.g., linear, constituency, and dependency) and implicit structures elicited by attention mechanisms. Empirical results on multiple datasets demonstrate salient patterns of substitutability between the two families of structures (explicit and implicit). Our findings not only provide intellectual foundations for the popular use of “linear LSTM + attention” architectures in NLP/QA research, but also have implications in other modalities and domains.