Integral quadratic constraints (IQC) provide a framework for robust stability analysis of feedback systems. In this paper we propose a way of exploiting certain special structures of interconnection topologies for analysing the stability of time-delayed interconnected passive systems. An advantage of this approach is that gridding over frequency can be avoided. Moreover, in contrast to previous results the problem complexity is independent of the network size. We compare results with a robust stability analysis method that uses the IQC framework and a set of frequency-independent LMIs in terms of conservatism. The comparison shows that the proposed approach is only slightly more conservative than methods where the problem complexity increases with network size.