This paper proposes a general convex framework for robustly identifying discrete-time affine hybrid systems from measurements contaminated by noise (both process and measurement) and outliers. Our main result shows that this problem can be formulated as a constrained polynomial optimization, for which a monotonically convergent sequence of tractable convex relaxations can be obtained by exploiting recent developments in sparse polynomial optimization. A salient feature of the proposed framework is its ability to incorporate existing a-priori information about the noise, co-ocurrences, or the switching sequence. These results are illustrated with several examples showing the ability of the proposed approach to make effective use of this additional information.