Testing AI Systems Leveraging Graph Perturbation

Automated testing for emerging AI-enabled systems is challenging, because data is often highly structured, semantically rich, and continuously evolving. Fuzz testing has been proven to be highly effective; however, it is nontrivial to apply traditional fuzzing to AI systems directly for three reasons: (1) it often fails to bypass format validity checks, which are crucial for testing the core logic of an AI application; (2) it struggles to explore various semantic properties of inputs; and (3) it is incapable of accommodating the latency of AI systems. In this paper, we propose a novel fuzz testing framework specifically for AI systems, called SynGraph. Our approach stands out in two key aspects. First, we utilize graph perturbations to produce syntactically correct data, as opposed to traditional bit-level data manipulation. To achieve this, SynGraph captures the structured information intrinsic to the data and represents it as a graph. Second, we conduct directed mutations that preserve semantic similarity by applying the same mutations to adjacent and similar vertices. SynGraph has been successfully implemented for 5 input modalities. Experimental results demonstrate that this approach significantly enhances testing efficiency.