Segment Anything Model (SAM) proposes promptable image segmentation based on various types of prompts like points and boxes. Although SAM presents impressive performance on segmentation and provides unparalleled versatility, it still suffers from the threat of adversarial attacks. In this paper, we investigate the problem of cross-prompt adversarial attacks on SAM, which considers whether the adversarial samples obtained by attacking with one or more prompts can lead to incorrect masks under other unseen test prompts. We analyze the factors influencing cross-prompt attacks and explore attacks on different numbers of prompts. Based on the analysis, we propose Omni-Attack-SAM, an innovative and effective attack method to produce adversarial samples that are transferable to unseen prompts. Our experiments on SA-1B and the Pascal VOC2012 dataset demonstrate that our method can decrease the mIoU value by over 35% compared to existing adversarial attack methods without using the ground truth of images. Additionally, we can also combine our method with the prompt information to achieve superior attack performance in various scenes.