Sleep apnea is the most prevalent sleep disorder. In severe cases, it can even lead to sudden death. To diagnose sleep apnea, it is critical to measure the number of sleep arousal times per hour. Current clinical standard procedures require patients to visit a sleep center or hospital for polysomnography (PSG), in which the patient's physiological signals are recorded during the night, and a sleep technician then interprets these signals manually to determine the sleep-wake pattern. With the rising number of sleep disorders, sleep technologists cannot afford such a high workload, which makes the development of an automated sleep arousal detection system critical to reducing medical costs. When PSG is recorded, a large number of wires will also interfere with the patient and cause measurement distortion. In this study, a spatial-channel attention (SCA) U-Net for automatic sleep arousal detection was proposed, which was trained and tested on PhysioNet 2018 sleep PSG dataset. Considering clinical applicability and reducing patient disturbance when recording physiological signals, only five physiological signal channels were required for our proposed model. It is much lower than other methods in the literature. The A AUPRC of our proposed model was and 39%. The results showed the proposed method is highly accurate and requires fewer channel signals, that also demonstrated the clinical applicability and robustness of the proposed method.