This paper presents a novel method for estimating important scenes in soccer videos based on collaborative use of audio-visual Convolutional Neural Network (CNN) features. In soccer games, since game situations influence not only players' movements but also audiences' cheers, analyses of their audio and visual sequences are useful for the estimation of important scenes. In our method, such scenes are estimated from audio and visual CNN features via support vector machine (SVM) in each feature. Furthermore, by applying weighted majority voting based on confidences defined from the SVM-based estimation results, accurate estimation of important scenes becomes feasible. Experimental results show the effectiveness of our method.