In many situations such as chat and discussion, we interact with other people, and interaction analysis is useful for improving communication skills and evaluating a learning effect of education. Although the audio information is possible to analyze the speaker behavior, it is difficult to analyze the non-speaker. On the other hand, the image information is possible to estimate the inside information, such as the concentration level and understanding level of the interaction of the non-speaker not only the speaker by using nodding, face direction, facial expression, and gaze. This paper proposes an image-based method to automatically estimate the attention level of participants for the interaction scene. We collected simulated interaction scenes by using the omnidirectional camera, and applied the proposed method. As a result, we confirmed the effectiveness of the proposed method.