In this paper, the system identification problem from the high-dimensional input and output is considered. If the relationship between the features extracted from the data is represented as a linear time-invariant dynamical system, the input-output manifold learning method has shown to be a powerful tool for solving such a system identification problem. However, in the previous study, the system is assumed to be initially relaxed because the transfer function model is used for system representation. This assumption may not hold in several tasks. To handle the initially non-relaxed system, we propose the alternative approach of the input-output manifold learning with state space model for the system representation. The effectiveness of our proposed method is confirmed by experiments with synthetic data and motion capture data of human-human conversation.