In this paper, we analyze the impact of channel estimation errors in an amplify-and-forward (AF)-based two-way relaying network (TWRN) where adaptive modulation (AM) is employed in individual relaying path. In particular, the performance degradation caused by channel estimation error is investigated over Nakagami-m fading channels. We first derive an end-to-end signal-to-noise ratio (SNR), a cumulative distribution function, and a probability density function in the presence of channel estimation error for the AF-based TWRN with adaptive modulation (TWRN-AM). By utilizing the derived SNR statistics, we present accurate expressions of the average spectral efficiency and bit error rates with an outage-constraint in which transmission does not take place during outage events of bidirectional communications. Based on our derived analytical results, an optimal power allocation scheme for TWRN-AM is proposed to improve the average spectral efficiency by minimizing system outages.