This paper describes an adaptive training technique for hidden semi-Markov model (HSMM). The adaptive training scheme conducts normalization of speaker differences and acoustic variability in both output and state duration distributions of a canonical model by using HSMM-based MLLR (maximum likelihood linear regression) adaptation. We incorporate the adaptive training into our HSMM-based speech synthesis system with MLLR adaptation and compare synthesized speech using the adaptive training with that using standard speaker independent training. From the results of subjective tests, we show that the adaptive training outperforms speaker independent training and also show that the speech synthesis system generates speech with better naturalness and intelligibility than the original HSMM-based speech synthesis system.