In this work we aim at contributing to the Social Signal Processing (SSP) and the Human-Agent Interaction (HAI) fields. For this we tackle several problems encountered in Non-verbal Conversation Expressions (NCE) processing. Ongoing work on building a naturalistic amused speech synthesis system is presented as well as initiated work on an amusement intensity level estimation system from speech. We also present our current and future work on synthesis and detection of out-of-speech NCE such as affect bursts and backchannels. The main goal is to contribute to the NCE processing integration in HAI applications considering their different types of occurrences in dialog: co-occurring with speech or separated from it.