Time series data from multiple modalities such as physiological and motion sensor data have proven to be integral for measuring mental wellbeing however individual differences between people limit the generalisability of deep learning models especially for those with intellectual disabilities. It is impractical, time consuming and extremely challenging to collect large realworld datasets of individuals’ wellbeing in their everyday life. Therefore, to address this challenge, we propose a Transfer Learning (TL) approach that develops personalised real-world affective models using few labelled samples by adapting a controlled stressor model. This approach to personalise models and improve cross-domain performance is completed on-device, automating the traditionally manual process saving time and labour. The results show adopting the TL approach significantly increased model performance with the multivariate physiological and motion affective model achieving an average accuracy of 93.5 % compared with the comparative non-TL model accuracy of 71.7%. The proposed methodology helps overcome problems with affective model personalisation, thus improving on the performance of conventional deep learning methods.