In this paper we present a self-learning speech controlled system comprising speech recognition, speaker identification and speaker adaptation. Our goal is the automatic personalization of speech controlled devices for some five recurring speakers without the requirement of a speaker specific training. We present a novel approach to detect unknown speakers based on a unified speech and speaker model. New users are detected in an unsupervised manner based on only a few utterances. New speaker profiles are initialized without any additional intervention of the user. Each profile is continuously adapted by tracking the speaker identity on successive utterances to enhance future speech recognition and speaker identification. Experiments on the evolution of such a system were carried out on a subset of the SPEECON database. The results show that in the long run the system produces adaptation profiles which give improvements in speech recognition rate only slightly lower than supervised or closed-set systems.