Source-free Domain Adaptation (SFDA) aims to adapt a model trained on a given (source) environment to the new (target) environment, without directly accessing the source data. Due to the lack of labeled source data, it is often difficult for SFDA methods to provide reliable class representations for the target data. To overcome this issue, we propose the idea of Confidence-based Subsets Feature Alignment (CSFA). CSFA divides the target data into two subsets: confident subset that consists of samples having low entropy class predictions from the source model, and non-confident subset with samples that do not. By using the pseudo-labels from the confident subset, we can frame the original SFDA problem as a Universal Domain Adaptation (UniDA) problem, and provide reliable class representations for the target data by aligning feature distributions of the two subsets. Specifically, we propose a multi-task framework that simultaneously applies a standard SFDA algorithm in combination with a UniDA-inspired algorithm, which further infuses class representations into the adaption process. We evaluate the proposed method on a wide range of cross-domain object recognition tasks and achieve higher or comparable accuracy compared to existing SFDA methods. Ablation studies are conducted to verify the effectiveness of the proposed method.