Human activity recognition (HAR) using smartphone sensors has attracted great attention due to its wide range of applications. A standard solution for HAR is to first generate some features defined based on domain knowledge (handcrafted features) and then to train an activity classification model based on these features. Very recently, deep learning with automatic feature learning from raw sensory data has also achieved great performance for HAR task. We believe that both the handcrafted features and the learned features may convey some unique information that can complement each other for HAR. In this article, we first propose a feature fusion framework to combine handcrafted features with automatically learned features by a deep algorithm for HAR. Then, taking the regular dynamics of human behavior into consideration, we develop a maximum full a posteriori algorithm to further enhance the performance of HAR. Our extensive experimental results show the proposed approach can achieve superior performance comparing with the state-of-the-art methodologies across both a public data set and a self-collected data set.