The deep neural network shows a consequential performance for a set of specific tasks. A system designed for some correlated task altogether can be feasible for `in the wild' applications. This paper proposes a method for the face localization, Action Unit (AU) and emotion detection. The three different tasks are performed by a simultaneous hierarchical network which exploits the way of learning of neural networks. Such network can represent more relevant features than the individual network. Due to more complex structures and very deep networks, the deployment of neural networks for real life applications is a challenging task. The paper focuses to find an efficient trade-off between the performance and the complexity of the given tasks. This is done by exploring the advantages of optimization of the network for the given tasks by using separable convolutions, binarization and quantization. Four different databases (AffectNet, EmotioNet, RAF-DB and WiderFace) are used to evaluate the performance of our proposed approach by having a separate task specific database.