WiFi-sensing systems that can accomplish multiple relative tasks simultaneously are attracting significant attention due to their potential for large-scale commercial applications. However, different WiFi sensing scenarios may often rely on various task-specific features, posing a challenge in balancing these different, or asymmetrical, characteristics across tasks. In this article, we propose a system called WiADN which aims to address the asymmetrical problems in the joint recognition of users’ locations and activities. Our system is composed of two critical parts: 1) an asymmetrical network architecture and 2) an adaptive weight loss (AWL) module employed during the training phase. First, we define activity recognition as a high-level task and location recognition as a low-level task based on their respective difficulty levels. Then, the proposed architecture leverages the cascading attention-based modules to transfer the prior knowledge between different tasks. The key insight of the proposed architecture is to mimic the skilled learners in similar situations, who often tackle easier problems first to enable them to solve more challenging problems later on. Based on this behavioral strategy, the proposed attention-based modules are designed to generate masks to select specific characteristics from the low-level task to help the high-level task to learn respective features more effectively. Additionally, the AWL module based on the task uncertainty theory is employed to balance two tasks’ asymmetry from the perspective of loss optimization. Furthermore, extensive experiment results based on two open data sets demonstrate the superiority of our system in the accuracy with other state-of-the-art methods. At last, the effectiveness and the robustness of our system are also verified through the comparative studies and ablation experiments. Our source codes are available at https://github.com/jzhoujg/WiADN.