This paper presents the design, implementation, and evaluation of MagWear, a novel biomagnetism-based system that can accurately and inclusively monitor the heart rate, respiration rate, and blood pressure of users. MagWear's contributions are twofold. First, we build a mathematical model that characterizes the magnetic coupling effect of blood flow under the influence of an external magnetic field. This model uncovers the variations in accuracy when monitoring vital signs among individuals. Second, leveraging insights derived from this mathematical model, we present a software-hardware co-design that effectively handles the impact of human diversity on the performance of vital sign monitoring, pushing this generic solution one big step closer to real adoptions. Following IRB protocols, our extensive experiments involving 30 volunteers demonstrate that MagWear achieves high monitoring accuracy with a mean percentage error (MPE) of 1.55% for heart rate (HR), 1.79% for respiration rate (RR), 3.35% for systolic blood pressure (SBP), and 3.89% for diastolic blood pressure (DBP). MagWear can also be extended to detect anemia and blood oxygen saturation, which is also our ongoing work.