Passive millimeter-wave imaging is one of the most important approaches for personnel security inspection to detect concealed objects hidden under clothing. Many imaging systems have been constructed and tried out experimentally in various real scenarios. However, plenty of imaging results show that some hidden objects placed on the body edge areas may have low contrast and are difficult to detect. There are different contrasts in different polarization images. In this article, we propose a physically-based subregional polarization fusion method by modulating multipolarization information to enhance the contrast between edge objects and the body background. The physical model and polarization properties of edge objects are investigated by theoretical calculation and simulation analysis. By dividing the body area into four typical subregions based on the second and third Stokes components ($T_{Q}$ and $T_{U}$), four linear polarization images are automatically fused to directly generate an enhancement image. Without complicated parameter adjustment, the proposed method can adaptively realize the global optimal contrast by fusing the local optimal regions of different polarization images at the pixel level. Simulation and measurement results demonstrate the effectiveness of the method. By comparing with three traditional fusion methods, differential signal noise ratio, receiver operating characteristic curve, and segmentation metric are utilized to quantitatively evaluate the superior performance of our fusion method.