Dynamic spectrum access (DSA) has been regarded as a promising solution to mitigate the serious spectrum shortage problem in the 6G networks, in which secondary users (SUs) are allowed to opportunistically access the licensed bands when primary users (PUs) are inactive. Due to the hardware limitation, partial spectrum sensing with a suitable sensing window (SW) is considered as an effective way to find the idle bands to access. It is noteworthy that the SW selection could determine how many bands are available to access, and the network performance after the access could be used to guide the SW selection. Thus, a sophisticated joint design on both spectrum sensing and access is necessary, which, however, is not an easy task considering the uncertainty and dynamics of the spectrum environment, and also the mutual impacts among SUs. In this paper, we propose a joint partial spectrum sensing and power allocation (PA) scheme to help each SU make the best SW and PA decisions that can optimize the network throughput. To achieve the best decision under the dynamic and uncertain of the environment, considering the mutual interference issue, we develop a multi-agent deep reinforcement learning approach to enable each SU to obtain the best SW and PA decisions autonomously and adaptively.