Due to the capability of controlling wireless environments, reconfigurable intelligent surface (RIS) can be exploited to assist radio channel-based physical-layer key generation (PKG). However, this may also pose great security threats to PKG when it is controlled by an attacker. To this end, this letter studies a RIS leakage (RISL) attack that assists the attacker to manipulate the generated keys between the legitimate ends in PKG systems. Specifically, in block fading environments, by alternatingly adapting the RIS reflection coefficients to all 0s and 1s in each block, the received signal strength (RSS) at the legitimate ends is likely to be boosted and attenuated regularly across time, causing predictable 0s and 1s in the quantized bits. To resist this attack, we propose a countermeasure based on dynamic private pilots (CDPP) and prove that the RISL success probability can be significantly reduced by CDPP. Simulation results verify the effectiveness of the proposed attack and countermeasure, highlighting the importance of introducing artificial randomness for secret key security in block fading channels.