Temperature dependence of charge capture and emission in HfO2 and ferroelectric doped HfO2 are examined over a wide temperature range. Sizeable threshold voltage (Vth) instabilities are observed under cryogenic conditions, contrary to expectation of Arrhenius-based defect freeze-out. The observed data is modelled with ultra-fast defect levels, located close to the silicon channel. The impact of these traps at room temperature on ferroelectric devices is significant: capture and emission times lie in the range applied for ferroelectric device operation, and can explain the read-after-write delay incorporated in Si-based ferro (FE-)FET operation.