Domain adaptive hashing has received increasing attention since it is capable of enhancing the performance of retrieval if the target domain for testing meets domain shift. However, owing to data security and transmission constraints nowadays, abundant source data is often not available. Towards this end, this paper investigates a novel yet practical problem named test-time adaptive hashing, which aims to enhance the performance of hashing models without access to the source domain data when tested on the target domain with domain shift. This problem is challenging due to both fugacious domain shift and label scarcity on the target domain. In this paper, we propose a novel hashing approach named Discrepancy and Structure-based Contrast (DISC) for effective test-time adaptive retrieval. In particular, DISC first trains the hashing model using the source domain data and stores the distribution of each class in the hidden space. During test-time adaptation, we generate simulated source features based on stored distributions and compare class-specific distributions across domains using maximum mean discrepancy (MMD) to overcome potential domain shift. Furthermore, to tackle the label scarcity, we estimate the graph structure using deep features on the target domain, which guides effective hashing contrastive learning for generating discriminative and domain-invariant hash codes. Extensive experiments on various benchmark datasets validate the superiority of our proposed DISC compared with a range of competing baselines.