Low-light image enhancement is an essential task in image restoration. Inspired by the diffusion model, the related methods have achieved remarkable results in low-level visual tasks. However, such methods are susceptible to large-scale images, generating problems such as overconsumption of resources and low recovery efficiency. To address this, we propose a detail-guided latent space low-light image enhancement diffusion model called DLDiff. Leveraging the generative power of the latent diffusion model, we explore ways to speed up inference better while producing excellent perceptual fidelity. Specifically, we initially employ the latent diffusion model to transform low-light image features into a latent space representation, thereby reducing computational resource consumption. Next, we design a lightweight detail prompt module that combines cross-convolution and vast-receptive-field convolution blocks. This module enhances the fine-grained details of the image, effectively supplements multiscale feature information, and minimizes feature loss in the latent space. Furthermore, we devise the content-aware loss group to facilitate learning noise and image information, enhancing the model's recovery capability, guiding stable sampling, and constraining diverse content generation. Through extensive experiments, we demonstrate the model's significant efficiency and quality advantages in low-light image enhancement tasks.