Encrypted medical data outsourced to cloud servers can be used for personal health certification, health monitoring, and medical research. These data are essential to support the development of the medical industry. However, the traditional peer-to-peer data-sharing paradigm can lead to data abuse by malicious data analysis centers. Moreover, the encryption used to protect users’ outsourced privacy restricts the flexibility of data retrieval. Based on the modified double trapdoor cryptosystem, we propose an authorized data retrieval scheme over aggregated encrypted medical data (ADR-AED) in cloud-assisted e-healthcare systems. In ADR-AED, patients can access and decrypt personal data and authorize the data analysis center (DAC) to retrieve corresponding data. Specifically, we design an authorized retrieval-test mechanism for a group of patients to DAC. This allows DAC to extract valuable information from a threshold number of authorized users. Additionally, each patient can flexibly retrieve fine-grained medical data in different periods and submit them to a doctor for diagnostic analysis. The security analysis and performance evaluation demonstrate the feasibility of ADR-AED in the deployment of cloud-assisted e-healthcare systems.