A recent trend in both industry and research is the Internet of Drones, which has applications in both civilian and military settings. However, drones (also known as unmanned aerial vehicles) are generally not designed with security in mind, and there are fundamental security and privacy issues that need study. Hence, in this article, we study the architecture and its security and privacy requirements. We also outline potential solutions to address challenging issues such as privacy leakage, data confidentiality protection, and flexible accessibility, with the hope that this article will provide the basis for future research in this emerging area.