With accelerated advances in various technologies, drones, better known as unmanned aerial vehicles (UAVs), are increasingly commonplace and consequently have a more pronounced impact on society. For example, Internet of Drones (IoD), a new communication paradigm offering fundamental navigation assistance and access to information, has widespread applications ranging from agricultural drones in farming to surveillance drones in the COVID-19 pandemic. The increasingly prominent role of IoD in our society also reinforces the importance of securing such systems against various data privacy and security threats. Operationally, it can be challenging to adopt conventional off-the-shelf security products in an IoD system due to the underpinning characteristics of drones (e.g., dynamic and open communication channel). Therefore in this article, we propose a lightweight and privacy-preserving mutual authentication and key agreement protocol, hereafter referred to as PMAP. The latter uses a physical unclonable function (PUF) and chaotic system to support mutual authentication and establish a secure session key between communication entities in the IoD system. To be specific, PMAP consists of two schemes, namely: 1) ${\mathrm{ PMAP}}^{D2Z}$ (that mutually authenticates drone and zone service provider (ZSP) and establishes secure session keys) and 2) ${\mathrm{ PMAP}}^{D2D}$ (that mutually authenticates drones and establishes secure session keys). In addition, PMAP supports conditional privacy preserving so that the genuine identity of drones can only be revealed by trusted ZSPs. We evaluate the security of PMAP using automated validation of Internet security protocols and application (AVISPA), as well as provide formal and informal security analysis to show the resilience of PMAP against various security attacks. We also evaluate the performance of PMAP through extensive experiments and compare its performance with existing AKA and IBE-Lite schemes, whose findings show tha...