Muscle-actuated organisms exhibit an extraordinary ability to learn a wide array of agile movements. However, replicating such versatility and efficiency in reinforcement learning (RL) poses significant challenges, primarily due to the complexity of over-actuated action spaces. These challenges are often attributed to the sample efficiency issues prevalent in RL, compounded by the inefficacy in exploration strategies within such expansive action domains. To address the challenge of ineffective exploration in over-actuated spaces, we leverage Differential Extrinsic Plasticity (DEP), an innovative self-organizing mechanism designed to enhance and expedite exploration across the state space. To further augment the sample efficiency of reinforcement learning techniques, we introduce an integration with the third generation of neural networks, namely Spiking Neural Networks (SNNs). This integration, forming the core of our DEP-RL framework, sets a new benchmark for rapid and effective learning in musculoskeletal systems. Our approach not only surpasses the performance of conventional DEP-RL methodologies but also marks a significant leap forward in advancing reinforcement learning capabilities within complex, muscle-driven biological architectures.