We consider the problem of defending neural networks against adversarial inputs. In particular, we extend the framework introduced in [1] to defend neural networks against ℓ₂, ℓ_∞, and ℓ₀ norm attacks. We call this defense framework Compressive Recovery Defense (CRD) as it utilizes recovery algorithms from the theory of compressive sensing. For defending against ℓ₂-norm and ℓ₀-norm attacks, we use Basis Pursuit (BP) as the recovery algorithm and for the case of ℓ_∞-norm attacks, we utilize the Dantzig Selector (DS) with a novel constraint. For each recovery algorithm used, we provide rigorous recovery guarantees that do not depend on the noise generating mechanism and can therefore be utilized by CRD against any ℓ₂, ℓ_∞, or ℓ₀ norm attacks. Finally, we experimentally demonstrate that CRD is effective in defending neural networks against state of the art ℓ₂, ℓ_∞ and ℓ₀-norm attacks.