Convolutional Neural Networks give heavy storage and computation burden to accelerators, whose energy efficiency can be improved by leveraging their sparsity. However, using sparsity in networks will introduce large overhead especially when networks have various sparse situations and multiple quantization. This work named STICKER-II firstly proposes an area/performance balanced chip with simultaneously sparsi-ty/quantization adaptive capability, enabled by multi-sparsity and multi-quantization compatible storage and computation circuits. Further more, N-way set-associate PE architecture is explored to trade off its performance and area with the collision-aware hardware scheduler. This chip achieves 4.64 TOPS/W energy efficiency on Alexnet, 1.65x better than the state-of-the-art sparse processors, along with 19.4% PE area reduction.