This paper presents a rigorous but tractable study of sparsity. We postulate a definition of sparsity that is as broad as possible, so that it generates all the various measures that are useful in practice, but narrow enough that the fundamental properties of generalized sparsity still hold. As we work through the various ways of demonstrating the advantageous properties of sparsity, we illustrate its meaning from geometrical and operational perspectives. Thereafter, we construct specific measures of sparsity which are successfully qualified in complexity analysis and sparse optimization scenarios. Overall, our main objective is to construct measures of sparsity that will facilitate and enhance the design of the next innovative sensing technologies.