We introduce spatial PACT (Principal component Analysis of Census Transform histograms), a new representation for recognizing instances and categories of places or scenes. Both place instance recognition (“I am in Room 113”) and category recognition (“I am in an office”) have been widely researched. Features that have different discriminative power/invariance tradeoff have been used separately for the two tasks. PACT captures local structures of an image through the Census Transform (CT), while large-scale structures are captured by the strong correlation between neighboring CT values and the histogram. The PCA operation ignores noise in the histogram distribution, computes important “primitive shapes”, and results in a compact representation. Spatial PACT, a spatial pyramid of PACT, further incorporates global structures in the image. Our experiments demonstrate that spatial PACT outperforms the current state-of-the-art in several place and scene recognition, and shape matching datasets. Besides, spatial PACT is easy to implement. It has nearly no parameter to tune, and evaluates extremely fast.