NetEyes is a system that allows remote and co-located partners to collaborate by annotating maps printed on digital paper. It combines natural language capabilities, in particular the interpretation of sketched symbols, with the display of three-dimensional representations of recognized objects, allowing users to jointly visualize a planned or evolving situation in detail. Visualization can take place either on a conventional monitor or through optical see-though, head-mounted displays. Seamless collaboration is promoted via the use of tangible paper maps, which makes it possible for multiple parties sitting around a table to place annotations as they would using regular paper and pen. To account for movements and rotation of the maps that are common during collaborative annotation sessions, a vision-based tracking component is used. This component recovers the location of the paper map in the real-world, and scales and rotates the digitally displayed objects so that they keep aligned with the map.