We propose a novel interface called Twinkle for interacting with an arbitrary physical surface using a handheld projector and a camera. When a user flashes a projection light on an object, the projected images react as if the user touched the object with the light. The handheld device recognizes the features of the physical environment and displays images and sounds that are generated in real-time according to the user's motion and collisions of projected images with objects. We realize this system by using several image-processing techniques and a collision detection algorithm. We also use an acceleration sensor to compensate for the image processing. In this paper, we explain the principle of interacting with a physical surface. Then, we describe the implementation of the prototype system and some application examples.