Dependable robotic manipulation of everyday objects and execution of household chores is one of the most desired and challenging skills for future service robots. Most of the current research in robotic grasping is limited to pick-and-place tasks, without paying attention to the whole range of different tasks needed in human environments, such as opening doors, interacting with furniture, household electrical appliances, etc. In this article, an integrated sensor-based framework for specifying both the grasp and the task is presented, with the goal of motivating task-oriented grasping and physical interaction based on multisensor information. The grasp is defined as a desired task-suitable relationship between the robot hand and the object being manipulated. The task is defined under the task frame formalism, which allows for sensor-guided compliant interaction. Some guidelines for multisensor-based execution of tasks with hands and tools under the proposed formalism are also given.