Teleoperation offers the possibility to combine human intelligence with robot power and endurance, making it a perfect solution for hostile-for-human environments. Nonetheless, when it concerns prolonged and repetitive operations, classical teleoperation interfaces that replicate one-to-one human commands can be quite demanding in terms of physical and mental efforts. Shared-autonomy approaches, which foresee the combination of direct teleoperation with autonomous control, can help the operator in overcoming such issues without sacrificing the task execution efficiency. Following this philosophy, in this letter we propose a novel technique to assist the operators in reaching and manipulation of objects with one or two arms, combining local visual perception and operator’s action monitoring. In more details, the reaching intention of an operator towards a target object is detected, which is used to adapt robot trajectories autonomously towards a successful grasping pose. In addition, based on the detected size of a target object, single- or dual-arm coordinated movements are autonomously generated without the need for additional human interventions. The experimental results on thirteen participants confirm the potential of the proposed framework in terms of success rate and operator effort.