Cooperative recognition, a method to achieve human-automation cooperation in the recognition phase of the autonomous driving system, has been proposed to address the challenges in the conventional control phase cooperation, e.g., taking over vehicle control. In cooperative recognition, the operator intervenes in recognition tasks that are difficult for the automated system alone to improve driving efficiency and safety. The challenge is the integration of both human and automated systems while both participants have different characteristics, processing capabilities, and uncertainty in the decisions (recognition results). The objectives of this study are task allocation (i.e., when and for which targets the operator should intervene) taking into account the intervention efficiency and human state. And also combine the human intervention and recognition result of the automated systems to solve the uncertainties in both participants. We formulated this problem with a Partially Observable Markov Decision Process (POMDP). The simulator experiment indicated that the recognition result of the automated system and the operator’s intervention were stochastically combined. The intervention requests to the operator adapted to the operator state and could be reduced while maintaining driving efficiency and minimizing risk omissions.