With the advances of Artificial Intelligence (AI) and autonomous robotics, robots are increasingly involved closely with human society, such as in manufacturing system, autonomous driving and intelligent service robots. While robots are good at handling repeated routine work, human are more adaptive and flexible to changing factors that may bring uncertainties and cost non-trivial efforts for robots to overcome. Thus efficient collaboration between human and robots can bring huge economic profits, but also require high standard for robot controller during task completion because the safety of human must be guaranteed. In this paper, we build a formal design theory enabling an automated human-robot collaboration with performance guarantees. Markov Decision Processes (MDPs) are used to model robots with uncertainties and Partially Observable Markov Decision Processes (POMDPs) are used to model human with hidden intents. Within a supervisory control framework, we propose to use L* learning algorithm to learn a correct supervisor such that global collaboration task given as Probabilistic Computation Tree Logic (PCTL) can be satisfied. An example from manufacturing system is discussed to further illustrate the design method.