This paper describes a new learning by observation algorithm based on Bayesian networks and game pattern graphs. Even with minimal knowledge of a game or human instructions, the robot can learn the game rules by watching human demonstrators repeatedly play the game multiple times. Based on the knowledge acquired from this learning process, represented in Bayesian networks and game pattern graphs, the robot can play games as robustly as humans do. Our learning algorithm for human-robot game interaction is implemented using a teddy bear-like robot and is demonstrated by application to well-known social games, specifically Rock-Paper-Scissors, Muk-Chi-ba and Blackjack.