Implicit feedback such as video watch time is commonly seen in many internet products. Though recommender systems with explicit feedback have been abundantly researched, there are not many methods proposed for building recommender systems with implicit feedback. Restricted Boltzmann Machine (RBM), which uses a two-layer graphical model to describe response variables through hidden units, is a promising approach. An RBM model for collaborative filtering was proposed in the literature, but it only deals with categorized ratings, which is explicit feedback. We propose a novel extension of the RBM method for collaborative filtering with implicit feedback. The model parameters can be learned efficiently through the contrastive divergence algorithm. Compared to other methods, the proposed RBM method can directly predict preferences for a new user given feedback on a few items. Also, it protects privacy by keeping user data locally and only sends updated parameters to a central server. The results on real data with several million records show it works superiorly compared to other prevalent methods.