Neural networks have attracted much attention due to their excellent performance in providing insightful predictions. The trained neural network models usually have millions of parameters requiring massive storage resources, which motivates the model owner to deploy their models to cloud servers for relieving the storage burden. The client can also directly enjoy the prediction service provided by the cloud server. Albeit convenient, untrusted cloud servers may violate the privacy of the model owners and the clients, which hinders the wide applications of such a prediction service. This survey reviews various privacy-preserving neural network prediction services, which protect the privacy of the model and the query. Many protocols are in the basic secure two-party computation (S2C) setting, which protects the secret of the querier and the model owner against the counterparty. Secure outsourcing further protects the privacy of the model against the cloud hosting it for the prediction service. We compare the existing approaches in terms of security, accuracy, and efficiency. We then propose an optimized neural network prediction scheme in the outsourcing setting, which simultaneously achieves high accuracy, model privacy, and low overheads, and conduct an experimental evaluation for computation time and communication costs. Finally, we highlight several future research directions and provide new insights into open problems in strengthening security and improving efficiency.