This article aims to solve the trajectory tracking task of the pneumatic musculoskeletal robot within a model-based reinforcement learning framework. Considering the limited sensors and short lifespan of self-made pneumatic artificial muscles, physics priors are encoded into Gaussian process regression to implement a semiparametric model for microdata system identification and the identified model is combined with cross-entropy method (CEM)-based model predictive control to plan for the optimal action online. To further compensate for the model imperfection and improve the control performance, a hybrid feedforward and feedback controller-like strategy is proposed to guide the search space of the original CEM solver. The effectiveness of our approach is verified on a real musculoskeletal manipulator with two degrees of freedom, and the results show that only 50 s of interacting with the environment is enough for the robot to learn writing alphabet letters from scratch.