Robots fabricated by soft materials can provide higher flexibility and thus better safety while interacting with natural objects with low stiffness such as food and human beings. However, as many more degrees of freedom are introduced, the motion simulation of a soft robot becomes cumbersome, especially when large deformations are presented. Moreover, when the actuation is defined by geometry variation, it is not easy to obtain the exact loads and material properties to be used in the conventional methods of deformation simulation. In this paper, we present a direct approach to take the geometric actuation as input and compute the deformed shape of soft robots by numerical optimization using a geometry-based algorithm. By a simple calibration, the properties of multiple materials can be modeled geometrically in the framework. Numerical and experimental tests have been conducted to demonstrate the performance of our approach on both cable-driven and pneumatic actuators in soft robotics.