This paper investigates a nonlinear control design for the vertical downhole drilling process. The drilling system dynamics are first built incorporating both the axial and the torsional dynamics together with a velocity independent drill bit-rock interaction model. Given the underactuated, nonlinear and non-smooth feature of the drilling dynamics, we propose a new control design method to prevent significant downhole vibrations and to achieve desired rate of penetration. Specifically, the controller is designed using a sequence of hyperplanes given in a cascade structure, which allows stabilization of the closed loop system without occurrence of stick-slip oscillations while ensuring robustness against modeling uncertainties and external disturbances. The cascade architecture of the controller also allows its extension to higher order cases if more complicated drilling dynamics model is needed for the control synthesis. To this end, simulation results are exhibited to demonstrate the effectiveness of the proposed control approach.