This paper will illustrate the application of constraint based Robust Generalized Dynamic Inversion (RGDI) control for balancing the Rotary Inverted Pendulum system. The proposed control law is established by hybridizing the equivalent control (conventional GDI) and the robust control. The equivalent control is constructed by formulating the dynamical constraints, based on the attitude deviation functions of the rotary arm and the vertical pendulum. The constraint dynamics are then inverted using dynamically scaled Moore-Penrose Generalized Inverse to solve for the equivalent control. For the robust term, sliding mode control is utilized, that will provide robustness against parametric uncertainties, system nonlinearities, such that semi-global practically stable angular position tracking is guaranteed. The controller's capability is verified through computer simulation and experimental studies on the Quanser's SRV02 Rotary Inverted Pendulum system.