Visible light-based simultaneous position and orientation estimation (SPO) has drawn intensive attention in recent years due to its high accuracy and generality. A cone geometry-based SPO system has been proposed in our previous work, which shows superior performance in terms of estimation accuracy when compared with other existing schemes. However, there are still two issues need to be addressed, i.e., the excessive required number of light-emitting diodes (LEDs), and susceptibility to various radiation sources, high levels of noise, and strongly reflected lights. In this letter, we try to overcome the above challenges and achieve a robust and low-complexity SPO scheme. Firstly, more appropriate arrangements of LEDs are exploited to reduce the number of LEDs and the estimation uncertainty. Then, a hierarchical edge detection strategy is designed to enhance the estimation robustness by suppressing outliers. Numerical results indicate that the proposed system achieves superior accuracy and robustness with low complexity.