Varifocal displays are a practical method to solve vergence-accommodation conflict in near-eye displays for both virtual and augmented reality, but they are reliant on knowing the user's focal state. One approach for detecting the focal state is to use the link between vergence and accommodation and employ binocular gaze tracking to determine the depth of the fixation point; consequently, the focal depth is also known. In order to ensure the virtual image is in focus, the display must be set to a depth which causes no negative perceptual or physiological effects to the viewer, which indicates error bounds for the calculation of fixation point. I analyze the required gaze tracker accuracy to ensure the display focus is set within the viewer's depth of field, zone of comfort, and zone of clear single binocular vision. My findings indicate that for the median adult using an augmented reality varifocal display, gaze tracking accuracy must be better than 0.541°. In addition, I discuss eye tracking approaches presented in the literature to determine their ability to meet the specified requirements.