Some state-of-the-art binary image steganographic methods aim to generate stego images with good visual quality, while others focus more on the statistical security of the anti-steganalysis. This paper proposes a binary steganographic scheme that improves both of them by selecting more appropriate flipped pixels. First, a fused distortion measurement is developed that combines the advantages of flipping distortion measurement (FDM) and two data-carrying pixel location methods, including the edge adaptive grid method (EAG) and the “Connectivity Preserving” criterion (CPc). The FDM measures the distortion score by statistical features and achieves high-statistical security, while the EAG and CPc select pixels by analyzing the local texture structures based on visual quality. Then, to eliminate the interference brought by adjacent flipped pixels, a flipping position optimization strategy is proposed to find better positions for flipping pixels to further improve the steganographic performance. Experimental results have demonstrated that the proposed steganographic scheme can achieve stronger statistical security with better visual quality without degrading the embedding capacity.