Satellite formation flying allows for multiple satellites working together to accomplish the objective of one larger, usually more expensive, satellite. Knowledge of the relative position is required for many formation flying missions. Autonomous relative navigation system based on RF (radio frequency) can be used to acquire such knowledge. Obtaining accurate measurements requires a balanced number of transmitting and receiving antennas with suitable configuration. So under the condition of fixed number of antennas and determined configuration the ranging accuracy, inter-satellite distance, azimuth and elevation are the main factors which affect precision estimation. A LS (Least Squares) algorithm and Newton iteration method are adopted to estimate inter-satellite positions and attitudes. Simulation analysis shows that when the inter-satellite distance increases, the position precision estimate remains minim, and the same applies to angles around the x-axis and y-axis where the measurements errors increase slightly in a linear relationship. Farther, when rotating the satellite around the azimuth (or elevation) angle it would have a small impact on the precision estimate, and contrariwise, it would be the worst when the azimuth (or elevation) is equal to π/4 or - π/4; therefore rises the need for installing additional antennas on other faces of the satellite to improve precision estimate.