Fitting primitives is of great importance for remote sensing applications, such as 3-D modeling and as-built surveys. This letter presents a method for fitting primitives that fuses the Bayesian sample consensus (BaySAC) algorithm with a statistical testing of candidate model parameters for unorganized 3-D point clouds. Instead of randomly choosing initial data sets, as in the random sample consensus (RANSAC), we implement a conditional sampling method, which is the BaySAC, to always select the minimum number of data required with the highest inlier probabilities. As the primitive parameters calculated by the different inlier sets should be convergent, this letter presents a statistical testing algorithm for the histogram of the candidate model parameter to compute the prior probability of each data point. Moreover, the probability update is implemented using the simplified Bayes formula. The proposed approach is tested with the data sets of planes, tori, and curved surfaces. The results show that the proposed optimized BaySAC can achieve high computational efficiency (five times higher than the efficiency of the RANSAC for fitting a subset of 12 500 points) and high fitting accuracy (on average, 20% higher than the accuracy of the RANSAC). Moreover, the strategy of prior probability determination is proven to be model-free and, thus, highly applicable.