Despite the extensive application of Monte Carlo analytic hierarchy process (MCAHP) in various fields of decision making, its performance has not been compared with the classic analytic hierarchy process (AHP). Both of these methods are heavily affected by individual or group preferences and thus provide subjective rankings. Since the mere difference between their results does not necessarily warrant the superiority of one against the other, a reliable and robust ranking of alternatives should be available as a comparison basis so that the results of these two methods can be evaluated. In this paper, we use a simulation approach to compare the results of AHP with MCAHP under different levels of uncertainty. We validate our simulation results by comparing the performance of these two alternatives against a real world and reliable ranking of blogs. Our simulation results show that as long as the variation in different pairwise comparisons is less than 0.24, the performance of AHP is not statistically different from the performance of MCAHP. When the uncertainty in terms of variation grows beyond 0.24, MCAHP provides more precise rankings. The findings of this research add to the current body of knowledge in the multicriteria decision analysis as well as Information Systems literature and provide insights for managerial applications of these techniques.