Driving on an uneven road causes vibration of the vehicle. These vibrations affect the comfort of the road users but also convey information about the road profile. This study demonstrates the feasibility, accuracy, and repeatability of the accelerometer system for pavement evaluation that computes the international roughness index (IRI) using multiple smartphones to collect data. It uses quarter-car and half-car suspension models calibrated through rides on speed bumps or a section with a known profile. We report on a field test with eight runs over an 18.6-km route of urban asphalt roads. The correlation of our results with the reference laser profilometer IRI oscillates around 0.7 for a single measurement and reaches 0.85 in the case of using multiple sensors and repeated rides. We also demonstrate that inaccuracies in the spatial attribution of the road localization are a similarly important source of errors as the uncertainty related to mechanical models.