Inertial measurement units (IMUs) are widely adopted for pedestrian tracking with high-frequent, scale-consistent, and environment-independent ego-motion measurements. However, such inertial readings in smartphones are usually plagued by heavy noises, causing unexpected tracking errors and impeding the deployment at large-scale. Currently, many research works explore motion perception and position estimation with deep neural networks (DNNs), but training a general model requires sufficient and widespread data covering most scenarios. In this article, we present a smartphone inertial measurement dataset (SIMD) with more than 4500 walking trajectories, which takes about 190 h with a total walking distance of more than $700\,\text {km}$ . It covers four cities, 12 indoor and outdoor scenarios, seven phone attitudes, and more than 150 volunteers with their smartphones. To locate pedestrians indoors, we propose a general inertial tracking framework to train with our dataset and infer user’s trajectory. Furthermore, we explore the potential deployment of model customization on individual smartphones. Extensive experiments have shown our effectiveness on pedestrian tracking and navigation, compared with the state-of-the-art.