Deformable object manipulation has potential for a wide range of real-world applications, but is still largely unsolved due to the complex dynamics and difficulty of state estimation. Learning-based approaches have recently accelerated progress, but generally depend heavily on large simulated datasets, human demonstrations or both. In this study, we propose a novel sample-efficient learning approach to deformable linear object manipulation (e.g. rope) that can be applied directly to the real world, without requiring any simulated or human demonstration data. We transform the observation and action space into a canonical image representation, allowing us to leverage the spatial structure preserving abilities of fully-convolutional networks to learn image-based predictive models. We name our approach Canonical Visual Forward model (CaVFM). We pair this learning approach with self-supervised data collection and learning on a real robot, and demonstrate its effectiveness at goal reaching tasks with only 1,000 samples of real-world data. Our method out-performs prior methods in this extreme low-data regime, without suffering performance loss when larger datasets are available.