Our world is non-static, and robots should be able to track its changing geometry. For tracking changes, data asso-ciations between 3D points over time are key. In this paper, we investigate the problem of associating 3D points on plant organs from different mapping runs over time while the plants grow. We achieve a high spatial-temporal matching performance by combining 3D RGB-D SLAM, visual place recognition, and 2D/3D matching exploiting background knowledge. We showcase our approach in a real agricultural glasshouse used to grow sweet peppers, using RGB-D observations from a mobile robot traversing the environment. Our experiments suggest that with our approach, we can robustly make data associations in highly repetitive scenes and under changing geometries caused by plant growth. We see our approach as an important step towards spatial-temporal data association for robotic agriculture.