Bridging the gap between the Internet of Things and collaborative robots, the recent advancements in the Internet of Robotic Things (IoRT) aim at significantly improving production and operation efficiency and quality. As the scope and complexity of IoRT continue to expand, involving also very large numbers of robots, there is a need for employment of innovative solutions such as federated learning. However, this growing demand is accompanied by multiple challenges, including threats to data privacy and model integrity. Besides, the heterogeneity of the robots and their interaction, multiplies these challenges. In this paper, we discuss the key concerns of collaborative training in IoRT, and propose a shuffling-based moving target defense approach for federated learning in heterogeneous cross-silo IoRT environments (FedMTD). Based on a hierarchical training structure with node clustering, FedMTD bounds heterogeneity by domains, thereby minimizing the learning error and privacy loss. It also enhances resistance to poisoning attacks through decentralized credit evaluation. Experimental results show that FedMTD brings significant improvements in learning performance, privacy enhancement, and poisoning resistance.