Add to BibSonomyAbstract
With the ever-increasing size of training models and datasets, network communication has emerged as a major bottleneck in distributed deep learning training. To address this challenge, we propose an optical distributed deep learning (ODDL) architecture. ODDL utilizes a fast yet scalable all-optical network architecture to accelerate distributed training. One of the key features of the architecture is its flow-based transmit scheduling with fast reconfiguration. This allows ODDL to allocate dedicated optical paths for each traffic stream dynamically, resulting in low network latency and high network utilization. Additionally, ODDL provides physically isolated and tailored network resources for training tasks by reconfiguring the optical switch using LCoS-WSS technology. The ODDL topology also uses tunable transceivers to adapt to time-varying traffic patterns. To achieve accurate and fine-grained scheduling of optical circuits, we propose an efficient distributed control scheme that incurs minimal delay overhead. Our evaluation on real-world traces showcases ODDL’s remarkable performance. When implemented with 1024 nodes and 100 Gbps bandwidth, ODDL accelerates VGG19 training by $1.6 \times$ and $1.7 \times$ compared to conventional fat-tree electrical networks and photonic SiP-Ring architectures, respectively. We further build a four-node testbed, and our experiments show that ODDL can achieve comparable training time compared to that of an ideal electrical switching network.
© 2024 Optica Publishing Group
Full Article | PDF ArticleMore Like This
Zhenguo Wu, Liang Yuan Dai, Yuyang Wang, Songli Wang, and Keren Bergman
J. Opt. Commun. Netw. 16(2) A157-A168 (2024)
Cen Wang, Noboru Yoshikane, Daniel Elson, Yuta Wakayama, Daiki Soma, Shohei Beppu, and Takehiro Tsuritani
J. Opt. Commun. Netw. 16(1) A40-A52 (2024)
Jie Xiao, Bin Wu, Xiaohong Jiang, Achille Pattavina, Hong Wen, and Lei Zhang
J. Opt. Commun. Netw. 6(3) 270-281 (2014)