Do We Really Need that Skip-Connection? Understanding Its Interplay with Task Complexity

Abstract

The U-Net architecture has become the preferred model used for medical image segmentation tasks. Since its inception, several variants have been proposed. An important component of the U-Net architecture is the use of skip-connections, said to carry over image details on its decoder branch at different scales. However, beyond this intuition, not much is known as to what extent skip-connections of the U-Net are necessary, nor what their interplay is in terms of model robustness when they are subjected to different levels of task complexity. In this study, we analyzed these questions using three variants of the U-Net architecture (the standard U-Net, a “No-Skip” U-Net, and an Attention-Gated U-Net) using controlled experiments on varying synthetic texture images and evaluated these findings on three medical image data sets. We measured task complexity as a function of texture-based similarities between foreground and background distributions. Using this scheme, our findings suggest that the benefit of employing skip-connections is small for low-to-medium complexity tasks, and its benefit appears only when the task complexity becomes large. We report that such incremental benefit is non-linear, with the Attention-Gated U-Net yielding larger improvements. Furthermore, we find that these benefits also bring along robustness degradations on clinical data sets, particularly in out-of-domain scenarios. These results suggest a dependency between task complexity and the choice/design of noise-resilient skip-connections, indicating the need for careful consideration while using these skip-connections.