Disentangled Cross-modal Fusion for Event-Guided Image Super-resolution | IEEE Journals & Magazine | IEEE Xplore

Disentangled Cross-modal Fusion for Event-Guided Image Super-resolution


Impact Statement:Image superresolution (SR) of intensity images has wide applications in medical imaging, security surveillance, autonomous driving, and other fields. Event cameras are no...Show More

Abstract:

Event cameras detect the intensity changes and produce asynchronous events with high dynamic range and no motion blur. Recently, several attempts have been made to superr...Show More
Impact Statement:
Image superresolution (SR) of intensity images has wide applications in medical imaging, security surveillance, autonomous driving, and other fields. Event cameras are novel bioinspired sensors that show several advantages such as high temporal resolution, no motion blur, and high dynamic range. Using event data to guide image SR can improve the quality of image superresolution. Compared with previous event-guided ISR methods, our method achieves better results with lower computational costs, thereby improving image processing in various fields. For example, in the field of surveillance, clear images can provide better evidence support. In summary, our method has broad application prospects and social impact. In addition, it is expected to bring practical improvements and innovations to fields such as medicine, security, and entertainment.

Abstract:

Event cameras detect the intensity changes and produce asynchronous events with high dynamic range and no motion blur. Recently, several attempts have been made to superresolve the intensity images guided by events. However, these methods directly fuse the event and image features without distinguishing the modality difference and achieve image superresolution (SR) in multiple steps, leading to error-prone image SR results. Also, they lack quantitative evaluation of real-world data. In this article, we present an end-to-end framework, called event-guided image (EGI)-SR to narrow the modality gap and subtly integrate the event and RGB modality features for effective image SR. Specifically, EGI-SR employs three crossmodality encoders (CME) to learn modality-specific and modality-shared features from the stacked events and the intensity image, respectively. As such, EGI-SR can better mitigate the negative impact of modality varieties and reduce the difference in the feature space between ...
Published in: IEEE Transactions on Artificial Intelligence ( Volume: 5, Issue: 10, October 2024)
Page(s): 5314 - 5324
Date of Publication: 28 June 2024
Electronic ISSN: 2691-4581

References

References is not available for this document.