Wavelength-Embedding-guided Filter-Array Transformer for Spectral Demosaicing

Haijin Zeng*, Hiep Luong, Wilfried Philips ;

"Spectral imaging offers the capability to unveil hidden details within the world around us. However, to fully harness this potential, it is imperative to develop effective spectral demosaicing techniques. Despite the success of learning based spectral demosaicing methods, three challenges hinder their practical use. Firstly, existing convolutional neural networks and attention-based models, struggle to capture spectral similarities and long-range dependencies. Secondly, their performance is unstable when optical characteristics, like multispectral filter array (MSFA) arrangement and wavelength distribution, change. Lastly, they lack a structured approach to incorporating imaging system physics, such as MSFA pattern. Addressing these challenges, our paper introduces the Wavelength Embedding guided Filter Array Attention Transformer (WeFAT) for effective spectral demosaicing. Specifically, inspired by the timestep embedding in denoising diffusion models, we propose a Wavelength Embedding guided Multi-head Self-Attention (We-MSA) mechanism to imbue our model with wavelength memory, facilitating adaptation to diverse cameras. This approach treats each spectral feature as a token, directly integrating wavelength information into attention calculation. Additionally, we developed a MSFA-attention Mechanism (MaM) steering We-MSA to focus on spatial regions yielding high-quality spectral data. Experimental results affirm that WeFAT exhibits strong performance consistency across diverse cameras characterized by varying spectral distributions and MSFA patterns, trained solely on ARAD dataset. It also outperforms current state-of-the-art methods in both simulated and real datasets."