Energy-induced Explicit quantification for Multi-modality MRI fusion

Xiaoming Qi*, Yuan Zhang, Tong Wang, Guanyu Yang*, Yueming Jin*, Shuo Li ;

"Multi-modality magnetic resonance imaging (MRI) is crucial for accurate disease diagnosis and surgical planning by comprehensively analyzing multi-modality information fusion. This fusion is characterized by unique patterns of information aggregation for each disease across modalities, influenced by distinct inter-dependencies and shifts in information flow. Existing fusion methods implicitly identify distinct aggregation patterns for various tasks, indicating the potential for developing a unified and explicit aggregation pattern. In this study, we propose a novel aggregation pattern, Energy-induced Explicit Propagation and Alignment (E2 PA), to explicitly quantify and optimize the properties of multi-modality MRI fusion to adapt to different scenarios. In E2 PA, (1) An energy-guided hierarchical fusion (EHF) uncovers the quantification and optimization of inter-dependencies propagation among multi-modalities by hierarchical same energy among patients. (2) An energy-regularized space alignment (ESA) measures the consistency of information flow in multi-modality aggregation by the alignment on space factorization and energy minimization. Through the extensive experiments on three public multi-modality MRI datasets (with different modality combinations and tasks), the superiority of E2 PA can be demonstrated from the comparison with state-of-the-art methods. Our code is available at https://github.com/JerryQseu/EEPA."