2.1 CNN-based fusion methods

CNN-based fusion methods can automatically learn the features of the input image by designing a specific network structure and loss function, and fuse these features to generate a high-quality fused image. This process is mainly divided into three steps: feature extraction, feature fusion and image reconstruction. For example, Zhang et al. [22] propose IFCNN, which first uses two convolutional layers to extract salient features from multiple input images, then selects appropriate fusion rules to fuse the extracted features, and reconstructs the fused image through two convolutional layers. Ma et al. [23] propose STDFusionNet, which uses salient object masks to assist fusion tasks. Considering illumination, Tang et al. [24] propose a progressive image fusion network based on illumination perception. Wang et al. [25] propose UMFusion, which generates pseudo-infrared images through a crossmodality perceptual style transfer network (CPSTN) and uses a multi-level refinement registration network (MRRN) for image registration. Finally, the feature interaction fusion module (IFM) is used to adaptively select features for fusion in the dual-path interaction fusion network (DIFN). Furthermore, transformer has shown excellent performance in the visual field due to its powerful modeling ability. Therefore, Tang et al. [19] propose YDTR, which obtains local features and important contextual information through the Y-shaped dynamic transformer module.

2.2 AE-based fusion methods

AE-based fusion methods achieve feature extraction and image reconstruction by using pre-trained autoencoders, and use manually designed fusion rules in the fusion process. Li et al. [13] propose DenseFuse. Unlike traditional convolutional networks, the encoder of DenseFuse consists of convolutional layers, fusion layers and dense blocks. Li et al. [14] propose NestFuse, which introduces a nest connection architecture and retains multi-scale feature information. Li et al. [20] propose an end-to-end fusion network architecture (RFN-Nest), using RFN to replace traditional methods.

Figure 3 The overall workflow of the proposed model. The upper part describes the architecture of the segmentation network. The middle part details the proposed stage-interactive network. The lower part outlines the structure of the fusion network.

2.3 GAN-based fusion methods

The Generative Adversarial Network (GAN) consists of a generator and a discriminator. GAN-based fusion methods use the adversarial training mechanism of the generator and the discriminator to extract features from the input image and generate the fused image. For example, Ma et al. [15] propose FusionGAN, which constructs an adversarial game mechanism between the generator and the discriminator. In addition, Ma et al. [16] propose a dual-discriminator conditional generative adversarial network (DDcGAN), which achieves the fusion of infrared and visible images with different resolutions through adversarial training between generators and two discriminators.

However, most of these existing deep learning-based image fusion methods are independent of other high-level visual tasks, such as object detection [30] and image segmentation [31]. Currently, Tang et al. [18] propose SeAFusion, which cascades the image fusion module and the semantic segmentation module, and designs a loss based on multi-task learning to constrain the fusion network. But existing multi-task learning methods are mainly applicable to tasks at the same level. As two vision tasks, image fusion and image segmentation have significant differences in feature representation, so bridging the feature gap between the two tasks is still a difficult problem. To bridge this gap, our method adopts the self-supervision idea to convert the segmentation features into pixel-level features that match the image fusion task, thereby narrowing the difference in feature representation between the two tasks.

3.1 Overview

Our SegFANet framework is shown in Figure 3, which consists of three sub-networks: the segmentation network aims to extract target-level features, while the fusion network focuses on pixel-level feature extraction and integration. In order to take advantage of the complementary advantages of multi-task learning, we introduce stage-interactive networks (SINets) between the encoder stages of the two networks. The stage-interactive network aims to assist the fusion network with the semantic information of the segmentation network to help the fusion network better understand the image content. This is achieved through 3 key modules: image reconstruction module (IRM), cross attention module (CAM) and feature fusion module (FFM).

Specifically, SegFANet conducts cross-task feature interactions between the corresponding encoder levels of the segmentation network and the fusion network through n stage-interactive networks. In each network, first, segmentation features and fusion features are extracted from the infrared and visible inputs:

where $I_{\text{ir}}$ and $I_{\text{vis}}$ denote the infrared and visible input images, $E_i^S(\cdot )$ and $E_i^F(\cdot )$ are the $i$-th encoders of the segmentation and fusion branches, $f_{e,i}^S$ and $f_{e,i}^F$ represent their corresponding encoder features.

Then, we construct IRM based on a self-supervision mechanism, which transforms the target-level semantic features $f_{e,i}^S$ from the segmentation network into pixel-level features $f_i^R$ to bridge the feature gap between image fusion and segmentation:

where $R_i(\cdot )$ denotes the function of the $i$-th IRM, which includes four "Convolution with 3×3 kernel + ReLU" layers, and $f_{e,i}^S$ denotes the segmentation feature produced by the segmentation branch's $i$-th encoder and serves as the input for IRM.

In addition, CAM takes the features $f_{e,i}^F$ and $f_i^R$ obtained by the fusion network and IRM as inputs, and interacts to obtain a new fusion feature ${\tilde{f}}_{e,i}^F$, which can be formulated as:

where $S_i(\cdot )$ denotes the function of the $i$-th CAM, $f_{e,i}^F$ is the fusion feature, and $f_i^R$ is the reconstructed feature.

Moreover, we develop the FFM, which takes the feature $f_i^R$ and the feature $f_{d,n-i+1}^F$ from the fusion decoder stage at the corresponding resolution as inputs and performs feature fusion. This process enhances the fusion network decoder's ability to understand semantic information, thereby enabling the fusion network to generate high-quality and semantically rich fusion images, which can be formulated as:

where $F_i(\cdot )$ denotes the function of the $i$-th FFM. In the following subsections, we introduce the detailed architectures of CAM and FFM, respectively. In addition, we elaborate on the design of the loss function, which plays a crucial role in guiding the model to improve image fusion quality.

3.2 Cross Attention Module

Figure 4 The structure of the cross attention module.

The specific structure of the cross attention module (CAM) is shown in Figure 4. In this module, the input feature $f_{e,i}^F$ is first processed by adaptive average pooling to compress the spatial dimension, and then the Query Q and Value V are generated through the batch normalization layer. At the same time, the same adaptive average pooling and batch normalization operations are performed on another input feature $f_i^R$ to generate a Key K, which provides a clear semantic prior. By calculating the similarity between K and Q, the fused feature is guided to focus on the target area. Then the softmax function is applied for normalization to generate the attention weight. Finally, the attention weight is used to compute a weighted sum of the V, completing the feature fusion operation. The interacted feature is concatenated with the original fused feature $f_{e,i}^F$ to obtain the final fused feature ${\tilde{f}}_{e,i}^F$. The concatenated feature ${\tilde{f}}_{e,i}^F$ is then directly fed into the convolutional layer of the next stage in the image fusion encoder as its input. This process can be defined as:

where $f_{e,i}^F$ denotes the output feature of the i-th stage of the fusion network encoder and $f_i^R$ denotes the reconstructed feature of the i-th IRM. $\text{BN}(\cdot )$ represents the batch normalization operation, which stabilizes training and accelerates convergence. $\text{AAP}(\cdot )$ denotes the adaptive average pooling operation, which dynamically adjusts the size of the feature map. $\text{Concat}(\cdot )$ represents feature concatenation operation, used to preserve more information. Additionally, softmax is used to normalize scores, ensuring that the sum of weights is 1, which is typically applied in attention mechanisms.

3.3 Feature Fusion Module

As shown in Figure 5, the feature fusion module (FFM) is mainly composed of convolutional layers and the activation function is ReLU, which is designed to enhance the fusion network decoder's ability to understand semantic information. Specifically, the reconstructed feature $f_i^R$ is firstly encoded with double $3\times 3$ convolutions and concatenated with the fusion network decoder feature $f_{d,n-i+1}^F$. The concatenated features are further deepened and fused through two $3\times 3$ convolutional layers. This process can be represented as:

where ${\text{Conv}}_{3\times 3}$ denotes the operation of a 3×3 convolutional layer, mainly used for feature extraction of input features.

Subsequently, the deepened features are concatenated with the original $f_{d,n-i+1}^F$ again, and finally the number of channels is adjusted through a $1\times 1$ convolutional layer and output:

where ${\text{Conv}}_{1\times 1}$ denotes a 1×1 convolutional layer operation, mainly used to adjust the number of feature channels.

Figure 5 The structure of the feature fusion module.

3.4 Loss function

To optimize the proposed model, we design a loss function. It converts semantic-level segmentation features into pixel-level features via IRM, bridging the feature gap between segmentation and fusion, enabling their interaction, and improving fusion quality. Specifically, we jointly train fusion, segmentation and reconstruction tasks. Therefore, the designed loss function can be expressed as:

where $L_f$ and $L_s$ represent the image fusion loss and segmentation loss, respectively. And $L_{\text{rec}}$ represents the reconstruction loss of segmentation semantic features.

In the fusion stage, the fusion loss is defined as:

where SSIM [26] represents the structural similarity index, which is used to evaluate the difference between the fusion result $I_F$ and the source images $I_{\text{ir}}$ and $I_{\text{vis}}^Y$.

The reconstruction loss mainly evaluates the similarity between the reconstructed image and the original infrared and visible images. The $L_{\text{rec}}$ is defined as:

where $L_{\text{rec},i}$ represents the reconstruction loss for the $i$-th image reconstruction module, and $I_{\text{res}i}$ is the feature output by the $i$-th image reconstruction module.

In the segmentation stage, the segmentation loss function $L_s$ is composed of the cross-entropy (CE) loss [29] and the dice coefficient loss [27, 28]:

where CE loss is used to measure the difference between the predicted probabilities and the true labels, providing a measure of classification accuracy. The dice coefficient loss, on the other hand, evaluates the similarity between the predicted and true segmentations, making it particularly useful for tasks where the goal is to match the predicted segmentation with the true segmentation.

4.1 Experimental configurations

The model is implemented with PyTorch on GTX 2080TI GPU. During training, we employ the Adam optimizer with a learning rate of 0.0001. The two momentum values of the Adam optimizer are set to 0.9 and 0.999, respectively. The batch size is set to 2, and the number of epochs is set to 50. We train the model on the WHU [44] and Potsdam [45]. The Potsdam dataset provides detailed information about urban environments, mainly including 6 categories: Impervious surfaces, Buildings, Low vegetation, Trees, Cars, and Clutter. It is divided into a training set of 10,830 images and a test set of 2,527 images. The WHU dataset describes the scenario of land, covering 7 categories: Farmland, City, Village, Water, Forest, Road, and Others [46]. It is divided into a training set of 17,280 images and a test set of 4,320 images. Before training, we preprocess the data by cropping the images into patches of size 320×320.

In addition, for quantitative evaluation, four metrics are selected to objectively evaluate the fusion performance, including spatial frequency (SF) [40], average gradient (AG) [41], the sum of the correlations of differences (SCD) [10], and visual information fidelity (VIF) [12]. SF measures the richness of detail information in the image. AG reflects the clarity of the image. SCD reflects the degree of correlation between the information transferred to the fused image and the corresponding source image. VIF measures the degree of visual information preservation of the fused image relative to the source image from the perspective of human visual perception. The larger the SF, AG, SCD and VIF of the fusion algorithm, the better the fusion performance.

Figure 6 Visual comparison of our method with five SOTA fusion methods on the WHU dataset. (a1-h1) and (a2-h2) represent visible image, infrared image, UMFusion, LiMFusion, ITFuse, Tardal, YDTR and our proposed model, respectively.

4.2 Results and analysis

In this section, we conduct subjective qualitative and objective quantitative experiments on the WHU and Potsdam datasets to evaluate the performance and advantages of our proposed fusion method. We select five state-of-the-art methods, including UMFusion [25], YDTR [19], Tardal [17], ITFuse [42] and LiMFusion [43], to compare with the proposed model. Next, we conduct a detailed analysis of the fusion results obtained by these methods on the WHU and Potsdam datasets from both subjective and objective dimensions.

4.2.1 Experimental results on the WHU dataset

First, we qualitatively compare the proposed method with five comparison methods. We select two representative infrared and visible images from the WHU dataset for subjective evaluation, as shown in Figure 6. In the picture, in order to visually compare the fusion effects of different methods, we mark the comparison area with a yellow box, and enlarge the details of the corresponding area and display it in the lower left corner of the image. From the visualization results, it can be observed that the fusion images generated by ITFuse and LiMFusion exhibit lower clarity, with blurred edges and loss of some detail information. Although UMFusion and YDTR have fused infrared and visible information to a certain extent, there is still room for improvement in detail preservation and clarity. Tardal has a high contrast and highlights the infrared thermal radiation target well, but there are local artifacts, and the preservation of texture and edge detail information is not as good as the proposed method. In contrast, the proposed method performs better in integrating the complementary features of infrared and visible images, and the generated fused images have higher clarity, can well preserve edge and texture detail information, and are more in line with the characteristics of the human visual system. Therefore, in the qualitative comparison of infrared and visible image fusion methods, the visualization results of the proposed method outperform those of existing state-of-the-art methods.

Figure 7 Visual comparison of our method with five SOTA fusion methods on the Potsdam dataset. (a1-h1) and (a2-h2) represent visible image, infrared image, UMFusion, LiMFusion, ITFuse, Tardal, YDTR, and our proposed model, respectively.

In addition, to comprehensively evaluate the performance of the proposed method and five comparison methods, four metrics, SF, AG, SCD and VIF are used to quantitatively analyze the fused image. As shown in Table 1, the average comparison results of the four metrics of the proposed method and other comparison methods on the WHU test set are shown, where the optimal value of each metric is marked in red and the suboptimal value is marked in blue. Obviously, the proposed method is higher than the existing comparison methods in the three evaluation metrics of AG, SCD and VIF, which shows that the fused image generated by the proposed method has the best performance in clarity and retains more feature information of the source image, with better visual performance. Although the proposed method is not the best in the metric of spatial frequency (SF), it is second only to LiMFusion, and the gap is not large, which shows that the fused image generated by the proposed method contains richer texture and edge detail information.

Table 1 Average quality metrics of different methods on the WHU dataset. The optimal result is highlighted in red, and the sub-optimal result is highlighted in blue.

Methods	SF	AG	SCD	VIF
YDTR [19]	14.365	5.01	0.829	1.015
Tardal [17]	15.465	5.867	1.033	0.796
UMFusion [25]	13.4735	4.8998	0.8378	1.0329
ITFuse [42]	6.6021	2.8736	0.2886	0.6699
LiMFusion [43]	17.4743	5.7829	0.6069	0.5147
Ours	16.8724	5.9824	1.3197	1.0969

4.3 Ablation study

4.3.2 Effect of image reconstruction module

The role of the image reconstruction module (IRM) is to align the target-level features of the segmentation network with the pixel-level features of the image fusion task, thereby bridging the feature gap between image fusion and image segmentation. In this section, to fully verify the effectiveness of the image reconstruction module, we design a series of comparative experiments. Firstly, for the fusion model containing two stage-interactive networks, we conduct two experiments: one retains the image reconstruction module and the other removes the module while keeping other structures unchanged, in order to observe the specific impact of the image reconstruction module on the quality of the fused image.

In addition, to further explore the performance of the image reconstruction module under different configurations, we also add a set of experiments to compare the performance difference between retaining and deleting the image reconstruction module in a fusion model that includes a three stage-interactive networks. The experimental results of the objective evaluation metrics are shown in Table 4.

Table 4 Effect study of image reconstruction module (IRM). The optimal result is bolded. And w/ means with, w/o means without.

Setting	SF	AG	SCD	VIF
Two-SINets (w/ IRM)	16.8693	5.9798	1.3182	1.0971
Two-SINets (w/o IRM)	16.8419	5.9713	1.3179	1.0967
Three-SINets (w/ IRM)	16.8724	5.9824	1.3197	1.0969
Three-SINets (w/o IRM)	16.8317	5.9695	1.3189	1.0964

Figure 8 Visual comparison of feature alignment results before and after IRM processing. (a1-e1) and (a2-e2) represent visible image, infrared image, segmentation feature, reconstructed feature, and fusion feature.

As shown in Table 4, by comparing the results of the first two experiments, it can be seen that the setting including the image reconstruction module is better than the setting without the module in all metrics, proving that the image reconstruction module can improve the quality of the fused image. Similarly, the comparison of the results of the last two experiments also verifies this point, further proving the effect of the image reconstruction module.

Moreover, we validate the alignment effect of the image reconstruction module by visualizing intermediate features (comparing segmentation features, reconstructed features, and fusion features). As shown in Figure 8, the segmentation features before alignment lack pixel-level detail information (e.g., the edge contours of farmland are blurred). In contrast, after adding the image reconstruction module, the boundaries between farmland and water are clearly presented, and semantic-level features are reconstructed into pixel-level features suitable for image fusion, achieving accurate feature alignment.

4.3.3 Effect of cross attention module

The cross attention module (CAM) is responsible for promoting the interaction of features between image segmentation and image fusion, thereby helping the fusion task to better fuse salient targets during the fusion process to improve the quality of the fused image. In this section, in order to fully verify the effect of the cross attention module, we design two sets of comparative experiments. Firstly, for the fusion model containing two stage-interactive networks, we conduct two experiments: one retains the cross attention module, and the other replaces the cross attention module with a simple feature addition operation. Secondly, we also compare the performance differences when retaining the cross attention module and replacing the cross attention module with a feature addition operation in the context of the three stage-interactive networks. The experimental results are shown in Table 5.

Table 5 Effect study of cross attention module (CAM). The optimal result is bolded. And w/ means with, w/o means without.

Setting	SF	AG	SCD	VIF
Two-SINets (w/ CAM)	16.8693	5.9798	1.3182	1.0971
Two-SINets (w/o CAM)	16.8699	5.9786	1.317	1.0966
Three-SINets (w/ CAM)	16.8724	5.9824	1.3197	1.0969
Three-SINets (w/o CAM)	16.8684	5.9815	1.3206	1.0969

As shown in Table 5, the results show that whether it is a two stage-interactive or a three stage-interactive, the introduction of the cross attention module is more helpful in improving the quality of the fused image than the simple feature addition operation.

To further validate the effectiveness of the CAM module, we visualize the distribution of attention weights using heatmaps (see Figure 9). As observed from the heatmaps, attention weights are predominantly focused on target regions (e.g., forest), which enables the segmentation task to effectively assist the fusion task and thereby enhancing the quality of image fusion.

Figure 9 Qualitative analysis of attention weights. (a1-c1) and (a2-c2) represent visible image, infrared image, and heatmap.