A feature level image fusion for Night-Vision context enhancement using Arithmetic optimization algorithm based image segmentation

doi:10.1016/j.eswa.2022.118272

Expert Systems with Applications

Volume 209, 15 December 2022, 118272

https://doi.org/10.1016/j.eswa.2022.118272 Get rights and content

Highlights

•
It is proposed an efficient method for image Fusion.
•
Weight maps are generated from segmentation using Arithmetic optimization algorithm.
•
The weight maps are optimized using a weighted least square (WLS) technique.
•
The quality of the fused results is verified using the ( $Q^{AB / F}$ ), SCD, SSIM, and $N^{A B / F}$ .
•
The quality of the fused results is better than other competitor algorithms.

Abstract

Images are fused to produce a composite image by combining key characteristics of the source images in image fusion. It makes the fused image better for human vision and machine vision. A novel procedure of Infrared (IR) and Visible (Vis) image fusion is proposed in this manuscript. The main challenges of feature level image fusion are that it will introduce artifacts and noise in the fused image. To preserve the meaningful information without adding artifacts from the source input images, weight map computed from Arithmetic optimization algorithm (AOA) is used for the image fusion process. In this manuscript, feature level fusion is performed after refining the weight maps using a weighted least square optimization (WLS) technique. Through this, the derived salient object details are merged into the visual image without introducing distortion. To affirm the validity of the proposed methodology simulation results are carried for twenty-one image data sets. It is concluded from the qualitative and quantitative experimental analysis that the proposed method works well for most of the image data sets and shows better performance than certain traditional existing models.

Introduction

Owing to weak night-time illumination conditions, visible images are mostly fused with the accompanying infrared (IR) pictures to improve the background of cinematic sequences. A hybrid approach using local Laplacian filter for edge preserving and enhancement along with segmentation-based weight map fusion approach has been proposed. It enhances the night vision context of infrared and visible image fusion and also keeps edges intact without adding any artifacts.

In recent times, the detectability of military targets has reduced substantially with defence system (Meher et al., 2022, James and Kavitha, 2014). The military specifications under those circumstances have stimulated the improvement of multi-mode image fusion technology, which typically uses IR and visible images to obtain complementary information. The IR image tracks heat energy emitted from objects in the scene and can be used to discover objectives because it has a hot contrast, whilst the visible picture has considerably more high-frequency background information, which is important if the target positions and circumstances are to be accurately identified.

The improved image processing system performance has led to the development of several image fusion techniques which combine information gathered through the various sensors (Jiang et al., 2018, Pan et al., 2017, Piella, 2003). The goal of image fusion is to combine the proportion able attributes of the images acquired with the best visual effects in a fused image. This fused image generates details which cannot be obtained by separately analysing various images. Other cameras such as infrared cameras, which capture images in different wavelengths, are often favoured along with a digital charge-coupled device (CCD) camera (Li & Yang, 2008). Using different imaging sensors to capture images for the same location helps to get enhanced outputs once they are fused (Singh, Singh, Gehlot, kaur, & Gagandeep, 2022). Multi-modal image fusion is a combination of data acquired from different sensors (Singh, 2013, Singh, 2020), so that the information finally generated has lesser uncertainties and more information as compared to the individual performance of each sensor (Li and Wang, 2022, Liu and Wang, 2015, Xu et al., 2020, Zhang et al., 2021). In order to perform segmentation, OTSU as an objective function to perform segmentation is discussed in the forthcoming section 2. Section 3 conferred fusion methodology AOA algorithm and proposed modified algorithm AOA. Result analysis along with performance metric are discussed section 4 and section 5 presents conclusion.

Section snippets

Related work

It is an important preliminary task which plays a crucial role in the field of medicine and other computer vision applications. It is a hot research topic gaining rapid pace in recent years. In region based fusion image thresholding provides significant support (Singh et al., 2020, Singh et al., 2021a), (Kaur & Singh, 2017). Some of the approaches to image segmentation are addressed in subsequent sections.

IR and visible image fusion methodology

In this section, a novel segmentation based fusion methodology is proposed. The objective behind the proposed technique is to fuse the Infrared $I_{IR} (x, y)$ and Visible $I_{Vis} (x, y)$ images without introducing any artefacts and keeping meaningful information intact. The detailed block diagram of proposed manuscript with appropriated images is shown in Fig. 1. Each step involved in this algorithm is described in depth in forthcoming sub-sections. In which, obtain the segmented image $I_{S} (x, y)$ using

Experimental results

The final fusion results of the proposed approach are compared with seven existing state-of-the-art fusion techniques. CCD cameras with low-light sensitivity or a standard sensitivity can capture the input visible images. Some image details in both visible and infrared imagery may have to be boosted to maximise their visibility. In addition, the infrared sensor frequently uses the mid-wave and long-wave spectral bands to better identify details from objects in dark and obstructed areas. We can

Discussion and conclusion

A novel feature level image fusion algorithm using segmentation based on AOA is proposed in this manuscript. Firstly, using AOA based segmentation, the IR image is segmented into different groups. These weights map functions are refined using WLS optimization. The resulting segmented image is used for measuring weight map function used in the process of fusion. Finally, the fused image is reconstructed using pixel-wise weighted average fusion. The efficiency of the approach proposed is measured

CRediT authorship contribution statement

Simrandeep Singh: Conceptualization, Methodology, Software. Harbinder Singh: Data curation, Writing – original draft. Nitin Mittal: Investigation, Visualization. Abdelazim G. Hussien: Writing – review & editing. Filip Sroubek: Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (48)

L. Abualigah et al.
The arithmetic optimization algorithm
Computer Methods in Applied Mechanics and Engineering
(2021)
A.K. Bhandari et al.
A context sensitive energy thresholding based 3D Otsu function for image segmentation using human learning optimization
Applied Soft Computing Journal
(2019)
V.K. Bohat et al.
A new heuristic for multilevel thresholding of images
Expert Systems with Applications
(2019)
M. Kamel et al.
Extraction of binary character/graphics images from grayscale document images
CVGIP: Graphical Models and Image Processing
(1993)
J. Li et al.
Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation
Information Fusion
(2022)
S. Li et al.
Multifocus image fusion by combining curvelet and wavelet transform
Pattern Recognition Letters
(2008)
C.H. Liu et al.
Infrared and visible image fusion method based on saliency detection in sparse domain
Infrared Physics and Technology
(2017)
Y. Liu et al.
Multi-focus image fusion with a deep convolutional neural network
Information Fusion
(2017)
Y. Liu et al.
Deep learning for pixel-level image fusion: Recent advances and future prospects
Information Fusion
(2018)
B. Meher et al.
Visible and infrared image fusion using an efficient adaptive transition region extraction technique
Engineering Science and Technology, an International Journal
(2022)

V.S. Petrović et al.

Sensor noise effects on signal-level image fusion performance

Information Fusion

(2003)

G. Piella

A general framework for multiresolution image fusion: From pixels to regions

Information Fusion

(2003)

H. Zhang et al.

Image fusion meets deep learning: A survey and perspective

Information Fusion

(2021)

Alexander, T. (2014). TNO Image Fusion Dataset. Retrieved from...

D.P. Bavirisetti et al.

Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform

IEEE Sensors Journal

(2016)

A.P. James et al.

Mean-variance blind noise estimation for CT images

Advances in Intelligent Systems and Computing

(2014)

Q. Jiang et al.

Multi-sensor image fusion based on interval type-2 fuzzy sets and regional features in nonsubsampled shearlet transform domain

IEEE Sensors Journal

(2018)

Kaur, R., & Singh, S. (2017). An artificial neural network based approach to calculate BER in CDMA for multiuser...

S. Krishnamoorthy et al.

Implementation and comparative study of image fusion algorithms

International Journal of Computer Applications

(2010)

Li, H., X.-J. Wu. (2018). Infrared and visible image fusion using Latent Low-Rank Representation. ArXiv Preprint...

H. Liang et al.

Modified grasshopper algorithm-based multilevel thresholding for color image segmentation

IEEE Access

(2019)

Y. Liu et al.

Image fusion with convolutional sparse representation

IEEE Signal Processing Letters

(2016)

Y. Liu et al.

Simultaneous image fusion and denoising with adaptive sparse representation

IET Image Processing

(2015)

B. Meher et al.

A survey on region based image fusion methods

Information Fusion

(2019)

Cited by (33)

Triple disentangled network with dual attention for remote sensing image fusion
2024, Expert Systems with Applications
Remote sensing applications, such as detection and recognition of objects, need a large number of high spatial resolution (HR) images. As a feasible technique, image fusion is considered to produce HR images. However, most of remote sensing image fusion methods based on deep neural network (DNN) are mainly limited by the extraction of spatial and spectral features. These methods ignore the redundancy among these features, which causes spatial and spectral distortions in the fused image. In this paper, we propose a novel image fusion method based on a triple disentangled network (TDNet) with dual attention to reduce the redundancy in the extracted features. In the proposed method, it is assumed that the information in panchromatic (PAN) and low spatial resolution multispectral (LRMS) images can be encoded as the spatial, spectral, and common features. Specifically, we construct a triple-stream network to extract these features. To efficiently model the spatial and spectral information in PAN and LRMS images, local–global attention and interdependency attention are designed and integrated into the network. Then, the redundancy among these features is reduced by disentangled learning, in which these features are recombined to reconstruct the PAN and LRMS images. Besides, these features should complement each other. So, we utilize the maximal coding rate reduction to balance the redundancy and complementarity among them. Finally, all features are recombined to synthesize the high spatial resolution multispectral image. The experimental results demonstrate that the proposed TDNet has a superior performance in terms of qualitative and quantitative evaluations. The code link is https://github.com/RSMagneto/TDNet.
Partial reinforcement optimizer: An evolutionary optimization algorithm[Formula presented]
2024, Expert Systems with Applications
In this paper, a novel evolutionary optimization algorithm, named Partial Reinforcement Optimizer (PRO), is introduced. The major idea behind the PRO comes from a psychological theory in evolutionary learning and training called the partial reinforcement effect (PRE) theory. According to the PRE theory, a learner is intermittently reinforced to learn or strengthen a specific behavior during the learning and training process. The reinforcement patterns significantly impact the response rate and strength of the learner during a reinforcement schedule, achieved by appropriately selecting a reinforcement behavior and the time of applying reinforcement process. In the PRO algorithm, the PRE theory is mathematically modeled to an evolutionary optimization algorithm for solving global optimization problems. The efficiency of the proposed PRO algorithm is compared to well-known Meta-heuristic Algorithms (MAs) using Wilcoxon and Friedman statistical tests to analyze results from 75 benchmarks of the CEC2005, CEC2014, and CEC-BC-2017 test suits, which include unimodal, multimodal, hybrid, and composition functions. Additionally, the proposed PRO algorithm is applied to optimize a Federated Deep Learning Electrocardiography (ECG) classifier, as a real case study, to investigate the robustness and applicability of the proposed PRO. The experimental results demonstrate that the PRO algorithm outperforms existing meta-heuristic optimization algorithms by providing a more accurate and robust solution.
Multiscale cyclic frequency demodulation-based feature fusion framework for multi-sensor driven gearbox intelligent fault detection
2024, Knowledge-Based Systems
Accurate fault detection is extremely important to ensure stable gearbox operation. Data-driven schemes using cyclic spectral have received significant attention due to their robust demodulation performance. However, these schemes are mainly applied to process single sensor signals, and they are unable to accurately obtain precise fault features. This paper proposed a novel multiscale cyclic frequency demodulation (MCFD)-based feature fusion framework for multi-sensor driven gearbox intelligent fault diagnosis. Firstly, the MCFD is proposed to analyze the vibration signals from multi-sensor driven gearbox, which acquires the multi-sensor mode information without setting parameters in advance. Thereafter, the grey relational degree between the multi-sensor mode information and original signal is calculated, and its results are normalized to obtain the relationship coefficients. Finally, the acquired coefficients are performed for multi-sensor information fusion to form the covariance matrix for gearbox fault diagnosis. The effectiveness of the proposed feature fusion framework is validated using the gearbox case. The comparative experiments indicate that this framework outperforms comparative algorithms for multi-sensor driven gearbox fault diagnosis.
Novel hybrid of AOA-BSA with double adaptive and random spare for global optimization and engineering problems
2023, Alexandria Engineering Journal
Archimedes Optimization Algorithm (AOA) is a new physics-based optimizer that simulates Archimedes principles. AOA has been used in a variety of real-world applications because of potential properties such as a limited number of control parameters, adaptability, and changing the set of solutions to prevent being trapped in local optima. Despite the wide acceptance of AOA, it has some drawbacks, such as the assumption that individuals modify their locations depending on altered densities, volumes, and accelerations. This causes various shortcomings such as stagnation into local optimal regions, low diversity of the population, weakness of exploitation phase, and slow convergence curve. Thus, the exploitation of a specific local region in the conventional AOA may be examined to achieve a balance between exploitation and exploration capabilities in the AOA. The bird Swarm Algorithm (BSA) has an efficient exploitation strategy and a strong ability of search process. In this study, a hybrid optimizer called AOA-BSA is proposed to overcome the limitations of AOA by replacing its exploitation phase with a BSA exploitation one. Moreover, a transition operator is used to have a high balance between exploration and exploitation. To test and examine the AOA-BSA performance, in the first experimental series, 29 unconstrained functions from CEC2017 have been used whereas the series of the second experiments use seven constrained engineering problems to test the AOA-BSA’s ability in handling unconstrained issues. The performance of the suggested algorithm is compared with 10 optimizers. These are the original algorithms and 8 other algorithms. The first experiment’s results show the effectiveness of the AOA-BSA in optimizing the functions of the CEC2017 test suite. AOABSA outperforms the other metaheuristic algorithms compared with it across 16 functions. The results of AOABSA are statically validated using Wilcoxon Rank sum. The AOABSA shows superior convergence capability. This is due to the added power to the AOA by the integration with BSA to balance exploration and exploitation. This is not only seen in the faster convergence achieved by the AOABSA, but also in the optimal solutions found by the search process. For further validation of the AOABSA, an extensive statistical analysis is performed during the search process by recording the ratios of the exploration and exploitation. For engineering problems, AOABSA achieves competitive results compared with other algorithms. the convergence curve of the AOABSA reaches the lowest values of the problem. It also has the minimum standard deviation which indicates the robustness of the algorithm in solving these problems. Also, it obtained competitive results compared with other counterparts algorithms regarding the values of the problem variables and convergence behavior that reaches the best minimum values.
A review of image fusion: Methods, applications and performance metrics
2023, Digital Signal Processing: A Review Journal
The same sensor or a number of image sensors are used to take a series of photographs in order to gather as much data as possible about the scene. Several imaging techniques are used to retrieve entire information from the source under observation. Image fusion (IF) is used to create a new image that incorporates comprehensive information from many photographs. The various images may be captured from different viewpoints, different imaging sensors i.e., visible (VIS) and IR camera, different modalities i.e., computed tomography (CT) and magnetic resonance image (MRI), hyper spectral images i.e., panchromatic and multi-spectral satellite images, multi-exposure images and multi-focus images. Owing to the growing mandates and development of image enhancement schemes, numerous fusion methods were recently formulated. Consequentially, we are doing a survey study to document the methodological development in IF techniques. The outline of picture merging technologies is described in this article. Ultimately, latest state-of-the-art fusion techniques are also demonstrated. Readers will gain insights on current discoveries and their implications for the future through a review of diverse image fusion in various areas and fusion quality metrics.
A novel Q-learning-based hybrid algorithm for the optimal offloading and scheduling in mobile edge computing environments
2023, Journal of Network and Computer Applications
Mobile Edge Computing (MEC) has arisen as a promising computing paradigm consisting of three tiers: Smart Mobile Devices (SMDs), fog nodes, and the cloud. The MEC enables computational offloading and execution schedules to cope with the problems of insufficient resources for the SMDs and the computational tasks' deadlines. The offloading problem determines in what order and source of the network the tasks should be performed to minimize execution time and power consumption. The main aim of the current paper is to reduce execution time and energy consumption by optimizing tasks' offloading and scheduling in MEC networks. As a result, the task scheduling and offloading are modeled as an optimization problem. Then, an enhanced hybridization of Artificial Ecosystem-based Optimization (AEO) and Arithmetic Optimization Algorithm (AOA), named E-AEO-AOA, is presented to optimize it. In the E-AEO-AOA, the AOA and AEO algorithms are initially discretized. Next, the Q-learning strategy is modified and recruited to hybridize the algorithms in a complementary manner. Subsequently, chaos theory is utilized in a local search procedure to enhance the exploitation capability of the E-AEO-AOA. Eventually, the performance of E-AEO-AOA is examined on fifteen MEC networks. In the experiments, the E-AEO-AOA is compared with AEO, AO, AOA, JS, MRFO, STOA, SCA, and TSA algorithms statistically. Besides, the algorithms' convergence rate and solutions dispersity are visually compared. Moreover, the algorithms are compared by the Wilcoxon signed-rank test. The experimental results indicate that the E-AEO-AOA surpassed competitor algorithms in 90% of cases. Likewise, in 6% of the cases, the E-AEO-AOA produced the same results as AEO, AOA and MRFO.

View all citing articles on Scopus

View full text

A feature level image fusion for Night-Vision context enhancement using Arithmetic optimization algorithm based image segmentation

Highlights

Abstract

Introduction

Section snippets

Related work

IR and visible image fusion methodology

Experimental results

Discussion and conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Computer Methods in Applied Mechanics and Engineering

Applied Soft Computing Journal

Expert Systems with Applications

CVGIP: Graphical Models and Image Processing

Information Fusion

Pattern Recognition Letters

Infrared Physics and Technology

Information Fusion

Information Fusion

Engineering Science and Technology, an International Journal

Information Fusion

Information Fusion

Information Fusion

Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform

IEEE Sensors Journal

Mean-variance blind noise estimation for CT images

Advances in Intelligent Systems and Computing

Multi-sensor image fusion based on interval type-2 fuzzy sets and regional features in nonsubsampled shearlet transform domain

IEEE Sensors Journal

Implementation and comparative study of image fusion algorithms

International Journal of Computer Applications

Modified grasshopper algorithm-based multilevel thresholding for color image segmentation

IEEE Access

Image fusion with convolutional sparse representation

IEEE Signal Processing Letters

Simultaneous image fusion and denoising with adaptive sparse representation

IET Image Processing

A survey on region based image fusion methods

Information Fusion