1 Introduction

At present, energy plays a pivotal role in improving the quality of human life and driving economic progress in every nation [1]. It is widely recognized that developing countries, especially rural areas, face significant challenges in accessing sufficient energy. Despite the global emphasis on employing clean energy sources to mitigate the emissions responsible for climate change, fossil fuels continue to dominate the production of electricity.

In areas that face electricity challenges, particularly remote or rural regions where extending the conventional power grid is difficult due to technical limitations or high construction expenses, a hybrid grid presents a more viable solution for electricity provision. This hybrid grid integrates renewable and conventional energy systems, energy storage units, and AC/DC loads, forming what is commonly referred to as microgrid systems [2]. The optimization of microgrid planning involves considering factors such as customer demands, reliability requirements, and economic considerations [2]. The objective of optimization is to identify the most economically beneficial or efficient alternative among various feasible options, taking into account the given constraints. Consequently, recent research has focused extensively on developing optimization tools, techniques, and applications [3].

Many researchers have employed various methodologies to address significant issues regarding the optimal design and sizing of components in hybrid systems. These methodologies include the use of simulation tools such as Hybrid Optimization Model for Electric Renewable (HOMER), HYBRID2, INSEL, General Algebraic Modeling System (GAMS), TRNSYS, RETSCREEN, Remote Area Power Supply Simulator (RAPSIM), SOMES, among others [4,5,6]. Additionally, optimization algorithms such as Differential evolution algorithm (DE), Genetic algorithms (GA), Simulated annealing (SA), Particle swarm optimization (PSO), Ant colony algorithm (ACS), and others [3] have been utilized.

In [7], a research paper discussed a study conducted on a hybrid power system consisting of photovoltaic (PV), diesel, and battery components in the remote Saharan village of Tiberkatine in southern Algeria. The main objective of this study was to simultaneously reduce the total system cost, unsatisfied load, and CO2 emissions by employing the PSO method. The results obtained from the PSO technique were compared with those from the HOMER program, revealing that PSO yielded superior cost-related outcomes compared to HOMER.

In a separate study described in [8], the authors investigated a power control strategy for an off-grid hybrid system combining PV, wind, and lead-acid battery technologies. They introduced a novel stochastic search algorithm based on the shark smell optimization (ESSO) algorithm. In [9], a novel method has been proposed for the optimal design of a hybrid system comprising PV, diesel, and battery units in the rural community of Ilamane in southern Algeria. The study addresses multi-objective optimization problems, including the total cost of the system, loss of load probability (LLP), and greenhouse gas (GHG) emissions. The approach utilizes the PSO technique.

In a separate paper, [10] introduces a new optimization approach for designing a hybrid PV, diesel, and battery system in the Gobi Desert, China. This study focuses on addressing multi-objective optimization problems, specifically LLP, CO2 emissions, and the system’s total annualized cost (TAC). The proposed method employs the converged Elephant Herd Optimization Algorithm (cEHO). To assess the algorithm’s effectiveness, the results obtained from the proposed system are compared with those obtained from the HOMER software and the PSO technique.

In [11], the researchers carried out simulations and optimizations of different configurations for hybrid energy systems utilizing PV units, wind turbines, and battery units. The objective was to minimize the net present cost (NPC) and the energy cost (COE) in the Yamunanagar district, a rural area in the State of Haryana, India. The HOMER software was employed for essential modeling, simulation, financial evaluation, and optimal sizing. In [12], a study was conducted in Pulau Banggi and Tanjung Labian, Malaysia, to investigate the impact of integrating PV units into small hybrid networks. Among the various suggested configurations, the hybrid PV/diesel/battery system demonstrated the best performance in terms of technical aspects and supported reliable daily power access.

Authors in reference [13] proposed a design for optimizing the cost of a PV/diesel/battery system in Yarkant, Xinjiang Uyghur Autonomous Region of China. To address the optimization problem, they employed an improved Henrygas solubility optimizer. The effectiveness of this optimizer strategy was demonstrated by comparing its simulation results with those obtained from HOMER and the PSO approach. Authors in [14], proposed an optimal configuration analysis using HOMER Pro software for a hybrid renewable system made up of solar panels, biomass, and batteries. This proposed system is designed, simulated, and modeled to meet the energy demands of a house in a remote area of Ecuador’s province of Guayas. Comparisons are made between the best configuration in terms of implementation based on the NPC, the COE, and the initial capital cost.

In order to meet the electricity demand of a village in Xuzhou, east China, different configuration analysis of a PV/biogas/diesel/battery hybrid system integrated with a battery storage has been carried out in Ref [15]. The analyses were conducted using HOMER software, and the results showed that the PV/BG/battery hybrid system was the most economically viable system. In [16] proposed a study to cover the demand of the 770 conventional houses of a residential area in the rural region of Punjab, India with renewable energy resources. In order to match the demand, an off-grid hybrid PV/biomass/battery system combination has been evaluated. The techno-economic-environmental analysis has been conducted using HOMER software.

Authors in [17] explored the economic feasibility for two isolated hybrid system scenarios, the first one based on the connection of PV and Battery and the second scenario is PV/Biomass/Battery based hybrid system. The suggested hybrid systems are considered for an un-electrified village located in Indian state of West Bengal. The optimal sizes and the system NPC have been investigated using the discrete gray wolf optimization (DGWO) algorithm. According to the results, a PV/Biomass/Battery based hybrid system scenario produced the lowest NPC and COE. In [18] introduced the optimum size of a grid-connected system based on the configuration of PV/biomass gasifier/battery units for a small village in India. The artificial bee colony algorithm has been applied to evaluate the techno-economic analysis and the optimum size of the suggested microgrid.

In reference [19], a novel optimization technique called the improved Archimedes optimization algorithm (IAOA) was introduced to design a hybrid power system in Farafra Oasis, Egypt. The system consisted of PV modules, a wind system, a diesel generator, and battery units. Among various system configurations, the hybrid PV/wind/diesel/battery setup demonstrated the lowest COE and the highest efficiency. Table 1 provides an overview of relevant literature references, showcasing different hybrid system configurations based on the biomass system.

Table 1 An overview of various hybrid system arrangements based on the biomass system

This study focuses on assessing the economic factors associated with an off-grid hybrid PV/biomass/battery system designed to fulfill the energy needs of a rural area. The primary objective is to utilize a newly developed algorithm called modified Firebug Swarm Algorithm (mFSO), which effectively reduces the size of the hybrid system while ensuring it can meet the load demand at the lowest COE. The Firebug swarm algorithm was specifically chosen due to its Bio-inspired behavior which employs the attraction behavior of firebugs toward better positions to discover new solutions, exhibits fast convergence rates compared to other metaheuristic search algorithms which makes it suitable for time-sensitive applications or scenarios where computational resources are limited. Additionally, FSO has shown resilience in noisy or uncertain environments and can be easily adapted or extended to incorporate problem-specific constraints and objectives. It can handle various types of optimization problems, including continuous, discrete, and constrained optimization, making it versatile in different domains.

The main goal is to compare the performance of this modified algorithm with other metaheuristic optimization algorithms, specifically evaluating their accuracy and convergence rate. The key contributions of this paper are as follows:

  • A novel modified optimization algorithm dubbed mFSO has developed in order to overcome the drawbacks of the original Firebug Swarm Optimizer (FSO), the FSO algorithm suffers from limitations in exploration and an imbalanced exploitation-exploration trade-off, leading to local optima and hindering its ability to find optimal solutions.

  • Assesse the effectiveness of the suggested new mFSO algorithm through various tests including the Wilcoxon test, boxplot analysis, and evaluating its performance on ten benchmark functions from the CEC2020 benchmark. The mFSO performance is compared with other recent optimization algorithms such as the original FSO, Slime mold algorithm (SMA), Seagull optimization algorithm (SOA), Harris Hawks optimization algorithm (HHO), Chernobyl disaster optimizer (CDO), WOA, Prairie Dog Optimization Algorithm (PDO), COVIDOA, and SCA.

  • Implementing the modified mFSO method in solving an engineering application, specifically in determining the optimal size of a standalone PV/biomass/battery hybrid power system in the Dehiba town, located in the eastern province of Tataouine, Tunisia. The objective is to minimize the COE while fulfilling the load demand for a rural region.

  • Comparing the performance of the proposed mFSO method with other algorithms such as the original FSO, SMA, and SOA to establish its superiority.

The subsequent sections of this paper are organized as follows: Sect. 2 provides an overview of the meteorological data for the specific area of the case study. Section 3 presents the mathematical model of the hybrid system components proposed in this study. Section 4 explores the system optimization problem and the selected reliability criteria for evaluation. Section 5 delves into the mathematical analysis of the optimization algorithms employed. In Sect. 6, the modeling outcomes are presented. Finally, Sect. 7 presents the conclusion of this work.

2 Case study area

The paper introduces a hybrid system specifically designed for the climatic conditions of Dehiba region, a small commune located in the eastern province of Tataouine, Tunisia. Dehiba is situated approximately 4 km west of the Libyan frontier and a similar distance east of Wazzin, a town in Libya. Figure 1 illustrates the precise location of the Dehiba commune [31]. Figure 2 indicates a hypothetical forecast of the AC loads for a residential building in this area region throughout one year, with average load of 49.13 kW and maximum load of 160.10 kW. The National Aeronautics and Space Administration (NASA) database, which provides hourly meteorological data, particularly for solar radiation and temperature, has been utilized [32]. The simulation and analysis were conducted using MATLAB software, version R2020a, resulting in the obtained findings. Figure 3 displays the yearly solar radiation data per hour [32], while Fig. 4 presents the ambient hourly temperature [32]. In this study, the amount of biomass feedstock per hour during one year was estimated based on the assumptions shown in Fig. 5 [33].

Fig. 1
figure 1

The case study area location on the Google map [31]

Fig. 2
figure 2

The suggested load profile

Fig. 3
figure 3

The hourly solar radiation per year for the suggested case study area [32]

Fig. 4
figure 4

The hourly profile of the ambient temperature per year [32]

Fig. 5
figure 5

The monthly biomass average rated profile [33]

3 Mathematical model of the suggested hybrid system components

This study presents a proposed optimal sizing approach for a renewable hybrid system, which involves the integration of PV units, a biomass gasification system, and batteries as backup units. The mathematical model describing the behavior of these components is outlined below, and the key parameters of these units are provided in Table 2.

Table 2 The main characteristics of the hybrid system’s parts

3.1 Photovoltaic model

The PV module is a crucial component in the production of solar energy in the suggested hybrid system, and the following formulas illustrate the PV mathematical modeling [34, 36]:

$$P_{{{\text{PV}}}} \left( t \right) = \left( {N_{{{\text{PV}}}} { }P_{{{\text{PV}}}}^{{{\text{rat}}}} { }\eta_{{{\text{wire}}}} { }\eta_{{{\text{PV}}}} } \right)\left( {\frac{{R_{I} \left( t \right)}}{{1000{ }}}} \right)\left( {1 - 0.0037\left( {T_{{{\text{cell}}}} \left( t \right) - { }T_{r} } \right)} \right)$$
(1)
$$T_{{{\text{cell}}}} \left( t \right) = R_{{\text{I}}} \left( t \right)\left( {\frac{{{ }T_{N } - 20{ }}}{{0.8{ }}}} \right) + T_{A} \left( t \right)$$
(2)

where \(P_{{{\text{PV}}}} \left( t \right)\) denotes the PV module’s power output, \(N_{{{\text{PV}}}}\) is the PV arrays number, \(P_{{{\text{PV}}}}^{{{\text{rat}}}}\) presents the PV rated power, \(\eta_{{{\text{wire}}}}\) and \(\eta_{{{\text{PV}}}}\) indicate the efficiencies of the wire and PV modules, respectively. \(R_{I} \left( t \right)\) indicates the hourly PV radiation intensity, \(T_{{{\text{cell}}}}\) is the cell temperature at time (t), \(T_{A} \left( t \right)\) is the ambient temperature (°C), \(T_{r}\) and \(T_{N }\) are the cell temperature under standard operating conditions and under a normal operating condition (°C), respectively.

3.2 Biomass system model

To account for the intermittent nature of solar modules and ensure the fulfillment of load demand, a biomass unit is incorporated as a significant power source in the proposed hybrid system. The biomass system plays a vital role in maintaining the stability and reliability of the suggested hybrid energy system. The following equation is utilized to calculate the output power of the biomass generator at time (t) \(\left( {P_{{{\text{BG}}}} \left( t \right)} \right)\) [22, 37, 38];

$$P_{{{\text{BG}}}} \left( t \right) = \frac{{N_{G} }}{{0.2998{ }}}\left( {\frac{{\eta_{{{\text{syn}}}} {\text{ LHV}}_{B} \,B_{{{\text{rat}}}} \left( t \right)}}{{{\text{LHV}}_{{{\text{syn}}}} { }}} - 0.0644{ }P_{{{\text{BG}}}}^{{{\text{rat}}}} } \right),$$
(3)

where \(N_{G}\) denotes the generator number, \({\text{LHV}}_{B}\) and \({\text{LHV}}_{{{\text{syn}}}}\) are the lower heat value of the biomass feed stock (14.8MJ/kg) and the syngas (4.766MJ/kg), respectively. \(B_{{{\text{rat}}}}\) presents the biomass feed stock consumption rate at time (t), \(P_{{{\text{BG}}}}^{{{\text{rat}}}}\) is the rated power of the biomass generator, and \(\eta_{{{\text{syn}}}}\) indicates the efficiency of the syngas which can be estimated by the following formula [39];

$$\eta_{{{\text{syn}}}} = \frac{{{\text{LHV}}_{{{\text{syn}}}} { }m_{{{\text{syn}}}} }}{{{\text{LHV}}_{B} { }m_{B} }},$$
(4)

where \(m_{{{\text{syn}}}}\) and \(m_{B}\) indicate the mass flow of the biomass syngas and feed stock, respectively. From the following formula, the average feedstock fuel consumption at time (t) \(\left( {F_{{{\text{cons}}}} } \right)\) can be computed [22, 37, 38];

$$F_{{{\text{cons}}}} \left( t \right) = \frac{{{\text{LHV}}_{{{\text{syn}}}} { }\left( {(0.0644\,{*}\,N_{G} \,{*}\,P_{{{\text{BG}}}}^{{{\text{rat}}}} ) + \left( {0.2998\,{*}\,P_{{{\text{BG}}}} \left( t \right)} \right)} \right)}}{{\eta_{{{\text{syn}}}} {\text{ LHV}}_{B} }},$$
(5)

3.3 Converter model

The equations provided below depict a simple inverter model based on input power \(P_{{{\text{inv}}}}^{{{\text{in}}}} \left( t \right)\) and output power \(P_{{{\text{inv}}}} \left( t \right)\), in addation to power generated from renewable sources \(\left( {P_{{{\text{Ren}}}} } \right)\) [22];

$$P_{{{\text{inv}}}} \left( t \right) = P_{{{\text{inv}}}}^{{{\text{in}}}} \left( t \right) \times \eta_{{{\text{inv}}}} ,$$
(6)
$$P_{{{\text{Ren}}}} \left( t \right) = P_{{{\text{PV}}}} \left( t \right) + {\raise0.7ex\hbox{${P_{{{\text{BG}}}} \left( t \right)}$} \!\mathord{\left/ {\vphantom {{P_{{{\text{BG}}}} \left( t \right)} {\eta_{{{\text{inv}}}} }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${\eta_{{{\text{inv}}}} }$}},$$
(7)

where \(\eta_{{{\text{inv}}}}\) is the inverter efficiency.

3.4 Battery model

During peak loads or periods of capacity generation, batteries are employed as an extra source of power in the system to maintain a steady voltage. During the charging and discharging phases, the following expressions can be used to compute the battery’s state of charge (\({\text{SOC}}_{B}\)), charging energy (\({\text{CH}}_{B}\)), and discharging energy (\({\text{DIS}}_{B}\)) [34, 40];

In the charging stage,

$${\text{CH}}_{B} \left( t \right) = \left( {P_{{{\text{Ren}}}} \left( t \right) - { }\left( {P_{L } \left( t \right)/\eta_{{{\text{inv}}}} } \right)} \right){* }\,\Delta t{*}\,\eta_{{{\text{CH}}}}$$
(8)
$${\text{SOC}}_{B} \left( t \right) = {\text{SOC}}_{B} \left( {t - 1} \right){*}\left( {1 - \gamma } \right) + {\text{CH}}_{B} \left( t \right)$$
(9)

In the discharging stage,

$${\text{DIS}}_{B} \left( t \right) = \left( {\left( {P_{L } \left( t \right)/\eta_{{{\text{inv}}}} } \right) - P_{{\text{Ren }}} \left( t \right)} \right){*}\Delta t{*}\,\eta_{{{\text{DIS}}}}$$
(10)
$${\text{SOC}}_{B} \left( t \right) = {\text{SOC}}_{B} \left( {t - 1} \right){*}\left( {1 - \gamma } \right) - {\text{DIS}}_{B} \left( t \right)$$
(11)

where \(P_{L }\) indicates the load power demand, \(\Delta t\) indicates the simulation time durations, \(\gamma\) denotes the rate of the self-discharge, \(\eta_{{{\text{ch}}}}\) and \(\eta_{{{\text{dis}}}}\) present the charging and discharging efficiencies of the battery unit, respectively.

4 Optimization problem and reliability criteria

The major goal of this study is to establish the number of components needed for the proposed hybrid power system to make it more cost effective. By employing the developed mFSO approach, the recommended hybrid system’s objective function is to minimize the COE, LPSP, and excess energy (\(E_{{{\text{exc}}}}\)) distributed in the dummy load (\(P_{{{\text{Dum}}}}\)). The PV panels’ number, the number of biomass generator, and battery module number are the decision variables in this study. The principal objective function, the optimization limitations, the cost and financial evaluation, and the system approach to management are all covered in this part.

  1. a.

    Cost analysis

    The NPC provides an estimation of the cumulative expenses, including significant investments and operational costs, throughout the project’s duration. On the other hand, the COE (Cost of Energy) represents the cost per kilowatt-hour ($/kWh) generated by the system. By employing the equations mentioned below, we can express the COE and NPC as follows [34, 41]:

    $${\text{NPC }} = \frac{{C_{{{\text{ann}}}}^{{{\text{tot}}}} { }}}{{{\text{CRF}}}}$$
    (12)
    $${\text{CRF}} \, = \,\frac{{R \left( {R + 1} \right)^{M} }}{{\left( {R + 1} \right)^{M} - 1}}$$
    (13)
    $$C_{{{\text{ann}}}}^{{{\text{tot}}}} = C_{{{\text{ann}}}}^{{{\text{cap}}}} + C_{{{\text{ann}}}}^{{{\text{fuel}}}} + \mathop \sum \limits_{i = 1}^{M} C_{O\& M,i} + \mathop \sum \limits_{i = 1}^{M} C_{{{\text{rep}},i}}$$
    (14)
    $${\text{COE }} = \frac{{C_{{{\text{ann}}}}^{{{\text{tot}}}} { }}}{{\mathop \sum \nolimits_{1}^{8760} \,P_{L} }}$$
    (15)

    where \(C_{{{\text{ann}}}}^{{{\text{tot}}}}\) is the total cost over a period of 8760 h, \({\text{CRF}}\) indicates the investment recovery factor, \(R\) stands for the interest rate parameter (6%), \(M\) stands for the overall lifetime of the project (20 year), \(C_{{{\text{ann}}}}^{{{\text{cap}}}}\) presents the yearly capital cost for each components, \(C_{O\& M}\) stands for the operational and upkeep value of the system elements, \(C_{{{\text{rep}}}}\) is the replacement cost for the required system elements, and \(C_{{{\text{ann}}}}^{{{\text{fuel}}}}\) stands for the fuel cost of the biomass.

  2. b.

    Objective function and constraints

    The subsequent equations demonstrate the objective function and constraints;

    $${\text{Min }}F\left( X \right) = {\text{Min}}\left( {\omega_{1} \,{\text{COE}} + \omega_{2} \,{\text{LSPS}} \, + \,\omega_{3} \,E_{{{\text{exc}}}} } \right)$$
    (16)

    where,

    $${\text{LPSP }} = \mathop \sum \limits_{1}^{T} \frac{{{\text{LPS}}\left( t \right)}}{{P_{L } \left( t \right)}}$$
    (17)
    $${\text{LPS}}\left( t \right) = P_{L } \left( t \right){-}\left( {P_{{{\text{Ren}}}} \left( t \right) + {\text{SOC}}_{B} \left( {t - 1} \right) - {\text{ SOC}}_{{{\text{min}}}} } \right)\,*\,\eta_{{{\text{inv}}}}$$
    (18)
    $$E_{{{\text{exc}}}} = \mathop \sum \limits_{1}^{T} \frac{{P_{{{\text{Dum}}}} \left( t \right)}}{{P_{L } \left( t \right)}}$$
    (19)

    The maximum number of system components constraints:

    $$\begin{gathered} 1\,\, \le \,\,N_{{{\text{PV}}}} \, \le \,\,N_{{{\text{PV}}}}^{{{\text{max}}}} , \hfill \\ 1\,\, \le \,\,N_{G} \,\,\, \le \,\,N_{{\text{G}}}^{{{\text{max}}}} , \hfill \\ 1\,\, \le \,\,N_{{{\text{Bat}}}} \, \le \,\,N_{{{\text{Bat}}}}^{{{\text{max}}}} , \hfill \\ \end{gathered}$$
    (20)

    where X stands for the optimization problem’s control variables (\(N_{{{\text{PV}}}}\), \(N_{G}\), and \(N_{{{\text{Bat}}}}\)) that must be optimized utilizing the suggested optimization techniques, \(\omega_{1}\), \(\omega_{2}\), and \(\omega_{3}\) are the weight parameter for each objective function and \({\text{LPS}}\) presents the loss of power supply at time (t). For a reliable system, the maximum allowable LPSP should be considered, in the following equation, \({\text{LSPS}}_{{{\text{max}}}}\) represents the maximum allowable power supply probability deficiency (0.05).

    $${\text{LSPS}} \le {\text{LSPS}}_{{{\text{max}}}}$$
    (21)
  3. c.

    Operational energy management strategy

    As for the operational energy management strategy, it is illustrated in Fig. 6 [22, 34, 41]. The operational energy management strategy of a hybrid PV, biomass system, and battery backup unit entails optimizing the use of these various energy sources to ensure efficient and effective energy generation and management. To meet the system’s energy needs, this strategy balances the energy generated by various sources. The suggested hybrid system is built to automatically switch between energy sources, reduce cost and increase efficiency.

    Fig. 6
    figure 6

    The operation strategy flowchart [22, 34, 41]

    In the first stage, the hybrid PV system is designed to generate electricity from solar panels during the day and store the extra energy in batteries for use at night and other times of low light. It is preferable to use the energy produced to satisfy the system’s immediate demand without the use of storage units. Second, the biomass system is intended to supplement the energy produced by the PV system. The system can then use the energy produced by the biomass system to supplement the energy produced by the PV system, supplying more of the system’s energy requirements. Last but not least, the battery backup unit serves as a backup energy source in case of blackouts or other events that prevent primary energy sources from producing electricity. When energy demand is low, the batteries can be charged, and when energy demand is high, they can be used.

5 Mathematical model of the proposed algorithms

5.1 Firebug swarm optimization (FSO) algorithm

FSO [42], draws inspiration from Pyrrhocorids apterous, commonly known as Firebugs, exhibiting two prominent behavioral traits. During the summer, firebugs can either explore individually or gather in groups. Socializing enables the bugs to reduce the risk of predation and locate suitable partners for reproduction. The firebugs’ quest for the optimal partner can be viewed as a process of discovering a set of optimal solutions for a given problem. FSO emulates these behaviors mathematically, serving as a foundation for a global optimization algorithm. These five optimization-related behaviors are simulated within FSO: a) construction of female bug colonies, b) partner selection, c) chemotactic behavior of females, d) attraction of males to the fittest female, and e) group cohesion. The mathematical model of FSO is represented as follows:

5.1.1 The construction of female colonies

During this stage, the construction of female colonies takes place. These colonies serve as the gathering spots where male bugs search for suitable partners. The primary aim at this stage is to minimize the associated cost, as the selection of mates should be done with the least cost possible. FSO algorithm starts with male bugs (BM) and female bug (BF) randomly distributed in the search space within the search space. The position of each bug is represented by a random uniform vector variable. Furthermore, each bug is assigned a cost function value.

5.1.2 Partner selection

Each BM’s initial position is set to match the position of its partner BF within the colony. Initially, the FSO designates the top-notch female bug in a group as the leader controlled by the dominant male. The males’ positions are constantly adjusted to align with the fittest female in their respective groups. The FSO updates the positions of all BF in the colony simultaneously. Furthermore, the cost function evaluations are highly optimized through vectorization, where the position vector of all individuals is consolidated into a single matrix, and their corresponding costs are stored in a single array to accelerate computation.

5.1.3 Chemotactic movement behavior of female bugs

The columns of the D × BF matrix, m(a).F, represent the position of the female bug (BF). The mathematical formula to immediately update all BF in each group using Hadamard arithmetic functions is shown in Eqs. (22) and (23):

After initialization, the location of each BF is updated as it moves toward the leading BM in its colony. The locations of all females in the group are stored as a matrix and updated synchronously. Assume that \(m \left( m \right). {\varvec{F}}\,\user2{ }{\text{is a matrix of}}\,{\varvec{D}} \times {\varvec{B}}_{{\varvec{F}}}\) dimensions, its columns represent the position of the female bugs, the equation to update all females in a specific group is given as follows:

$$M_{x} \leftarrow {\text{repmat}}\left( {{\varvec{m}}\left( m \right).x,1,B_{F} } \right)$$
(22)
$$M_{y} \leftarrow {\text{repmat}}\left( {{\varvec{m}}\left( a \right).x,1,B_{F} } \right)$$
(23)

where: a is a randomly generated integer in the range between 1 and BF, repmat(A, m, n) is a function that replicates a matrix “A” in a tiled manner to create a larger matrix of size m-by-n. The resulting matrix has m copies of A in the row direction and n copies of A in the column direction. The repmat(A, m, n) function takes three arguments: A: The matrix or array that needs to be replicated, m: The number of times the matrix A will be replicated in the row dimension, and n. In the given equation,  \({\mathbf{Mx}}\, \leftarrow \,{\text{repmat}}({\mathbf{m}}(m).{\mathbf{x}},{\text{ 1}},B_{F} )\), the repmat function is used to replicate the matrix \({\mathbf{m}}(m).{\mathbf{x}}\). NF times in the column direction. The resulting matrix, Mx, will have the same rows as m(m).x and BF columns, with each column being a copy of m(m).x

$${\varvec{m}}\left( m \right).F \leftarrow m\left( m \right).F + {\text{Col}}_{1} \odot \left( { M_{x} - \user2{ m}\left( m \right).F } \right) + \text{Col}_{2} \odot \left( { M_{y} - \user2{ m}\left( m \right).F } \right)$$
(24)

where \({\text{Col}}_{1}\) and \({\text{Col}}_{2}\) are the strength of attraction toward the first and the second colonies. These are coefficient vectors or matrices that control the movement of the firefly. They are typically random values within a specified range.

The term \({\text{Col}}_{1} \odot \left( { M_{x} - \user2{ m}\left( m \right).F } \right)\) represents the attraction toward the target position Mx. It calculates the difference between the target position and the current position of the firefly m, and then scales this difference by the coefficients in Col1. The element-wise multiplication adjusts the movement magnitude for each dimension of the position vector. Similarly, the term \({\text{Col}}_{2} \odot \left( { M_{y} - \user2{ m}\left( m \right).F } \right)\) represents the attraction toward the target position My. It calculates the difference between the target position and the current position of the firefly m, and then scales this difference by the coefficients in Col2. Finally, the updated position of the firefly, \(m\left( m \right).F\), is obtained by adding the attraction terms to the current position, \(m\left( m \right).F\). By iteratively applying this equation to all fireflies in the swarm, the FSO algorithm simulates the movement and attraction behavior of fireflies toward brighter positions, aiming to converge toward optimal or near-optimal solutions.

It’s worth noting that the specific values and calculations of Col1 and Col2 may vary depending on the implementation and problem being solved. These coefficients are often generated randomly or adaptively adjusted during the optimization process to balance exploration and exploitation.

5.1.4 The male’s attraction to the fittest female

Every male is drawn to the most physically fit female worldwide, regardless of which colony she belongs to. If each male were to only move toward the fittest female within its own colony, it could result in the group dispersing. Hence, males are capable of moving toward the fittest female outside their colony to avoid premature or early convergence. This behavior also allows for the exploration of larger areas within the search space, as illustrated in the following equation.

$${\varvec{m}}\left( m \right).{\mathbf{x}} \leftarrow {\varvec{m}}\left( m \right).{\mathbf{x}} + {\text{Col}}_{3} \odot \left( { g - \user2{ m}\left( m \right).{\mathbf{x}}} \right)$$
(25)

where: This term \(m\left( m \right).x\) represents the updated position of the male bug after applying the movement equation, Col3 is a coefficient that controls the movement of the bug toward the global best position. It is typically a random value within a specified range, \(\odot\) represents the element-wise multiplication (Hadamard product) between two vectors or matrices, and g represents the global best position in the search space. It is often the position associated with the highest fitness or objective function value among all fireflies.

By iteratively applying this equation to all bugs in the swarm, the algorithm aims to simulate the attraction behavior of male bugs toward the global best position, promoting exploration and exploitation to converge toward optimal or near-optimal solutions.

5.1.5 Group cohesion

The entire group moves together as a unified unit, with each male bug following the movement direction of another BM, as demonstrated in Eq. (26). Every male bug replicates the movement direction of a randomly selected male bug that is heading toward the fittest female bug. This approach allows all males to converge toward a favorable solution and prevents them from getting trapped in local minima. Furthermore, Female Sexual Ornamentation (FSO) aims to facilitate independent movements of male bugs along different dimensions, promoting improved exploration of novel solutions. The update equation employs element-wise Hadamard multiplication.

$${\varvec{m}}\left( m \right).{\mathbf{x}} \leftarrow {\varvec{m}}\left( m \right).{\mathbf{x}} + {\text{Col}}_{4} \odot \left( { g\user2{ } - \user2{ m}\left( b \right).{\mathbf{x}}} \right)$$
(26)

where g is the new global best position in the search space, and \({\text{Col}}_{4}\) is a coefficient that influences the movement of bugs toward the positions associated with the fourth colonies.

Algorithm 1
figure a

Firebug swarm optimization (FSO)

5.2 Proposed modified firebug swarm algorithm

This section introduces the modified Firebug swarm optimization algorithm (mFSO), which addresses the limitations of the FSO algorithm [42]. It begins by discussing the drawbacks of the FSO algorithm and subsequently presents the design of the modified firebug swarm optimization algorithm (mFSO).

5.2.1 Limitations of the FSO algorithm

The primary drawback of the FSO algorithm lies in its limited exploration capability and imbalanced exploitation-exploration trade-off, leading to the occurrence of local optima and restricting its ability to discover optimal solutions. This issue arises due to exploration being predominantly emphasized in the initial iterations, while exploitation dominates in the later iterations.

5.2.2 Architecture of the mFSO

The mFSO model being proposed incorporates three primary operators, namely: a) a logistic-based chaotic search, b) an opposite-based learning operator (OBL), and c) operators for phasor and transition (TO & PO) operations.


Logistic–based Chaotic search Chaotic-based search exhibits a semi-random behavior, which enables superior exploration compared to ergodic methods. Ergodic methods rely on probability-based searches, which lead agents to spend more time in search zones with a higher likelihood of finding solutions. It has shown good results in some optimizers like [43, 44]. In our proposed mFSO (modified Firefly Algorithm), we utilize the logistic map during the chaotic local search (CLS) phase, as outlined as follows.

$$o^{s + 1} = {\text{Co}}^{s} \left( {1 - o^{s} } \right)$$
(27)

where: \(s{ } = { }1,{ }2,{ }...,\,T{ }, \,o^{s} \in \left( {0 ,1} \right) , \,{\text{and}}\, o^{s} \ne 0.25 , 0.5, 0.75\).

where \(o^{s}\) is the chaotic number in iteration k and the CLS solution is represented as follows:

$${{\complement}}_{s} = \left( {1{ } - { }\mu } \right) * T + \mu {\grave {{\complement}}_{i}} , \,\,\,\,\,\,i = 1,2,\ldots.,n$$
(28)

where: Cs is the best solution’s value, \({\text{C}}_{i}\) i: the solution’s measurement value, and \(\mu\) is represented as follows:

$$\mu \user2{ } = \user2{ }\frac{{\user2{T } - \user2{ t} + 1}}{{\varvec{T}}}$$
(29)

where T is the number of iterations, t is the current iteration, and, \(\grave {{\complement}}\) I is represented as:

$$\grave {{\complement}}_{i} = {\text{LOB}} + {\complement}_{i} * \left( {{\text{UPB}} - {\text{LOB}}} \right)$$
(30)

where LOB & UPB are the lower and the upper boundary, respectively.

The Opposite-based Learning (OBL) [45] enhances the algorithm’s ability to exploit and mitigates the risk of local optima. OBL involves calculating the opposite solution and comparing it with the original solution to determine the optimal solution. This approach effectively improves the algorithm’s performance.

Let x is a real number falls within [lob, upb]. \(\overline{{\text{x}}}\) is computed as: \({ }\overline{{\text{x}}} = { }\left( {{\text{upb }} + {\text{ lob}}} \right){ } - { }x{ }\). Where: upb is the upper boundary, lob is the lower boundary. For N dimensions \(\overline{{\text{x}}}\) can be calculated as follows:

$$\overline{x}_{i} = \left( {upb_{i} + lob_{i} } \right) - x_{i}$$
(31)

where \(x_{i}\) represents position of the male bug in \(i{\text{th}}\) dimension, \({\text{upb}}_{i}\), \({\text{lob}}_{i}\) are the upper and upper boundaries of dimension i.

The Phasor operator (PO) is grounded on periodic function in the interval of \(\left[ {\pi , 2 \pi } \right]\). The periodic functions are suitable to represent any algorithms’ arguments using an angle ϴ where a function of angle ϴi is defined for each agent i. Equations (32) and (33) represent the agent as follows:

$$p\left( {\theta_{i}^{t} } \right) = \left| { \cos \theta_{i}^{t} } \right|^{{2*{\text{sin}}_{i}^{t} }}$$
(32)
$$g\left( {\theta_{i}^{t} } \right) = \left| { \sin \theta_{i}^{t} } \right|^{{2*{\text{cos}}_{i}^{t} }}$$
(33)

where p \(p\left( {\theta_{i}^{t} } \right)\) and \(g\left( {\theta_{i}^{t} } \right)\) are the values generated by the Phasor operator.

The calculation of the nonlinear transition operator (TO) involves the use of the exploration-oriented PO. The TO plays a crucial role in transitioning from exploration to exploitation and is determined by the following formula:

$${\text{TO}} = {\text{exp}}^{{\frac{ - t}{T}}}$$
(34)

where: t represents the present step and T represents the maximum steps number.

The utilization of TO serves the purpose of avoiding local optima during the search phase. The revised search equations, incorporating the TO operator, are presented below.

$$\begin{aligned} \user2{m}\left( m \right).{\mathbf{F}} \leftarrow & \user2{m}\left( m \right).{\mathbf{F}} + C_{1} \odot \left( {M_{x} - \user2{TO*~m}\left( m \right).{\mathbf{F}}} \right) \\ & + C_{2} \odot \left( {M_{y} - \user2{TO~~*m}\left( m \right).\user2{F}} \right) \\ \end{aligned}$$
(35)
Algorithm 2
figure b

Modified firebug swarm optimization (mFSO)

6 Simulation results

The results in this section are categorized into two sections: the evaluation of the mFSO method’s performance and the application of the mFSO approach in determining the optimal design for the proposed hybrid system.

6.1 Performance evaluation of mFSO

The evaluation of the mFSO quality is conducted by utilizing it to determine the optimal values for the IEEE CEC’20 test suite [46] functions. The CEC’20 test suite consists of ten test functions that are categorized into unimodal (F1:F4), hybrid (F5:F7), and composition functions (F8:F10). These categories are presented in Table 3.

Table 3 Description with fitness score of CEC2020 functions

The mFSO’s performance is assessed by comparing it to several commonly employed algorithms, namely FSO [42], SMA [47], SOA [48], HHO [49], CDO [50], WOA [51], PDO [52], COVIDOA [53], and SCA [54]. The evaluation criteria encompass various metrics such as the minimum, maximum, average (mean), and standard deviation of the fitness scores, as well as the application of the Wilcoxon rank-sum test [55].

  1. 1.

    Mean is the ratio of the optimization defined as follow:

    $${\text{Mean}} = \frac{1}{N}\mathop \sum \limits_{i - 1}^{N} {\text{op}}^{i}$$
    (36)

    where: N is the total number of operations, opi is the optimal solution at operation i.

  2. 2.

    Minimum represents the optimal or minimum score of the fitness achieved by the algorithm throughout N operations. It is defined as follows:

    $${\text{Min}} = {\text{min}}_{i = 1}^{N} \,{\text{op}}^{i}$$
    (37)
  3. 3.

    Maximum denotes the poorest or maximum fitness score produced by the algorithm during N operations. It can be expressed as follows:

    $${\text{Max }} = {\text{max}}_{i = 1}^{N} \,{\text{op}}^{i}$$
    (38)
  4. 4.

    Standard deviation (Std) assesses the stability and resilience of the algorithm. A smaller standard deviation indicates greater stability, ensuring consistent convergence to the same solution. Conversely, a larger Std value suggests that the algorithm produces more random outcomes, exhibiting less predictability.

    $${\text{Std}} = \sqrt {\frac{1}{N - 1}\sum {\left( { {\text{op}}^{i} - {\text{Mean}}} \right)^{2} } }$$
    (39)
  5. 5.

    Wilcoxon rank-sum P values involves employing the Wilcoxon test to examine the correlation between the outputs of the tested algorithms. The null hypothesis assumes that the comparison results are nearly identical, while the alternative hypothesis assumes distinguishable differences between the compared algorithms. The Wilcoxon test produces a P value. Rejecting the null hypothesis (P < 0.05) indicates non-correlated results, whereas accepting the null hypothesis (P > 0.05) suggests correlated results.

6.2 Result discussion

The performance evaluation of the proposed mFSO was conducted by comparing it with several baseline algorithms using the ten functions from the CEC2020 benchmark. The results, presented in Table 4, provide insights into the performance of the algorithms based on various evaluation metrics such as minimum, maximum, mean, and standard deviation (Std) of the fitness scores. The best results for each metric are highlighted in bold.

Table 4 Description of the fitness score of CEC2020 benchmark functions

Upon analyzing the results, it is evident that the mFSO algorithm demonstrates superiority over the tested algorithms across multiple evaluation metrics for a significant number of test functions. Specifically, mFSO outperforms all other algorithms in terms of achieving the minimum or best fitness score, as well as maintaining the lowest maximum fitness score, mean fitness score, and minimum standard deviation for seven out of the ten test functions (F3, F4, F6, F7, F8, F9, and F10). This indicates the algorithm’s robustness and ability to find high-quality solutions consistently.

For function F5, while mFSO does not achieve the minimum/best fitness value, it demonstrates the best mean score, standard deviation, and minimum worst fitness score. On the other hand, the SMA algorithm performs exceptionally well for F2, achieving the minimum/best fitness value.

Moreover, in the case of function F1, mFSO achieves competitive scores in terms of the minimum and maximum fitness values. The HHO algorithm, on the other hand, achieves the best mean score, closely followed by the SMA algorithm, which exhibits the second-best standard deviation.

These findings highlight the search capabilities and stability of the mFSO algorithm, showcasing its effectiveness in addressing a wide range of optimization problems. The algorithm consistently outperforms most of the tested algorithms across multiple evaluation metrics, indicating its competitiveness and potential as a robust optimization approach.

Based on the discussed results, mFSO exhibits several advantages over its competitors, as evidenced by the previously discussed results. First and foremost, mFSO consistently outperforms the tested algorithms in terms of achieving superior fitness scores across multiple evaluation metrics. It demonstrates the ability to obtain the minimum or best fitness score, maintain the lowest maximum fitness score, and achieve the best mean fitness score for a significant number of test functions. This indicates the algorithm’s effectiveness in finding high-quality solutions and its robustness in addressing optimization problems.

Furthermore, mFSO showcases remarkable stability and search capabilities. It consistently delivers low standard deviations, indicating the algorithm’s ability to converge to reliable solutions consistently. The algorithm’s performance is particularly notable in terms of maintaining low standard deviations for various test functions, showcasing its stability and reliability in finding optimal or near-optimal solutions.

In addition to evaluating the performance of the proposed mFSO (modified Firebug Swarm Algorithm), the Wilcoxon rank sum test was employed to assess the distinguishability of mFSO from other competing algorithms. The utilization of statistical tests adds a rigorous and quantitative analysis to the evaluation process. The Wilcoxon rank-sum fitness scores, as presented in Table 5, provide insights into the significance of the observed differences.

Table 5 Wilcoxon ranksum test p-value of the mFSO VS. competitor algorithms for benchmark functions

The results obtained from the Wilcoxon rank sum test reveal that a majority of the p-values are below the significance level of 0.05. This indicates a statistically significant difference between the performance of mFSO and the other algorithms for the majority of the CEC2020 functions. The significance level of 0.05 is commonly used to determine if the observed differences are unlikely to have occurred by chance.

The findings from the Wilcoxon rank sum test contribute to the credibility and reliability of the mFSO results. By demonstrating a significant difference in performance compared to the competing algorithms, it strengthens the argument for the effectiveness of mFSO as a competitive optimization approach.

6.3 Convergence behavior analysis

To assess the stability and convergence behavior of the mFSO algorithm, a comprehensive convergence analysis was conducted to compare the mFSO algorithm with its competitor algorithms. The convergence curves, as depicted in Fig. 7, provide valuable insights into the convergence rates of the algorithms across the CEC2020 functions.

Fig. 7
figure 7

Convergence curve of mFSO compared to the other algorithms—CEC2020

The results of the convergence analysis highlight a significant advantage of the mFSO algorithm in terms of its convergence rate when compared to its competitors. The convergence curves clearly demonstrate that mFSO exhibits a noticeably faster convergence, characterized by steep descent and rapid progress toward optimal or near-optimal solutions. This accelerated convergence is observed for the majority of the CEC2020 functions.

The superior convergence rate of mFSO translates into several advantages for the algorithm. Firstly, it indicates that mFSO has a strong ability to navigate complex optimization landscapes efficiently. By quickly converging toward promising solutions, mFSO demonstrates its effectiveness in exploring and exploiting the search space, enabling it to find high-quality solutions in a shorter timeframe compared to its competitors.

Furthermore, the rapid convergence rate of mFSO contributes to its computational efficiency. The algorithm requires fewer iterations or evaluations to reach convergence, reducing the computational burden and associated time requirements. This advantage is particularly significant in scenarios where time-sensitive optimization tasks need to be performed or when computational resources are limited.

The competitive and efficient convergence behavior positions mFSO as a promising solution for tackling global optimization problems. Its rapid convergence rate enhances its applicability to real-world applications where finding high-quality solutions within limited computational resources is crucial.

6.4 Boxplot behavior analysis

To further analyze the performance results, a boxplot analysis was performed to examine the distribution of achieved results within the first three quartiles. Figure 8 illustrates the boxplots for the ten functions of the CEC2020 benchmark. The boxplot analysis reveals distinct advantages of the mFSO algorithm compared to its competitors. The boxplots representing mFSO are relatively narrow, indicating a more concentrated and consistent performance across the evaluated functions. This suggests that mFSO consistently achieves results closer to the optimal or near-optimal solutions, exhibiting less variation in its performance.

Fig. 8
figure 8

Boxplot of mFSO compared to the other algorithms—CEC2020

Furthermore, mFSO demonstrates the lowest scores among all the algorithms for eight out of the ten tested functions. This highlights its superior performance in terms of achieving better fitness scores compared to its competitors. The consistent dominance of mFSO in terms of lower scores accentuates its effectiveness in finding high-quality solutions and its competitive edge in the optimization landscape.

The narrower boxplots and the consistently lower scores achieved by mFSO emphasize its advantage over the competing algorithms. The concentrated performance distribution indicates that mFSO consistently converges to favorable solutions, while the wider boxplots of the other algorithms indicate more variability and less consistent performance.

6.5 Exploration–exploitation analysis

The exploration–exploitation curves provide insights into how the mFSO algorithm balances its exploration and exploitation processes while searching for optimal solutions. Figure 9 illustrates the performance of mFSO as it explores the search space and transitions to the exploitation phase for the ten functions of the CEC2020 benchmark.

Fig. 9
figure 9

Exploration–exploitation curves of mFSO—CEC2020

The results reveal that mFSO achieves a balanced ratio of exploration to exploitation in the majority of the tested functions. Initially, the algorithm dedicates more time to exploration, which involves searching a wide range of solutions to gain a comprehensive understanding of the search space. During this phase, mFSO aims to discover potential promising regions that may contain optimal or near-optimal solutions.

As the exploration phase progresses, mFSO gradually transitions to the exploitation phase, where it focuses on refining and exploiting the discovered promising regions. This shift allows the algorithm to concentrate its search efforts on the most favorable areas of the search space, aiming to converge toward the optimal solution.

The balanced exploration–exploitation strategy of mFSO demonstrates its ability to efficiently explore the search space while also exploiting the valuable information gained during the exploration phase. This approach contributes to the algorithm’s effectiveness in finding high-quality solutions.

By striking a balance between exploration and exploitation, mFSO avoids getting trapped in local optima and maintains the potential to discover globally optimal or near-optimal solutions. The exploration phase ensures that the algorithm explores diverse regions of the search space, while the exploitation phase enables the algorithm to exploit the most promising areas, refining the solutions toward optimality.

Overall, the exploration–exploitation curves of the mFSO algorithm signify its ability to achieve a balanced approach in searching for optimal solutions. The initial emphasis on exploration followed by a transition to exploitation showcases the algorithm’s capability to efficiently explore and exploit the search space, ultimately contributing to its high-performance results.

6.6 Diversity analysis

The diversity analysis of the mFSO algorithm plays a crucial role in assessing its ability to maintain population diversity and avoid getting trapped in local optima. The analysis aims to ensure a balanced exploration–exploitation strategy and enhance the algorithm’s capability to explore promising search areas. Figure 10 provides insights into the diversity analysis of mFSO.

Fig. 10
figure 10

Diversity curves of mFSO—CEC2020

The findings of the diversity analysis indicate that the mFSO algorithm effectively sustains population diversity throughout the optimization process. By maintaining a diverse set of solutions within the population, mFSO can explore a wide range of search areas, increasing the likelihood of discovering optimal or near-optimal solutions.

The sustained population diversity in mFSO is crucial for achieving a balanced exploration–exploitation strategy. A diverse population enables the algorithm to explore different regions of the search space, preventing it from prematurely converging to suboptimal solutions. This diversity allows mFSO to continue exploring and refining its solutions, enhancing its chances of finding superior solutions.

By sustaining population diversity, mFSO also mitigates the risk of premature convergence. If the algorithm lacks diversity, it may get trapped in local optima, unable to escape and explore other potentially better solutions. However, with effective diversity maintenance, mFSO can avoid such traps and continue its search for globally optimal or near-optimal solutions.

The ability of mFSO to sustain population diversity indicates its robustness and adaptability in handling diverse optimization problems. It showcases the algorithm’s capacity to explore different regions of the search space, even in challenging scenarios with complex landscapes and multiple optima.

In summary, the diversity analysis of the mFSO algorithm demonstrates its effectiveness in maintaining population diversity throughout the optimization process. The sustained diversity enables mFSO to explore promising search areas, enhance its exploration–exploitation strategy, and increase the likelihood of attaining superior solutions. By preventing premature convergence and facilitating efficient exploration, the algorithm showcases its robustness and adaptability in tackling various optimization problems.

6.7 The hybrid system results

This paper introduces the mFSO approach, which is a novel and improved method based on the Firebug Swarm Optimization method (FSO). The mFSO method is utilized to determine the optimal size of an isolated hybrid system configuration. This configuration involves the utilization of PV panels, a biomass gasifier system, and battery bank units as a backup storage system. By comparing the results of the mFSO technique with those of the original FSO, SMA, and SOA techniques, it is demonstrated that the recommended mFSO approach possesses superior properties. The control parameters for each optimization algorithm are adjusted to allow for a maximum of 100 iterations, 50 time runs, and a search agent count of 20. Figure 11 depicts the convergence curves for the proposed mFSO optimization technique applied to the hybrid PV/Biomass/Battery system, considering 100 iterations and 50 time runs.

Fig. 11
figure 11

The mFSO Convergence profile for a 100 iteration 50 time run

Figure 12 illustrates the convergence curve of the optimal objective function for the proposed mFSO approach, along with the FSO, SMA, and SOA approaches, applied to the recommended hybrid system. It is evident that the suggested mFSO technique effectively minimizes the objective function for the given scenario, achieving a value of 0.09700573 after 43 iterations. The SOA method follows closely with a value of 0.09703978 after 18 iterations, followed by the SMA algorithm with a value of 0.09709755 after 29 iterations. Lastly, the FSO technique yields a value of 0.09782792 after 23 iterations.

Fig. 12
figure 12

The most effective function’s convergence curve using mFSO, FSO, SMA, and SOA for 100 iterations

Table 6 showcases the top outcomes achieved for the primary objective function and the most suitable component sizing constraints utilizing the recommended mFSO, FSO, SMA, and SOA approaches for the suggested system configuration. The mFSO optimization technique recommended attained the lowest COE, amounting to 0.191861 $/kWh, and the least NPC, totaling 1,055,450 $, in comparison to the alternative optimization methods employed in this study.

Table 6 The optimization factors for the recommended mFSO and other optimizers

In order to assess the statistical efficacy of each utilized optimization technique, the following parameters were selected to evaluate the effectiveness of the suggested approaches. These parameters include the selection of search agents, the number of iterations, and the number of runs, all set to 20, 100, and 50, respectively. This statistical evaluation aims to determine various metrics, including maximum values (Max.), standard deviation (SD), mean, relative error (RE), median, root mean square error (RMSE), mean absolute error (MAE), and efficiency values. These values are presented in Table 7 as part of the statistical analysis.

Table 7 The statistical performance of the studied optimization algorithms

The annual cost distribution of the hybrid system’s components using the mFSO method is depicted in Fig. 13. It is evident that the battery units account for the highest percentage cost, representing 46% of the total. This is followed by the biomass system with 44%, the inverter with 31%, and the PV arrays with 4%.

Fig. 13
figure 13

The mFSO breakdown of the yearly cost of the hybrid system’s parts

7 Conclusion

In this article, a novel and enhanced optimization algorithm known as the modified Firebug swarm algorithm (mFSO) has been employed to address the optimal sizing problem of a hybrid PV/biomass/battery energy system. The improvements made to the mFSO approach aimed to overcome the limitations of the original Firebug swarm optimization (FSO) technique by incorporating three key operators: a) logistic chaotic local search, b) opposite-based learning operator (OBL) technique, and c) phasor and transition operators (TO & PO). The performance of the proposed mFSO algorithm was assessed using the ten functions from the CEC2020 benchmark. The results obtained demonstrate that the mFSO algorithm outperformed the other algorithms tested (FSO, Slime mold algorithm (SMA), Seagull optimization algorithm (SOA), Harris Hawks optimization algorithm (HHO), Chernobyl disaster optimizer (CDO), Whale Optimization Algorithm (WOA), Prairie Dog Optimization Algorithm (PDO), COVIDOA, and Sine Cosine Algorithm (SCA)) in solving a majority of the optimization problems evaluated. To determine whether the results of mFSO differed significantly from those of the rival algorithms, a Wilcoxon test was conducted. The majority of the computed p-values were below 0.05, indicating a significant difference between the mFSO and the other algorithms for most of the CEC2020 functions. Furthermore, assessing the convergence of the mFSO algorithm was essential to evaluate its stability. Hence, the convergence of mFSO and its rival algorithms was analyzed. The findings revealed that mFSO exhibited a considerably higher convergence rate compared to its competitors for most of the functions tested. This fast convergence rate establishes mFSO as a competitive and effective method for addressing global optimization problems.

To demonstrate the superior qualities of the recommended mFSO algorithm, its results for the proposed hybrid power system were compared with the optimization outcomes obtained using the original FSO, SMA, and SOA techniques. The investigation revealed that the suggested mFSO technique successfully minimized the objective function for the hybrid scenario to 0.09700573 after 43 iterations. Following mFSO, the SOA algorithm achieved a minimized value of 0.09703978 after 18 iterations, while the SMA algorithm obtained 0.09709755 after 29 iterations. Lastly, the FSO method yielded a value of 0.09782792 after 23 iterations. When comparing the results from the mFSO technique with other optimization methods employed in this study, it was observed that the mFSO approach produced the lowest energy cost (COE) of 0.191861 $/kWh and the lowest net present cost (NPC) of 1,055,450 $.

However, one of the main weaknesses of the mFSO algorithm lies in its sensitivity to parameter settings. Although the algorithm has shown promising results in our experiments, determining the optimal parameter values can be challenging. In future work, we plan to conduct a thorough sensitivity analysis to identify the impact of different parameter configurations on the algorithm’s performance. This analysis will allow us to provide better guidelines for selecting suitable parameter values based on the problem domain and characteristics. Another area for improvement is the algorithm’s scalability. While the mFSO has demonstrated effectiveness for moderate-sized optimization problems, its performance may degrade when applied to large-scale or high-dimensional problems. To address this limitation, we intend to explore techniques such as parallelization, adaptive parameter control, and problem decomposition to enhance the scalability of the algorithm. These enhancements will enable the Firebug Swarm Algorithm to handle more complex and computationally demanding optimization tasks. Additionally, we aim to investigate hybridization of the mFSO with other optimization techniques to everage the strengths of multiple algorithms can potentially improve the overall search performance and enhance the algorithm’s ability to handle challenging optimization landscapes.

Finally, a potential avenue for future research is to explore the applicability of the modified Firebug swarm algorithm (mFSO) in dynamic environments. Investigating the algorithm’s performance and adaptability when faced with changing conditions, such as time-varying energy generation or load demands, would provide valuable insights. This could involve developing strategies to dynamically adjust the algorithm’s parameters or operators to optimize system configurations in real time. Additionally, exploring the algorithm’s robustness against uncertainties and disturbances would further enhance its practicality and effectiveness in real-world applications of hybrid energy systems.