Model predictive control and moving horizon estimation for adaptive optimal bolus feeding in high-throughput cultivation of E. coli
Introduction
Mathematical descriptions of the dynamics of the microbial systems used in biomanufacturing are key to exploiting bioprocess automation (Leavell et al., 2020). From the early screening phases to the industrial-scale reactor, a consistent model-based approach helps to accelerate product development (Neubauer et al., 2013). The mathematical description of mechanistic models allows for performing optimizations, which leads to cost-effective, consistent, and highly confident bioprocess design. In this context, there has been significant progress related to the model-based bioprocess automation, such as structuring a mechanistic model that describes key metabolic pathways, parameter estimation and design of experiment, optimal control, and modularized computational frameworks (Gomes et al., 2015, Narayanan et al., 2020, Hemmerich et al., 2021, Herwig et al., 2021).
One of the distinguishing characteristics of biological systems is that the growth of the microorganism exhibits a high level of uncertainty. Moreover, the most used cultivation strategy is fed-batch, in which no steady-state exists and the process variables vary over a wide range. Hence, a large amount of data is needed to obtain a mathematical model which is capable of describing the non-steady-state dynamics of highly nonlinear and complex systems such as cells (Cruz Bournazou et al., 2017). The recent development of high-throughput (HT) technology allows for performing a large number of laborious and time-consuming experiments by automatizing, parallelizing, and miniaturizing the experimental facilities (Puskeiler et al., 2005, Bunzel et al., 2018, Hemmerich et al., 2018). Liquid handling stations support the parallel cultivation of a large number of mini-bioreactors, based on the automated sampling, online and at-line analytics operation, real-time control, and data acquisition (Kusterer et al., 2008, Tai et al., 2015, Haby et al., 2019, Hans et al., 2020b). HT technology plays an important role, especially in the early stage of bioprocess development. Very large amounts of data from testing the cell clones under different conditions such as media, pH, temperature, induction strengths, and feeding strategies become available (Hans et al., 2020a) allowing us to understand the process better and to find the best-operating conditions in very large search spaces. This has naturally led to the enhancement of the conditional screening and strain phenotyping (Schmideder et al., 2016, Sawatzki et al., 2018, Janzen et al., 2019, Fink et al., 2021).
To exploit the full potential of the HT bioprocess development (HTBD), it is essential to couple the model-based methods with robotic facilities. The reliability of computed decisions depends on the accuracy of model predictions, e.g., each cell type and organism needs tailored feeding strategies for optimal growth or product formation. Because of limited reproducibility at the small-scale reactors at the L and mL scale, these strategies need to be adaptively controlled. In addition, monitoring of cultivations is often challenging because of the limited accuracy of online sensors, e.g., small delays and shifts in oxygen signals, or large delays in the processing of liquid samples. Finally, the satisfaction of experimental constraints is critical, such as minimum levels of sugar and/or dissolved oxygen concentrations to minimize specific stresses, i.e., mainly glucose excess, limitation and starvation, and oxygen exhaustion (Delvigne et al., 2009).
In previous works, intermittent-feeding strategies have been derived from established exponential feeding strategies considering the (previously or online identified) maximum specific growth rate of the organism (Sawatzki et al., 2018, Anane et al., 2019b, Hans et al., 2020a). Because of the limited information and feedback, it has been observed that this approach can lack robustness in real applications. There are only a few studies that incorporate model-based methods for HT experiments. However, the studies consider optimal experimental design for the model fitting instead of optimal operation for maximizing biomass (Cruz Bournazou et al., 2017, Barz et al., 2018, Kim et al., 2021a).
Model predictive control (MPC) and moving horizon estimation (MHE) are popular methods in engineering (Rawlings et al., 2017) and there has been a large number of applications for fed-batch bioreactor cultivations. Given the state trajectory from the pre-defined operating strategy, MPC has been applied for the optimal tracking control of the bioprocesses described with basic Monod equations (Ramaswamy et al., 2005, Tebbani et al., 2008), and further extended to economic objectives such as to maximize the product (Ashoori et al., 2009, Raftery et al., 2017). In the presence of measurement and model uncertainties, optimal state or parameter estimators such as Kalman filter (Markana et al., 2018) or MHE (Abdollahi and Dubljevic, 2012, del Rio-Chanona et al., 2016) are combined with MPC in order to adapt to the data. Various optimization methods have been studied such as maximization principle (Pčolka and Čelikovskỳ, 2016, Luna and Martínez, 2017) and evolutionary strategies (Freitas et al., 2017). As an alternative to MPC, the reinforcement learning (RL) method has been studied to reduce the online computation time by obtaining the closed-loop policy and for optimal operation even without the mechanistic model (Martínez et al., 2013, Petsagkourakis et al., 2020, Kim et al., 2021b).
Although several studies have proven that the model-based approach comprising a controller and estimator is promising for automated bioprocess operation (Lucia et al., 2017a), it has, to our knowledge, never been implemented in parallel mini-bioreactors for HTBD. The extension to this application is not trivial and has important challenges that can lead to poorly operated experiments. We discuss the most relevant challenges and how they have been tackled in the presented framework. First, inputs are given in a pulse form by bolus injections. In highly parallelized milliliter scale mini-bioreactors for example, individual feeding with high accuracy is very challenging and costly (Faust et al., 2014). As an alternative, pulse-based feeding has been widely utilized. In contrast to continuous feeding, intermittent feeding fails to achieve high cell density cultivation, because it induces large heterogeneities in the operating conditions such as oscillating pH, oxygen, temperature, glucose concentrations, and toxic compounds (Neubauer et al., 2013). This can, however, when properly designed and operated using scale-down techniques, be used to mimic the industrial scale heterogeneity (Neubauer and Junne, 2016). According to Anane et al. (2019b), the combination of a pulse-based scale-down approach is successful to test strain robustness and physiological constraints at the early stages of bioprocess development. The pulse-based (bolus) feeding is different compared to the standard process control problems where systems have continuous input and dynamics. This requires a different approach, namely, impulsive control systems (Yang, 2001).
Second, the measurements are multi-rate, the sampling times between the parallel bioreactors are not aligned, and the sample delays are on the order of tens of minutes such that a multi-rate formulation of the MHE is necessary. The system is limited to extracellular measurements, which makes it very difficult to generate sufficient information to properly identify mechanistic models that aim to describe a highly nonlinear and large biochemical network. For this reason, efficient methods to assure a well-posed state and parameter estimation problem are needed.
Third, given the configuration of multiple cultivation conditions and bioreactors, the proper design of the experimental campaign and distribution of the operating strategies are not trivial. The information in parallel systems (e.g. replicates) needs to be efficiently exploited to improve the probability of finding the optimal process condition.
Our contribution focuses on the computational model-based framework, namely, MPC and MHE for the optimization of the cultivation conditions for maximal growth of E. coli in parallel mini-bioreactors. Three bottlenecks encountered in the HT experiment, bolus injection, multi-rate measurements, and multiple bioreactors, are addressed: First, we illustrate how to design the objective and constraint function for the impulsive control systems. Unlike the zero-order hold discretization which has been used for impulsive systems, a full-discretization method is implemented to capture the fast dynamics happening within the pulse-feed interval. Second, we propose an arrival cost design and identifiable parameter selection method for the multi-rate and partially ill-conditioned MHE. Third, for the purpose of determining the reactor configuration for the MHE optimization, three criteria, size of the identifiable parameter subset, root mean squared error of the MHE, and computational time are suggested. The developed framework is validated through in silico studies on fed-batch cultivations of E. coli. As a proof-of-concept, we design a 24 parallel cultivation experiment, which is divided into 8 different conditions with 3 replicates. We focus on whether the MPC and MHE are capable of providing a consistent/feasible pulse-feeding strategy regardless of the operating condition. The experimental validation is conducted in the companion paper (Krausch et al., 2022).
The remainder of this paper is organized as follows: Section 2 describes the in silico experiments of mini-bioreactors and the challenges of the HT experiment. Section 3 introduces model-based methods, MHE and MPC. In Section 4, the simulation results are discussed. Finally, a discussion and some concluding remarks are provided in Section 5.
Section snippets
Experimental setup, sensors, and automated liquid handling
The robotic facility of the HTBD platform is capable of performing 24 parallel cultivations. The liquid handling station assists in sampling, measuring, glucose feeding, medium balancing, and pH control (acid and base). The 24 mini-bioreactors are placed in three columns and eight rows, and we put a numeric order to the columns (i.e., 1, 2, and 3), and an alphabetical order to the rows (i.e., A, B, …, H). Based on this configuration, reactors that have the same columns are replicates and the HT
Problem description and formulation
We describe the HT experiment and the macro-kinetic growth model in formal expressions. The states , manipulated variables , and measured variables comprise of
The experimental procedures are characterized by the following finite sets : where is the index set of the mini-bioreactors; is the discrete pulse-feeding times;
Determination of the MHE configuration and identifiability analysis
The in silico cultivation for 24 mini-bioreactors is performed in parallel, with the proposed model-based methods. To generate the in silico data, we add random uniform noise with 20% of their scale to the growth parameters in Table 2 for each row and add additional uniform noise with 5% of their scale to the replicate in each column. For the reactor-dependent parameters in Table 3, 20% uniform noises are added for all bioreactors. Moreover, Gaussian noise with 5% variance to the scales is
Concluding remarks
In this study, we develop a macro-kinetic growth model-based MHE and MPC framework for high-throughput experiments with 24 parallel mini-bioreactors. Three crucial aspects of the system are addressed. First, the pulse-based feed is formulated as an impulsive control system. We perform the analysis of the objective and constraint function and implemented a full-discretization method for solving the MPC. Second, multi-rate MHE is formulated. The MHE method utilizes parameter sensitivity as an
CRediT authorship contribution statement
Jong Woo Kim: Conceptualization, Methodology, Software, Writing – original draft. Niels Krausch: Conceptualization, Methodology, Software, Writing – original draft. Judit Aizpuru: Methodology, Software. Tilman Barz: Methodology, Formal analysis, Writing – review & editing. Sergio Lucia: Methodology, Software, Formal analysis, Writing – review & editing. Peter Neubauer: Supervision, Writing – review & editing. Mariano Nicolas Cruz Bournazou: Supervision, Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported by the German Federal Ministry of Education and Research through the Program International Future Labs for Artificial Intelligence (KIWI Biolab - Grant number 01DD20002A).
References (70)
- et al.
Lipid production optimization and optimal control of heterotrophic microalgae fed-batch bioreactor
Chem. Eng. Sci.
(2012) - et al.
Output uncertainty of dynamic growth models: effect of uncertain parameter estimates on model reliability
Biochem. Eng. J.
(2019) - et al.
Modelling overflow metabolism in Escherichia coli by acetate cycling
Biochem. Eng. J.
(2017) - et al.
Optimal control of a nonlinear fed-batch fermentation process using model predictive approach
J. Process Control
(2009) - et al.
Multirate moving horizon estimation combined with parameter subset selection
Comput. Chem. Eng.
(2021) - et al.
Real-time adaptive input design for the determination of competitive adsorption isotherms in liquid chromatography
Comput. Chem. Eng.
(2016) - et al.
Adaptive optimal operation of a parallel robotic liquid handling station
IFAC-PapersOnLine
(2018) - et al.
Speeding up enzyme discovery and engineering with ultrahigh-throughput methods
Curr. Opin. Struct. Biol.
(2018) - et al.
Model-based real-time optimisation of a fed-batch cyanobacterial hydrogen production process using economic model predictive control strategy
Chem. Eng. Sci.
(2016) - et al.
Model reduction of aerobic bioprocess models for efficient simulation
Chem. Eng. Sci.
(2020)
A comparative review of multi-rate moving horizon estimation schemes for bioprocess applications
Comput. Chem. Eng.
Integrated robotic mini bioreactor platform for automated, parallel microbial cultivation with online data handling and process control
SLAS Technol.: Transl. Life Sci. Innov.
Event driven modelling for the accurate identification of metabolic switches in fed-batch culture of S. cerevisiae
Biochem. Eng. J.
Model-based reinforcement learning and predictive control for two-stage optimal control of fed-batch bioreactor
Comput. Chem. Eng.
Multirate state estimation using moving horizon estimation
IFAC Proc. Vol.
High-throughput screening for improved microbial cell factories, perspective and promise
Curr. Opin. Biotechnol.
A moving horizon estimator for processes with multi-rate measurements: A nonlinear programming sensitivity approach
J. Process Control
Adaptive nonlinear predictive control and estimation of microaerobic processes
IFAC-PapersOnLine
Rapid development of modular and sustainable nonlinear model predictive control solutions
Control Eng. Pract.
Iterative modeling and optimization of biomass production using experimental feedback
Comput. Chem. Eng.
Multi-criterion control of a bioprocess in fed-batch reactor using EKF based economic model predictive control
Chem. Eng. Res. Des.
Dynamic optimization of bioreactors using probabilistic tendency models and Bayesia active learning
Comput. Chem. Eng.
Reinforcement learning for batch bioprocess optimization
Comput. Chem. Eng.
Computation of arrival cost for moving horizon estimation via unscented Kalman filtering
J. Process Control
Control of a continuous bioreactor using model predictive control
Process Biochem.
Bilevel parameters identification for the multi-stage nonlinear impulsive system in microorganisms fed-batch cultures
Nonlinear Anal. RWA
Open-loop optimization and trajectory tracking of a fed-batch bioreactor
Chem. Eng. Process.: Process Intensif.
Recent progress in impulsive control systems
Math. Comput. Simulation
Distributed moving horizon state estimation of two-time-scale nonlinear systems
Automatica
Challenges and opportunities on nonlinear state estimation of chemical and biochemical processes
Processes
Modelling concentration gradients in fed-batch cultivations of E. coli - towards the flexible design of scale-down experiments
J. Chem. Technol. Biotechnol.
CasADi – A software framework for nonlinear optimization and optimal control
Math. Program. Comput.
Nonlinear Programming: Concepts, Algorithms, and Applications to Chemical Processes
Online optimal experimental re-design in robotic parallel fed-batch cultivation facilities
Biotechnol. Bioeng.
Bioreactor mixing efficiency modulates the activity of a prpos::gfp reporter gene in e. coli
Microb. Cell Factories
Cited by (7)
Adaptive control of the E. coli-specific growth rate in fed-batch cultivation based on oxygen uptake rate
2023, Computational and Structural Biotechnology JournalControl strategy for biopharmaceutical production by model predictive control
2024, Biotechnology ProgressHigh-throughput process development from gene cloning to protein production
2023, Microbial Cell FactoriesModel-Based Characterization of E. coli Strains with Impaired Glucose Uptake
2023, Bioengineering