Enhanced assessment of the eReefs biogeochemical model for the Great Barrier Reef using the Concept/State/Process/System model evaluation framework

https://doi.org/10.1016/j.envsoft.2020.104707Get rights and content

Highlights

  • In politically-contentious contexts, models require a high standard of assessment.

  • We demonstrate the four-level (Concept/State/Process/System) assessment framework.

  • The performance of the eReefs marine models is assessed.

  • The approach highlights strengths and weaknesses of the models.

  • Emergent properties are particularly useful to enhance model assessment.

Abstract

The Great Barrier Reef is a UNESCO World Heritage Area that has been assessed as having a very poor outlook and needing urgent intervention. The eReefs hydrodynamic-biogeochemical models are used to complement monitoring, facilitate data interpretation and support policy decisions. Management and policy for the Great Barrier Reef are politically contentious, so a high standard of model evaluation is required. We report the application of a recently-developed four-level “CSPS” (Concept/State/Process/System) model evaluation framework to the eReefs biogeochemical model. The framework considers: Level 0, conceptual evaluation; Level 1, simulated state variables; Level 2, process rates; and Level 3, system-level patterns and emergent properties. This paper is the first complete application of this model assessment framework. It highlights both strengths and weaknesses of the eReefs biogeochemical model that might not have been apparent from a traditional model evaluation. The framework can be applied to evaluation of any complex, process-based numerical model.

Introduction

The Great Barrier Reef (GBR) has been identified by UNESCO as a World Heritage Area of “outstanding universal value” (Lucas et al., 1997). It is one of the most biodiverse regions in the world, has enormous cultural value, and has been assessed as adding at least AU$5.7B (in uncorrected 2012 Australian dollars) and 64,000 jobs to Australia's economy (Deloitte Access Economics, 2013). The Great Barrier Reef experiences diverse pressures that threaten these values. These include changes in water quality attributed to agricultural clearing (Bainbridge et al., 2018; Kroon et al., 2016), recurrent outbreaks of coral-eating crown-of-thorns starfish (which may also be due to water quality changes, Brodie et al., 2005), structural damage due to tropical cyclones and other storms (Massel and Done, 1993), and devastating coral bleaching events associated with climate change and marine heatwaves. In 2016, 2017, coral bleaching events caused the death of a substantial proportion (estimates vary from 10 to 50%) of coral on the Great Barrier Reef (Babcock et al., 2019; Hughes et al., 2018; Kennedy et al., 2018; Lewis and Mallela, 2018; Stuart-Smith et al., 2018). This was part of the wider global coral bleaching event that occurred between 2014 and 2017 (Eakin et al., 2019). In light of these threats, the Great Barrier Reef Marine Park Authority has recently assessed the outlook for the Reef as “very poor” (Great Barrier Reef Marine Park Authority, 2019). Policy and intervention options to preserve and restore the Great Barrier Reef and improve its resilience to ongoing climate change are now being urgently explored.

The need for marine models to support policy and management decisions for the Great Barrier Reef motivated the eReefs project, a collaboration between CSIRO, the Australian Institute of Marine Science, the Bureau of Meteorology, the Government of Queensland, and the Great Barrier Reef Foundation (Chen et al., 2011; Schiller et al., 2015). The eReefs project developed a tailored suite of nested hydrodynamic, sediment dynamic, and biogeochemical models that simulate physical conditions and water quality in three dimensions in near real-time throughout the Great Barrier Reef World Heritage Area and beyond (Baird et al., 2016a, 2016b, 2018; Herzfeld et al., 2016; Jones et al., 2016; Robson et al., 2013b, 2018). The eReefs models have been applied to investigations of the effects of ocean acidification on the Great Barrier Reef (Mongin and Baird, 2014; Mongin et al., 2016b), the effects of rivers on marine water quality (Margvelashvili et al., 2018; Robson et al., 2014; Wolff et al., 2018), the physical oceanography that drives the exposure of reefs to hot water (Schiller et al., 2015), and the interactions between physical conditions and biological mechanisms that lead to coral bleaching (Baird et al., 2018). The models are now being used to test the likely effects of interventions (e.g. Albright et al., 2016; Mongin et al., 2016a) and to help set new, more ecologically relevant targets for water quality and land runoff (Brodie et al., 2017).

Monitoring and adaptive management are not straightforward in the region because of the size of the Great Barrier Reef (>300,000 km2), its remoteness from major population centres, and the large inter-annual variability associated with the tropical climate of the region -- including cyclone events (Wolff et al., 2016) and effects of fluctuations in the El Nino Southern Oscillation Index (Leonard et al., 2016). The health and condition of the system is therefore monitored through a combination of quarterly water quality surveys, mostly in the inshore region (Waterhouse et al., 2018a), regular annual and alternate-year ecological surveys of reef (Thompson et al., 2018) and seagrass (McKenzie et al., 2018) habitats, in situ logging of the physical properties, fluorescence and turbidity of water at a few sites (Lynch et al., 2014), and opportunistic sampling of major flood plume events (e.g. Cherukuru et al., 2017; Howley et al., 2018). This in situ sampling regime is supplemented by satellite observation (Brando et al., 2015a, 2015b; Devlin et al., 2015) and more recently, by the eReefs marine models to provide synoptic, near real-time reporting of hydrodynamic and biochemical conditions on the GBR (Steven et al., 2019).

In this context of policy urgency, data sparsity, and politically contentious management and regulation, it is important to understand the performance of the models so that they can be used with confidence where we lack direct observational data, yet with due recognition of their limitations.

A traditional evaluation of the eReefs biogeochemical model against routine marine monitoring observations, following the principles laid out by Bennett et al. (2013), has been presented by Herzfeld et al. (2016) with respect to water quality and physical conditions, and Skerratt et al. (2019) with respect to plankton. These evaluations directly compare observed water quality at marine monitoring program (MMP) and Integrated Marine Observing System (IMOS) monitoring sites using time-series comparisons over several years (2011–2016), along with measures of bias, correlation, error and skill (Willmott et al., 1985) for each observed water quality and plankton variable at each monitoring site. Results generally indicate strong model performance, although this evaluation should be treated with caution for several reasons: a) monitoring sites are predominantly shallow inshore sampling, and thus represent only a small fraction of the area modelled; b) the water quality variables and sites used in the evaluation were the same as those used for model calibration, although evaluation extended beyond the calibration period and to more variables than used in the calibration (i.e. Secchi depth); c) many variables included in the model are not directly observed. These unobserved variables include, for instance, intracellular concentrations of carbon, nitrogen, phosphorus and chlorophyll within phytoplankton cells, concentrations of labile and refractory particulate organic material, and concentrations of small zooplankton.

Recently, Hipsey et al. (2020) proposed a new, four-level framework (hereafter referred to as the “Concept State Process System” or CSPS framework) for evaluation of aquatic ecosystem models. In brief, it describes:

  • Level 0 (Concept): The sanity check. Does the model produce plausible behaviour and reproduce key expected system properties?

  • Level 1 (State): Traditional evaluation achieved by comparing values of model state variables (e.g. nutrient and chlorophyll concentrations) with observed values of these state variables at matched times and locations. This evaluation should ideally include measures of error, bias and correlation for key state variables, with weighting functions that consider how the models will be used, and therefore which aspects of state variable responses are most important (Bennett et al., 2013).

  • Level 2 (Process): Evaluation of process rates and ecosystem function; for example, nutrient flux rates and system metabolism. While maximum process rates are typically represented in biogeochemical models as assigned parameter values (Bainbridge et al., 2018), actual process rates arise from interactions among state variables and environmental conditions, and these are rarely validated against observational data (Hipsey et al., 2020).

  • Level 3 (System): Evaluation of system-scale emergent properties; that is, properties and relationships that emerge from complex system dynamics in a way that is not obvious from simple inspection of the equations underlying the model. The behaviour of flocks of birds in flight has often been presented as a classic example of an emergent property of simple rules governing the individual behaviour of each animal (Phelan, 1999). When a complex, process-based model is able to reproduce emergent properties without additional calibration after Level 1 (State) validation, it gives confidence that the model is getting ‘the right results for the right reasons’ (Robson et al., 2017).

Here, we demonstrate for the first time an application of the CSPS framework to evaluate the eReefs marine biogeochemical model. We show how this approach can be used to highlight the weaknesses as well as the strengths of the model, which is important to build trust and encourage appropriate use.

We also synthesise a body of previously published and in-press work that describes the development of the eReefs marine models, their calibration and evaluation, and the application of these models to enhance understanding of drivers of water quality in the Great Barrier Reef World Heritage Area. In addition, we show several new evaluations that were necessary to complete the full CSPS model assessment and provide a narrative of the modelling and model delivery process.

Section snippets

Models

The eReefs marine models are a complex system of three-dimensional hydrodynamic, biogeochemical and sediment dynamics models driven by hydrological monitoring in 22 rivers feeding into the Great Barrier Reef, near real-time meteorological models, and global ocean models that provide the outer boundary conditions. The eReefs marine models are an implementation of the CSIRO Environmental Modelling Suite (EMS), which has previously been applied on a smaller scale in a range of coastal and estuary

Level 0: Conceptual evaluation

Check 1 (inception workshop, involving authors of this paper): A number of improvements to the model previously developed for application to estuarine ecosystems (e.g. Margvelashvili et al., 2003) were identified as important for achieving plausibility and credibility in its application to the Great Barrier Reef. Priority improvements included development of a coral metabolism and carbon cycle submodel (Mongin and Baird, 2014), development of a submodel to represent the nitrogen-fixing marine

Discussion

We have demonstrated the application of the CSPS model evaluation framework to a complex biogeochemical model. The framework allowed us to identify the aspects of the eReef model suite that agree well with observations, and those that require further work. It also provided an approach for model development. While traditional evaluation focuses on the comparison of observed and modelled state variable concentrations (e.g. chlorophyll a, total nitrogen and temperature), a more comprehensive model

Conclusions

The CSPS model evaluation framework demonstrated here provides a more systematic way than traditional model assessment approaches to consider multiple aspects of model performance. This framework may help to improve the evaluation of any complex mechanistic environmental model, especially where the number of free parameters, the computational requirements of the model, or the sparsity of data limit the application of robust parameter estimation, uncertainty analysis and/or independent model

Software and data availability

The CSIRO-EMS hydrodynamic, sediment and biogeochemical model software is available open source via https://github.com/csiro-coasts/EMS. Detailed science and user manuals can be found at https://research.csiro.au/cem/software/ems/ems-documentation/

Both near real-time and archived model outputs for the eReefs application of EMS are made freely available in standard netCDF format via https://research.csiro.au/ereefs/ to facilitate third-party scientific and technical applications. To support

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

eReefs simulations were developed as part of the eReefs project (http://ereefs.org.au/ereefs), a public-private collaboration between Australia's leading operational and scientific research agencies, government, and corporate Australia. Integrated Marine Observing System (IMOS) supplied IMOS mooring data. IMOS is supported by the Australian Government through the National Collaborative Research Infrastructure Strategy and the Super Science Initiative. The Marine Monitoring Program (MMP) managed

References (89)

  • R. Albright et al.

    Ocean acidification: linking science to management solutions using the Great Barrier Reef as a case study

    J. Environ. Manag.

    (2016)
  • G.B. Arhonditsis et al.

    Evaluation of the current state of mechanistic aquatic biogeochemical modeling

    Mar. Ecol. Prog. Ser.

    (2004)
  • R.C. Babcock et al.

    Severe continental-scale impacts of climate change are happening now: extreme climate events impact marine habitat forming communities along 45% of Australia's coast (vol 6, 411, 2019)

    Front. Mar. Sci.

    (2019)
  • Z. Bainbridge et al.

    Fine sediment and particulate organic matter: a review and case study on ridge-to-reef transport, transformations, fates, and impacts on marine ecosystems

    Mar. Pollut. Bull.

    (2018)
  • M. Baird et al.

    The use of mechanistic descriptions of algal growth and zooplankton grazing in an estuarine eutrophication model

    Estuar. Coast Shelf Sci.

    (2003)
  • M.E. Baird et al.

    A biophysical representation of seagrass growth for application in a complex shallow-water biogeochemical model

    Ecol. Model.

    (2016)
  • M.E. Baird et al.

    Remote-sensing reflectance and true colour produced by a coupled hydrodynamic, optical, sediment, biogeochemical model of the Great Barrier Reef, Australia: comparison with satellite data

    Environ. Model. Software

    (2016)
  • M.E. Baird et al.

    A mechanistic model of coral bleaching due to temperature-mediated light-driven reactive oxygen build-up in zooxanthellae

    Ecol. Model.

    (2018)
  • Baird, M.E., Wild-Allen, K.A., Parslow, J., Mongin, M., Robson, B., Skerratt, J., Rizwi, F., Soja-Woznaik, M., Jones,...
  • H. Barnes et al.

    A statistical examination of the catches, numbers, and biomass taken by three commonly used plankton nets

    Mar. Freshw. Res.

    (1965)
  • N.D. Bennett et al.

    Characterising performance of environmental models

    Environ. Model. Software

    (2013)
  • V. Brando et al.

    High-resolution satellite turbidity and sea surface temperature observations of river plume interactions during a significant flood event

    Ocean Sci.

    (2015)
  • V.E. Brando et al.

    Reef Rescue Marine Monitoring Program

    (2015)
  • R.J.W. Brewin et al.

    Mapping size-specific phytoplankton primary production on a global scale

    J. Maps

    (2010)
  • J. Brodie et al.

    Development of basin-specific ecologically relevant water quality targets for the Great Barrier Reef

    (2017)
  • J. Brodie et al.

    Are increased nutrient inputs responsible for more outbreaks of crown-of-thorns starfish? An appraisal of the evidence

    Mar. Pollut. Bull.

    (2005)
  • N.J. Car

    The eReefs information architecture

  • F. Carlotti et al.

    Modeling Zooplankton Dynamics, ICES Zooplankton Methodology Manual

    (2000)
  • Y. Chen et al.

    eReefs - a new perspective on the Great Barrier Reef

  • N. Cherukuru et al.

    Impact of wet season river flood discharge on phytoplankton absorption properties in the southern Great Barrier Reef region coastal waters

    Estuar. Coast Shelf Sci.

    (2017)
  • Deloitte Access Economics

    Economic contribution of the Great Barrier Reef

    (2013)
  • M. Devlin et al.

    Mapping the pollutants in surface riverine flood plume waters in the Great Barrier Reef, Australia

    Mar. Pollut. Bull.

    (2012)
  • M.J. Devlin et al.

    Water quality and river plume monitoring in the Great Barrier Reef: an overview of methods based on ocean colour satellite data

    Rem. Sens.

    (2015)
  • C.M. Eakin et al.

    The 2014–2017 global-scale coral bleaching event: insights and impacts

    Coral Reefs

    (2019)
  • M.M. Edson et al.

    Emergent properties" and ecological research

    Am. Nat.

    (1981)
  • K.E. Fabricius et al.

    Changes in water clarity in response to river discharges on the Great Barrier Reef continental shelf: 2002-2013

    Estuar. Coast Shelf Sci.

    (2016)
  • P. Gillibrand et al.

    A mass-conserving advection scheme for offline simulation of scalar transport in coastal ocean models

    Ocean Model.

    (2016)
  • Great Barrier Reef Marine Park Authority

    Great Barrier Reef Outlook Report 2019. GBRMPA: Townsville

    (2019)
  • M. Herzfeld et al.

    eReefs Marine Modelling: Final Report

    (2016)
  • M. Herzfeld et al.

    Active open boundary forcing using dual relaxation time-scales in downscaled ocean models

    Ocean Model.

    (2015)
  • M.R. Hipsey et al.

    A system of metrics for the assessment and improvement of aquatic ecosystem models

    Environ. Model. Software

    (June 2020)
  • T. Hirata et al.

    Synoptic relationships between surface Chlorophyll-a and diagnostic pigments specific to phytoplankton functional types

    Biogeosciences

    (2011)
  • C. Howley et al.

    Assessment of water quality from the Normanby River catchment to coastal flood plumes on the northern Great Barrier Reef, Australia

    Mar. Freshw. Res.

    (2018)
  • T. Hughes et al.

    Large‐scale bleaching of corals on the Great barrier reef

    Ecology

    (2018)
  • A.J. Jakeman et al.

    Ten iterative steps in development and evaluation of environmental models

    Environ. Model. Software

    (2006)
  • J.H. Janse et al.

    Estimating the critical phosphorus loading of shallow lakes with the ecosystem model PCLake: sensitivity, calibration and uncertainty

    Ecol. Model.

    (2010)
  • E.M. Jones et al.

    Use of remote-sensing reflectance to constrain a data assimilating marine biogeochemical model of the Great Barrier Reef

    Biogeosciences

    (2016)
  • E.V. Kennedy et al.

    Coral bleaching in the southern inshore Great barrier reef: a case study from the Keppel islands

    Mar. Freshw. Res.

    (2018)
  • F.J. Kroon et al.

    Towards protecting the Great Barrier Reef from land‐based pollution

    Global Change Biol.

    (2016)
  • N. Leonard et al.

    Evidence of reduced mid‐holocene ENSO variance on the Great barrier reef, Australia

    Paleoceanography

    (2016)
  • S.C. Lewis et al.

    28. A multifactor risk analysis OF the record 2016 great barrier reef bleaching: anthropogenic greenhouse gases likely increased the risk of the extreme Great Barrier Reef bleaching event through anomalously high sea surface temperature and the accumulation of thermal stress

    Bull. Am. Meteorol. Soc.

    (2018)
  • S.S. Lewis et al.

    Synthesizing Historical Land Use Change, Fertiliser and Pesticide Usage and Pollutant Load Data in the Regulated Catchments to Quantify Baseline and Changing Pollutant Loads Exported to the Great Barrier Reef

    (2014)
  • S. Libralato et al.

    Bridging biogeochemical and food web models for an End-to-End representation of marine ecosystem dynamics: the Venice lagoon case study

    Ecol. Model.

    (2009)
  • C. Lønborg et al.

    Organic matter bioavailability in tropical coastal waters: the Great Barrier Reef

    Limnol. Oceanogr.

    (2018)
  • Cited by (10)

    • Reef state and performance as indicators of cumulative impacts on coral reefs

      2021, Ecological Indicators
      Citation Excerpt :

      We only detected negative impacts of WQ (low salinity and elevated River DIN) in Performance metrics affecting only few coral groups in Central GBR when WQ metrics were evaluated as long-term averages. While we assess a very comprehensive dataset for WQ metrics for the GBR, these are modelled products rather than in-situ observations, which limits their accuracy and potential explanatory power as some metrics may under-estimate real concentrations as well as under-estimating WQ variability (Skerratt et al., 2019; Robson et al., 2020). Despite this, WQ metrics appeared in competing models evaluating Performance (albeit low explanatory power), suggesting they are likely to be better metrics for assessing the rate of cover increase potential than State (i.e., cover).

    View all citing articles on Scopus
    View full text