Global Sensitivity Analysis Using the Ultra-Low Resolution Energy Exascale Earth System Model

Abstract

For decades, Arctic temperatures have increased twice as fast as average global temperatures. As a first step toward quantifying parametric uncertainty in Arctic climate, we performed a variance-based global sensitivity analysis (GSA) using a fully coupled, ultra-low resolution (ULR) configuration of version 1 of the U.S. Department of Energy's Energy Exascale Earth System Model (E3SMv1). Specifically, we quantified the sensitivity of six quantities of interests (QOIs), which characterize changes in Arctic climate over a 75 year period, to uncertainties in nine model parameters spanning the sea ice, atmosphere, and ocean components of E3SMv1. Sensitivity indices for each QOI were computed with a Gaussian process emulator using 139 random realizations of the random parameters and fixed preindustrial forcing. Uncertainties in the atmospheric parameters in the Cloud Layers Unified by Binormals (CLUBB) scheme were found to have the most impact on sea ice status and the larger Arctic climate. Our results demonstrate the importance of conducting sensitivity analyses with fully coupled climate models. The ULR configuration makes such studies computationally feasible today due to its low computational cost. When advances in computational power and modeling algorithms enable the tractable use of higher-resolution models, our results will provide a baseline that can quantify the impact of model resolution on the accuracy of sensitivity indices. Moreover, the confidence intervals provided by our study, which we used to quantify the impact of the number of model evaluations on the accuracy of sensitivity estimates, have the potential to inform the computational resources needed for future sensitivity studies.

Key Points

• We perform the first global sensitivity analysis using the fully coupled ultra-low resolution Energy Exascale Earth System Model

• Uncertainty in cloud physics parameters is found to most greatly impact Arctic climate predictions

• Our inferred quantity of interest parameter correlations uncover key physical feedbacks and can guide model tuning

Plain Language Summary

Feedbacks associated with Arctic warming are consequential for both the region and the strongly coupled global climate system. To assess the variability of the impacts of global warming and associated feedbacks in model-based predictions, we quantified the sensitivity of the Arctic climate state to nine uncertain variables parameterizing the U.S. Department of Energy's global climate model known as the Energy Exascale Earth System Model (E3SM). Because the computational cost of repeatedly running high-resolution configurations of E3SM was prohibitive, we used an ultra-low resolution (ULR) configuration as a physics-based surrogate for sensitivity analysis. Our first ever global sensitivity study of version 1 of the E3SM identified that the atmospheric parameters in E3SM's cloud physics model had the most impact on the atmosphere, sea ice, and ocean quantities of interest. This result demonstrates the importance of fully coupled climate analyses, which are necessary to identify such cross-component influences. While we constructed confidence intervals that quantify the error in our estimates of parameter sensitivity introduced by using a limited number of ULR E3SM model runs, future investigation is needed to quantify the impact of resolution on error.

1 Introduction

Understanding the impact of warming on the Arctic is important because regional events can lead to high-consequence global changes (Bathiany et al., 2016; Lenton, 2008, 2012), including tipping points (irreversible changes in the global climate system (Lenton, 2008; Peterson et al., 2020)). Melting of the Greenland ice sheet will result in global sea level rise, with risks to coastal infrastructure (Graeter et al., 2018). Sea ice loss will lead to increased maritime activity and possibly geopolitical conflict as more nations vie for access to the region (L. C. Smith & Stephenson, 2013). In addition, there is evidence that loss of sea ice and Arctic warming can induce changes in midlatitude weather and precipitation (Cohen, Pfeiffer, & Francis, 2018; Cohen, Zhang, et al., 2018; Cvijanovic et al., 2017), potentially leading to food and water shortages (Parry et al., 2001).

According to recent data (IPCC, 2021; Richter-Menge et al., 2019; Snow, Water, Ice, and Permafrost in the Arctic (SWIPA), 2017), the Arctic is warming at more than twice the rate of the rest of the globe. This accelerated Arctic warming leads to changes in a variety of physical systems influencing Arctic climate. For instance, the well-known ice-albedo feedback effect has been shown to contribute to sea ice loss. As highly reflective sea ice is lost, the surface albedo is reduced and solar radiation absorption in the darker ocean water is increased (Goosse et al., 2018). This positive feedback is counteracted by a negative feedback mechanism whereby thinner sea ice grows more quickly in response to thermodynamic forcing from the ocean and atmosphere. Permafrost thaw is increasing greenhouse gas release, thereby increasing warming (Parazoo et al., 2018; Schuur et al., 2015). Both sea ice and land ice melt are increasing freshwater flux into the North Atlantic, which can lead to ocean current disruptions and further changes to climate (Sevellec et al., 2017).

As a first step toward identifying possible tipping events stemming from climate change-driven processes in the Arctic with quantified uncertainty, we present a global sensitivity analysis (GSA) of climate projections of version 1 of the U.S. Department of Energy's fully coupled Energy Exascale Earth System Model (E3SMv1). To motivate the main contributions of this paper, we first provide a brief overview of related past work, focusing on studies aimed at addressing the sensitivity of Earth System Model (ESM) components and coupled models to various model parameters.

1.1 Overview of Related Work

Recent years have seen a number of studies aimed at understanding the sensitivity of various climate models to relevant parameters. The vast majority of this work has focused on individual components of a global ESM, for example, the ocean, sea ice, and atmosphere components. Several authors have investigated the sensitivity of ocean models to parameters, most of them examining subgrid mixing parameterizations, wind drag, model domain and grid resolution, and numerical formulations and topography (Alexanderian et al., 2012; Asay-Davis et al., 2018; Bernard et al., 2006; Hurlburt & Hogan, 2000; Maltrud & McClean, 2005; M. Hecht & Smith, 2008; M. W. Hecht et al., 2008; Reckinger et al., 2015). A handful of studies have examined the sensitivity of model predictions to model parameters in stand-alone configurations of sea ice models, including (Kim et al., 2006; Peterson et al., 2010; Uotila et al., 2012; Urrego-Blanco et al., 2016). In the most recent of these works (Urrego-Blanco et al., 2016), Urrego-Blanco et al. conducted a comprehensive sensitivity analysis of sea ice thickness and area to 39 sea ice model parameters using Sobol sequences together with a fast emulator for the Los Alamos sea ice model, CICE (Community Ice CodE) (Hunke et al., 2015). Similar sensitivity studies have been done for stand-alone atmosphere models, for example (Covey et al., 2013; Guo et al., 2014; Qian et al., 2018; Rasch et al., 2019; Zhao et al., 2013). Zhao et al. (2013) evaluated the sensitivity of radiative fluxes at the top of the atmosphere to various cloud microphysics and aerosol parameters. Covey et al. (2013) used Morris one-at-a-time (MOAT) screening to estimate sensitivity with respect to 27 atmospheric parameters. Qian et al. (2018), estimated the sensitivity of the model fitness of generalized linear models (GLMs) of response variables obtained from short (3-day) simulations of a 1° resolution E3SM atmosphere model (EAM) with respect to 18 parameters from various parts of the atmospheric dycore, including parameterizations of deep convection, shallow convection, and cloud macro/microphysics. Guo et al. (2014), used GLMs to determine the most influential parameters of the Cloud Layers Unified by Binormals (CLUBB) physics parameterization within the single-column version of the Community Atmosphere model version 5 (SCAM5). In related recent work focused on the EAM, Rasch et al. (2019) demonstrated the utility of using lower-resolution versions of the EAM atmospheric component and short-term hindcast to guide tuning and sensitivity analysis of higher-resolution models.

While the aforementioned studies provide much insight into individual ESM components, without considering a fully coupled ESM, it is impossible to identify the interaction among various climate components. Hence, studies focusing on a single climate component have the danger of significantly overlooking relevant climate feedbacks. Performing sensitivity studies on fully coupled climate models is far more challenging than considering an individual climate component. The main hurdle is the fact that running a fully coupled ESM is far more computationally expensive than running a single climate component. Since sensitivity studies typically require many simulation ensembles, sensitivity analyses using fully coupled models are typically intractable without the use of efficient surrogates, especially at “production” grid resolutions. The authors are aware of only one reference focusing on a sensitivity study involving several climate components using a fully coupled ESM, namely (Urrego-Blanco et al., 2019). In Urrego-Blanco et al. (2019), Urrego-Blanco et al. use the 1° resolution of the E3SM v0-HiLAT (EHV0) fully coupled climate system (developed for the simulation of high-latitude processes) to identify emerging relationships between sea ice area, net surface longwave radiation, and atmospheric circulation over the Beaufort gyre. The authors consider five model parameters, two from the atmosphere model (version 5 of the Community Atmosphere Model, or CAM5 (Dennis et al., 2012)), two from the sea ice model (version 5 of the Los Alamos Sea Ice Model, or CICE5 (Hunke et al., 2015)) and one from the ocean model (version 2 of the Parallel Ocean Program, or POP2 (R. Smith et al., 2010)), and initialize their model using preindustrial forcing. By employing an elementary effects or MOAT method (Morris, 1991) for their sensitivity analysis (an approach that perturbs one input parameter at a time, rather than all parameters together), the authors are able to keep the number of ensemble members (or E3SM simulations) required down to just 24.

It is worthwhile to note that there are other works utilizing global climate models for sensitivity analyses targeting a single climate component. For instance, the authors of Rae et al. (2014) perform a sensitivity study of the sea ice simulation within the global coupled climate model HadGEM3. Here, both the Arctic and Antarctic are considered. In a similar vein, Uotila et al. (2012) explore the sensitivity of the global sea ice distribution of the Australian Climate Ocean Model (AusCOM) to a range of sea ice physics-related parameters within a global ocean-ice model comprised of AusCOM coupled with the Los Alamos CICE model. While studies such as these have the advantage of incorporating feedbacks from the global climate system, they have a similar limitation of single-component sensitivity studies in that they preclude the identification of cross-component parameter interactions.

1.2 Contributions and Organization

Our present work was motivated by the recent study in Urrego-Blanco et al. (2019), but differs in several important ways. First, we considered version 1 of the E3SM (referred to herein as E3SMv1). Second, we employed a much lower spatial resolution grid than those considered in Urrego-Blanco et al. (2019). We refer to our resolution model as the “ultra-low resolution” (ULR) model, which corresponds to a 7.5° grid resolution in the atmosphere and a 240 km grid resolution for the ocean and sea ice. The ULR configuration of the E3SM was originally created primarily to enable rapid turnaround testing, and was recently used to develop an approach for ensuring statistical reproducibility of climate model simulations on a variety of conventional as well as hybrid architectures (Mahajan et al., 2019a, 2019b). In contrast, our primary objective here was to investigate for the first time the ULR E3SM's skill as a physics-based surrogate of the fully coupled E3SM. Employing the ULR configuration, which is more than 100 times less expensive to run than the “standard” 1° configuration, enabled us to reduce the computational cost of our sensitivity analysis, so that we could run enough simulations to identify important parameters with sufficient statistical confidence. Consequently, we were able to consider more parameters and employ more sophisticated sensitivity analysis approaches than the method used in Urrego-Blanco et al. (2019). Performing the same study using higher-resolution configurations of the E3SM (e.g., the 1° configuration) is currently prohibitive even with access to large distributed computing systems due to the large computational cost of running the model at these resolutions (see the beginning of Section 3 for more details).

Our variance-based GSA studied the impact of nine parameters, spanning three E3SM components, the sea ice model (MPAS-SeaIce; Turner et al., 2022), the EAM (Rasch et al., 2019), and the ocean model (MPAS-Ocean; Petersen et al., 2018), on six Arctic-focused quantities of interests (QOIs). To maximize the accuracy of our estimates of sensitivity, we constructed Gaussian process emulators for these QOIs using the PyApprox library (J. D. Jakeman, 2022) and 139 75-year ensemble runs of the fully coupled ULR E3SMv1. Each simulation was initialized from a spun-up initial condition generated specifically for this study (a spun-up initial condition was not readily available at the considered resolution) and forced with preindustrial control conditions. Using each emulator, we calculated the Sobol, main effect, and total effect sensitivity indices of our nine parameters. Main effect indices were used to quantify the effects of single parameters acting in isolation, and Sobol and total effect indices were used to identify strong parameter interactions.

The 139 ensemble runs used in this study exhibited significant variability, with several runs resulting in the complete loss of Arctic sea ice and several other runs exhibiting an apparent exponential growth in the amount of Arctic sea ice. The main takeaway from our study is that the parameters in the cloud physics parameterization within the atmosphere component of the E3SMv1 have the most impact on the Arctic climate state. Our study identified several relationships between QOIs, which matched physics-based intuition (e.g., ensemble members with low sea ice extent had high surface air temperature), and led to plausible conclusions regarding feedback processes important to the Arctic climate state (e.g., seasonal cloud convective regimes can create a feedback that affects Arctic sea ice extent). These results suggest that the ULR configuration is a plausible physics-based surrogate for the coupled climate state. By constructing univariate functions through a marginalization of all but a single parameter, we were additionally able to determine whether increasing/decreasing a given parameter will increase or decrease a given QOI. These results are useful in guiding model spin-ups and are consistent with the parameter-QOI correlations uncovered by our manual spin-up of the ULR E3SMv1.

The remainder of this paper is organized as follows. We detail the methods employed in this study in Section 2. This includes a description of our coupled model (E3SMv1) and our tuning of the ULR configuration of this model, some comparisons of our model with observational data and the 1° resolution E3SM, and a discussion of the design and implementation of our global sensitivity study using this coupled model. In Section 3, we present the main results of our global sensitivity study applied to the ULR E3SMv1, and provide a discussion of their significance. We end with a concluding summary (Section 4).

2 Methods

2.1 E3SMv1 Coupled Climate Model

In the present study, E3SMv1 was used to investigate changes in Arctic sea ice in response to internal variability related to ocean and atmosphere modes as well as in response to perturbations in the model parameters. E3SM consists of component models for the atmosphere, ocean, ice, land, and river transport. The EAM (Rasch et al., 2019) has a spectral element dynamical core discretized on a cubed sphere grid using 72 vertical levels. The standard resolution E3SM configuration uses a 1° grid for both EAM and the E3SM Land Model (ELM; Bisht et al., 2018), which corresponds to approximately 110 km at the equator. The ocean and sea ice models are based on the Model for Prediction Across Scales (MPASs) framework (Heinzeller et al., 2016). MPAS-Ocean (Petersen et al., 2018) uses a finite volume discretization on an unstructured Voronoi grid, which is shared with MPAS-SeaIce (Turner et al., 2022). At the standard resolution, the ocean and sea ice grid have a resolution varying between 60 km at midlatitudes and 30 km at the poles. The Model for Scale Adaptive River Transport (MOSART) (Cornette, 2012) is also employed, and has a resolution of 50 km.

The present study was based on an ULR configuration of E3SMv1, designed for rapid turnaround testing of the fully coupled E3SM. We chose an ULR configuration as it would provide a computationally tractable way to generate larger numbers of ensemble runs to explore the parameter space in the coupled model. This ULR configuration has a grid resolution of approximately 7.5° for EAM and ELM and 240 km, or approximately 2.2° for MPAS-Ocean and MPAS-SeaIce. Plots of the ULR grids employed in this study are provided in Figure 1. It is noted that, while the atmospheric resolution within the ULR E3SM is quite coarse (Figure 1a), the MPAS-Ocean and MPAS-SeaIce grids employed in this resolution are more realistic (see Figure 1b). To quantify the computational advantages of the ULR configuration, we note that it achieves approximately 4 simulated years per day per node on the Skybridge cluster (described in Section 2.5), in comparison to 0.035 simulated years per day per node for the 1° standard resolution configuration of E3SM. This results in an estimate that the ULR configuration is more than 100 times faster than the standard resolution configuration.

Figure 1

Ultra-low resolution grid for atmosphere (a) and ocean (b) used in our Energy Exascale Earth System Model study.

In the following section, we assess the predictive performance of the ULR E3SM. We find that ULR predictions capture the large scale features of the 1° model, which suggests that the ULR model can help inform sensitivity analysis and uncertainty quantification (UQ) of higher-resolution models.

2.2 E3SMv1 Ultra-Low Configuration Tuning

For our ULR simulations, we first performed a spin-up (i.e., running the model until an equilibrium state is achieved) using preindustrial control (piControl) forcing for 500 simulated years with default parameter values. It is desirable at the end of the spin-up to have a near-zero long-term average net top-of-atmosphere (TOA) energy flux, a constant global average mean surface air temperature, and stable yearly sea ice coverage in order to initialize the perturbed runs with a stable state. Our original 500-year spin-up simulation exhibited a warm bias, with surface temperature elevated, compared with observations and declining sea ice over the 500-year period (see Figure 2). To improve the model tuning, we ran an additional 180 years starting from year 500 of the spin-up simulation using atmospheric parameter values modified to match the final tuning from the Golaz et al., paper (Golaz et al., 2019). Parameter values are given in Table 1.

Figure 2

Yearly averaged global surface air temperature (°C) (a), yearly averaged net flux at top-of-atmosphere (W/m²) (b), and yearly averaged sea ice extent (10⁶ km²) (c). The blue line is from the 500-year ultra-low resolution model spin-up with default parameter values and the red line is from the 180-year branch run with modified parameter values as shown in Table 1. Bold lines indicate linear trends.

Table 1. Default Atmospheric Parameter Values for Ultra-Low Resolution Configuration and Corresponding Values From Golaz et al. (2019)

Parameter	Default value	Golaz et al. value
zmconv_ke	1.5 × 10^−6	5.0 × 10^−6
so4_sz_thresh_icenuc	7.53 × 10^−8	5.0 × 10^−8
clubb_c14	1.3	1.06

• Note. In this table, zmconv_ke is the coefficient for evaporation of convective precipitation, so4_sz_thresh_icenuc is the Aitken mode SO₄ size threshold used for homogeneous ice nucleation, and clubb_c14 is the damping coefficient for u′² and v′² in the Cloud Layers Unified by Binormals (Larson, 2020) aerosol physics parameterization.

The branch run with the Golaz et al. values did result in a more realistic climate, with improvements in the linear trends of surface temperature, net TOA flux, and sea ice extent. In Figure 2, time series plots of these quantities for the 500-year spin-up using default parameter values are shown in blue, with the final 180-years from the simulation with modified parameter values shown in red. Bold lines indicate linear trends over the years 26 through 500 for the initial spin-up and years 526 through 680 for the branch run. Trends are much closer to zero for the branch run, with a slope of −0.00082 for surface temperature, a slope of 0.0005 for net top of atmosphere flux, and a slope of 0.0012 for Arctic sea ice extent over the time range.

Year 675 of this branch run was used as the initial condition for all perturbed sensitivity analysis simulations as well as for a baseline simulation that continued with the same parameter values for an additional 75 years. The linear trends were small enough for the branch run that the values of our selected QOIs did not exhibit significant drift over the 75 years. Investigations of the impact of the equilibrium values of the initial state on sensitivity analysis results are beyond the scope of this study, but this could be addressed in future work using additional tunings of the ULR model informed by our results involving the marginalized main effect indices (Section 3.5).

To investigate the performance of the ULR configuration as a physically based surrogate model of the standard resolution, we computed the climatology of the branched run over years 526–680 and compared to observational data climatologies as well as to the average of the 1° E3SMv1 model over 500 years with preindustrial control forcing. The 1° resolution E3SMv1 simulations have been scientifically validated and provide a reference for these quantities in the ULR simulation (Golaz et al., 2019). The ULR model is not expected to capture the small-scale variations and regional-scale processes simulated with higher-resolution models, but large-scale patterns should be represented.

Figure 3 plots the global annual average top of model net flux (W/m²) for the ULR branched run climatology in comparison with the 1° resolution climatology as well as observational data from CERES-EBAF Ed4.1 (Loeb et al., 2018). Figure 4 plots the global annual average total precipitation (mm/day) for the ULR and 1° simulations in comparison with the ERA-Interim reanalysis (Dee et al., 2011) fields. In both cases, we see that, although the ULR simulation does not capture small-scale features seen in the higher-resolution simulation, the large-scale patterns are similar. This behavior is also evident in zonal means. Figures 5 and 6 plot zonal means for the temperature and zonal winds in comparison with ERA-Interim reanalysis products, demonstrating the vertical variation in the atmosphere. Given the warm bias after our spin-up, it is not surprising that zonal temperature shows the most divergence from the observations.

Figure 3

Top of atmosphere flux (W/m²) for: (a) years 526–675 of the branched ultra-low resolution spin-up simulation, and (b) years 1–500 of the 1° preindustrial control, compared with CERES-EBAF Ed4.1 data (Loeb et al., 2018). Top panel shows the model result, middle panel shows the observational data, and bottom panel shows the difference.

Figure 4

Total precipitation (mm/day) for: (a) years 526–675 of the branched ultra-low resolution spin-up simulation, and (b) for years 1–500 of the 1° preindustrial control, compared with ERA-Interim data (Dee et al., 2011). Top panel shows the model result, middle panels shows the observational data, and bottom panel shows the difference.

Figure 5

Zonal temperature (°C) for: (a) years 526–675 of the branched ultra-low resolution spin-up, and (b) for years 1–500 of the 1° preindustrial control simulation, compared with ERA-Interim data (Dee et al., 2011). Top panel shows the model result, middle panels shows the observational data, and bottom panel shows the difference.

Figure 6

Zonal wind (a), for years 526–675 of the branched ultra-low resolution spin-up compared with ERA-Interim data (b) for years 1–500 of the 1° preindustrial control simulation compared with ERA-Interim data. Top panel shows the model result, middle panels shows the observational data, and bottom panel shows the difference.

Figure 7

Global sensitivity analysis workflow. Here, N denotes the total number of perturbed Energy Exascale Earth System Model (E3SM) simulations launched, and M ≤ N is the number of runs that completed successfully (simulated the global climate state to time T_final).

Figure 8

Ensemble trajectories of the quantities of interests in Table 3 (SIE (a), SIV (b), SST (c), TS (d), FLNS (e), CLDLOW (f)) for the ensemble members that made it to year 75. The baseline run is distinguished from the others by the red markers.

Figure 9

Box-and-whiskers plots showing ensemble statistics for the first six quantities of interests from Table 3 (SIE (a), SIV (b), SST (c), TS (d), FLNS (e), CLDLOW (f)). The red central mark indicates the median of the data, whereas the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. Outliers are plotted using the “+” symbol.

2.3 Design of Global Sensitivity Study (GSA)

The first step in designing a sensitivity study, given a spun-up initial condition, is selecting the set of parameters (which will be denoted by {z_i}) to be perturbed, together with the set of relevant QOIs on which the parameters are expected to have an effect. A description of the parameters, their baseline values, and the range of their perturbed values is given in Table 2. The parameters were chosen based on their significance in previous sensitivity studies involving both individual components as well as fully coupled climate simulations, most notably (Asay-Davis et al., 2018; Qian et al., 2018; Rasch et al., 2019; Reckinger et al., 2015; Urrego-Blanco et al., 2016, 2019). Of the nine parameters, three are from the sea ice component (MPAS-SeaIce), two are from the ocean component (MPAS-Ocean), and four are from the atmosphere model (EAM)—more specifically, the CLUBB (Larson, 2020) turbulence closure and cloud physics parameterization within EAM.

Table 2. Global Sensitivity Analysis Parameters

Component	Variable	Parameter	Baseline	Min	Max	Description (units)	Short name
MPAS-SeaIce	z1	ksno	3.0 × 10^−1	2.0 × 10^−1	6.0 × 10^−1	Snow conductivity (Wm^−1K^−1)	ksno
	z2	lambda_pond	1.1574 × 10^−6	1.15 × 10^−8	1.15 × 10^−4	Drainage timescale of ponds (s^−1)	lambda_pond
	z3	dragio	5.36 × 10^−3	2.0 × 10^−4	1.6 × 10^−1	Ocean-ice drag (−)	dragio
EAM	z4	cldfrc_dp1	4.5 × 10^−2	2.0 × 10^−2	1.0 × 10^−1	Deep convection cloud fraction parameter in CLUBB (−)	cldfrc_dp1
	z5	clubb_c1	1.335	1.0	5.0	Constant associated with dissipation of variance w^2 in CLUBB (−)	clubb_c1
	z6	clubb_c8	4.3	2.0	8.0	Constant associated with Newtonian damping of w^3 in CLUBB (−)	clubb_c8
	z7	gamma_coeff	3.2 × 10^−1	1.0 × 10^−1	5.0 × 10^−1	Constant width of probability density function in w coord in CLUBB (−)	gamma_coeff
MPAS-Ocean	z8	standardgm_tracer_kappa	1.8 × 10^3	6.0 × 10^2	1.8 × 10^3	Bolus coefficient of GM parameterization of eddy transport (m^2/s)	GM_bolus_kappa
	z9	cvmix_kpp_critical_bulkrichardsonnumber	2.5 × 10^−1	2.0 × 10^−1	1.0	Bulk Richardson number used in KPP vertical mixing scheme (−)	crit_bulk_rich

Our GSA is based upon random realizations of the nine parameters, randomly selected from a uniform distribution over the ranges defined by the “Min” and “Max” values given in Table 2. The sampling and associated model evaluations were managed using the DAKOTA library (Adams et al., 2013), an open-source software package for optimization, UQ, and advanced parametric analysis. Much like the parameters themselves, the selection of the parameter ranges was guided by past analyses (Asay-Davis et al., 2018; Qian et al., 2018; Rasch et al., 2019; Reckinger et al., 2015; Urrego-Blanco et al., 2016, 2019). It is worthwhile to note that the two MPAS-SeaIce parameters selected in our GSA were hard-coded to their default values in the master branch of the E3SM. In order to enable the straightforward specification of these parameters in the relevant input file, a fork of the E3SM was created and used in the present study. Instructions for cloning this fork as well as building the code and submitting a perturbed run are provided in Appendix B of Peterson et al. (2020).

In the present study, we report sensitivity metrics for a set of six QOIs, summarized in Table 3. This set of QOIs is selected for several reasons, including: (a) their overlap with QOIs considered in similar past works (Rasch et al., 2019; Urrego-Blanco et al., 2019) (to enable comparisons), (b) their importance and relevance to studying the Arctic climate state (e.g., the CLDLOW QOI, which represents low cloud coverage, is selected because low clouds are particularly important in the Arctic and may impact sea ice coverage), and (c) the fact that they span the three climate components targeted by this study (sea ice, ocean, and atmosphere). Following the approach in (Urrego-Blanco et al., 2016, 2019), we look at the QOIs in Table 3 annually as well as seasonally.

Table 3. Global Sensitivity Analysis Quantities of Interests (QOIs)

QOI	Units	Description	Component
SIE	km^2	Total Arctic sea ice extent	Sea ice
SIV	km^3	Total Arctic sea ice volume	Sea ice
SST	°C	Sea surface temperature averaged over 60°–90°N	Ocean
TS	°C	Surface air temperature averaged over 60°–90°N	Atmosphere
FLNS	W/m^2	Net longwave flux at surface over 60°–90°N	Atmosphere
CLDLOW	−	Low cloud coverage below 700 hPa averaged over 60°–90°N	Atmosphere

Note that we originally obtained results for a larger set of QOIs than those summarized in Table 3, as discussed in (Peterson et al., 2020). Specifically, we considered five additional QOIs: the surface air specific humidity averaged over 60°–90° (QS), the large-scale snow precipitation averaged over 60°–90° (PRECSL), and the mean sea level pressure over the Beaufort Sea, the Aleutian Low, and the Siberian High (BH, AL, and SH, respectively). We omit these results here largely for the sake of brevity. The former two QOIs (QS and PRECSL) were highly correlated with other QOIs, so including those results would not add much to the discussion. Additionally, our sensitivity analysis results for the latter three QOIs (BH, AL, and SH) precluded us from making strong conclusions about the impact of parameter variations on these QOIs, as the relevant ensemble trajectories resembled white noise (indicating there was no clear signal) and high errors in the sensitivity indices were observed.

Each perturbed simulation in our study was run up to time T_final and was given a spin-up period of T_spin-up < T_final to equilibrate the simulation (i.e., to get past the inevitable transient period that occurs when the run commences). Here, we prescribed a spin-up period of 50 years (T_spin-up = 50 years), and each perturbed model configuration was run until time T_final = 75 years. In general, it is not expected for all the perturbed simulations to run to completion, and indeed crashes (discussed in more detail in Section 3) occurred for a handful of our runs. For the successful runs (runs that made it to year 75), our six QOIs were calculated by averaging annually and seasonally over the last 25 years of the simulations (i.e., between times t = T_spin-up + 1 and T_final).

As discussed earlier in Sections 2.1 and 2.2, the GSA study performed herein used the ULR configuration of the E3SMv1 and preindustrial (piControl) forcing. Repeating the study with a different forcing, such as one of the forcings in Golaz et al. (2019) and Eyring et al. (2016), would be an interesting and useful follow-on exercise to the present study.

2.4 Variance-Based Global Sensitivity Analysis

In this section, we describe the variance-based GSA used to determine the relative sensitivity of model predictions to uncertain model parameters.

2.4.1 Sobol Indices

In this paper, Sobol sensitivity indices (Sobol, 2001) are used to quantify the relative importance of parameter combinations on a given QOI. With this goal, let f denote a model output QOI that depends on some model parameters . Any function f with finite variance parameterized by a set of independent variables z with probability distribution and support can be decomposed into the following finite sum, referred to as the analysis of variance (ANOVA) decomposition,

(1)

or more compactly

(2)

where quantifies the dependence of the function f on the variable dimensions i ∈ u and .

The functions can be obtained by integration, specifically

(3)

where and . The first-order terms , ‖u‖₀ = 1 represent the effect of a single variable acting independently of all others. Similarly, the second-order terms ‖u‖₀ = 2 represent the contributions of two variables acting together, and so on.

The terms of the ANOVA expansion are orthogonal, that is, the weighted L² inner product , for u ≠ v. This orthogonality facilitates the following decomposition of the variance of the function f

(4)

where dρ_u(z) = ∏_j∈udρ_j(z).

Two popular measures of sensitivity are the main effect and total effect indices given respectively by

(5)

where e_i is the unit vector, with only one nonzero entry located at the ith element, and . Main effect values represent the expected decrease in variance obtained from observing z_i. The total effects measure the variance that remains after learning the values of every variable except z_i. In the following, we also report Sobol indices (Sobol, 2001)

which measures the contribution of the interaction between the parameter subset u on the variance of the function f.

Note that three aforementioned quantities (Sobol indices, main effect indices, and total effect indices) measure some aspect of global sensitivity. In particular, they reflect a variance attribution over the range of the input parameters, as opposed to the local sensitivity reflected by a derivative.

2.4.2 Gaussian Process

The Sobol indices (Equation 4) can be computed using a number of different methods, for example, via (Quasi) Monte Carlo sampling (Saltelli et al., 2010), using surrogates (such as polynomial chaos expansions (Sudret, 2008)), or with sparse grids (J. Jakeman et al., 2019). Herein, we employ the software library PyApprox (J. D. Jakeman, 2022), a flexible and efficient open-source tool for high-dimensional approximation and UQ, which utilizes Gaussian processes (Harbrecht et al., 2020; Rasmussen & Williams, 2006).

Gaussian processes are well-suited to computing approximations of high-dimensional computationally expensive models, such as the one we consider in this paper. They have a number of desirable properties. First, Gaussian processes can accurately approximate the output of a complex model with limited training data. Second, sensitivity indices can be computed easily from the Gaussian process. Finally, the surrogate and the Sobol indices are endowed with probabilistic error estimates, which measure the influence of using a finite set of training data. These error estimates can be used to weight the confidence placed in decisions made from the output of the Gaussian process.

Building a Gaussian process requires specifying a correlation function, C(z, z′) and a trend function. The Gaussian process leverages the correlation between training samples to approximate the residuals between the training data and the trend function. In this work, we set the trend function to zero and consider the squared exponential kernel

where is vector hyperparameters that determine the exact nature of the correlation function.

A Gaussian process is a distribution over a set of possible functions. Given a set of training samples , and associated function values (realizations of the random output Y) the posterior mean and variance of the Gaussian process are

respectively, where

and A is a matrix with elements A_ij = C(z⁽ⁱ⁾, z^(j)) for i, j = 1, …, M. In this work, we use scikit-learn (Pedregosa et al., 2011) to construct the Gaussian process and estimate the hyperparameters. Because of the differing magnitudes of the ranges of the training samples and values, we found it essential to normalize the training data. Specifically, we transformed the training samples to [−1,1]^d and normalized the training values to have a mean zero and unit variance. Once the Gaussian process is constructed, we post-process the approximation using PyApprox to obtain main effect functions and sensitivity indices. Because the Gaussian process is itself random, the aforementioned quantities are also random.

2.4.2.1 Marginalized Main Effect Functions

The main effect functions are linear functionals of the Gaussian process and thus the posterior distributions of are also Gaussian. For tensor product densities ρ and separable kernels of the form , such as the squared-exponential used here, we can compute the posterior mean and variance of the main effect functions using one-dimensional (1D) quadrature rules (Oakley & O’Hagan, 2004). Specifically, the posterior mean of is where

(6)

Here, the superscript ⋆ indicates we are taking the expectation over the posterior distribution of the Gaussian process and we have used the separability of the kernel to write . We use 100-point Gaussian quadrature rules to compute the 1D integrals in Equation 6. We use a similar technique to compute the posterior variance of the main effect functions:

where ◦ is the Hadamard (element-wise) product and

The left expression above requires a two-dimensional tensor-product quadrature, but since we are not evaluating the simulation model, this is inexpensive to apply. In Figures 17 and 18, mean of the main effect functions marginalized over one parameter at a time, plus or minus two standard deviations, that is

2.4.2.2 Sensitivity Indices

Given the presentation above, the posterior distribution of Sobol, main effect, and total effect indices cannot be obtained analytically. Following (Oakley & O’Hagan, 2004), we compute the posterior mean and variance as the sample average of the estimates of the indices obtained using 1,000 different random realizations of the Gaussian process. For each realization, we compute the sensitivity indices accurately (close to machine precision) using a procedure similar to that used for constructing the main effect functions. We omit the exact expressions used because they are overly complex. In Figures 10-15 we plot the median sensitivity indices (red line), the interquartile range (box), and the minimum and maximum values (whiskers).

Figure 10

GSA results: main effects, Sobol, and total effects indices (from left to right) for the sea ice extent (SIE) QOI calculated annually (a) and by season (b)–(e). The box-and-whiskers plots depict GSA results obtained using a Gaussian process emulator, which provides uncertainty bounds: the red central mark indicates the median of the data; the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 11

GSA results: main effects, Sobol, and total effects indices (from left to right) for the sea ice volume (SIV) QOI calculated annually (a) and by season (b)–(e). The box-and-whiskers plots depict GSA results obtained using a Gaussian process emulator, which provides uncertainty bounds: the red central mark indicates the median of the data; the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 12

GSA results: main effects, Sobol, and total effects indices (from left to right) for the sea surface temperature averaged over 60°–90° (SST) QOI calculated annually (a) and by season (b)–(e). The box-and-whiskers plots depict GSA results obtained using a Gaussian process emulator, which provides uncertainty bounds: the red central mark indicates the median of the data; the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 13

GSA results: main effects, Sobol, and total effects indices (from left to right) for the surface temperature averaged over 60°–90° (TS) QOI calculated annually (a) and by season (b)–(e). The box-and-whiskers plots depict GSA results obtained using a Gaussian process emulator, which provides uncertainty bounds: the red central mark indicates the median of the data; the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 14

GSA results: main effects, Sobol, and total effects indices (from left to right) for the low cloud coverage averaged over 60°–90° (CLDLOW) QOI calculated annually (a) and by season (b)–(e). The box-and-whiskers plots depict GSA results obtained using a Gaussian process emulator, which provides uncertainty bounds: the red central mark indicates the median of the data; the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 15

GSA results: main effects, Sobol, and total effects indices (from left to right) for the net longwave surface radiation averaged over 60°–90° (FLNS) QOI calculated annually (a) and by season (b)–(e). The box-and-whiskers plots depict GSA results obtained using a Gaussian process emulator, which provides uncertainty bounds: the red central mark indicates the median of the data; the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 16

Global sensitivity analysis results: seasonal variation of the mean total sensitivity (total effects) indices of the four most influential parameters. The box represents 25%–75% confidence intervals. The median of the data is denoted by the red line. The mean of the data is denoted by the blue dot. Whiskers designate the minimal and maximal values of the total effect indices. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 17

Marginalized main effects of the most important parameters affecting annual sea ice extent. The black solid line represents the median of the main effects calculated using a Gaussian process and the gray shading represents the 95% confidence intervals of the main effects calculated using the Gaussian process emulator. Descriptions of the parameters {z_i} are provided in Table 2.

Figure 18

Marginalized main effects of the most important parameters affecting the annual surface temperature. The black solid line represents the median of the main effects calculated using a Gaussian process. The gray shading represents the 95% confidence intervals of the main effects calculated using the Gaussian process emulator. Descriptions of the parameters {z_i} are provided in Table 2.

2.5 Global Sensitivity Analysis Workflow

Figure 7 summarizes our GSA workflow. First, an appropriate initial condition is obtained by spinning up the E3SM to equilibrium, as discussed in Section 2.2. Next, after selecting T_spin-up and T_final (ensuring that these values are large enough to avoid initial transients in the ensemble runs), we employ the DAKOTA library (Adams et al., 2013) to generate N random samples of the parameters {z_i} from the selected parameter ranges or probability distributions (Table 2). We then create namelist files for each of our E3SM runs, corresponding to each of the N randomly selected parameter sets (for our study, the relevant namelist files are user_nl_cam, user_nl_mpaso, and user_nl_mpascice), and set off N runs of the E3SM, branching off the spun-up initial condition. Finally, we post-process the perturbed runs to extract from them the relevant QOIs (see Table 3), and perform the GSA by providing M QOI parameter pairs to PyApprox, where M ≤ N is the number of runs that completed successfully (simulated the global climate state to time T_final). The workflow depicted in Figure 7 was largely automated through the creation of shell scripts that execute the relevant commands, comprising these steps. These scripts are stored in a repository containing the E3SM fork used for this study; for details, please see the Acknowledgments section of this paper. All of our runs were performed on the Skybridge high-capacity cluster located at Sandia National Laboratories, which contains 1,848 nodes, each having 16 2.6 GHz Intel Sandy Bridge processors.

3 E3SM Simulation Results

In the present study, a total of N = 212 sets of parameter combinations were generated, assuming uniform probability distributions given by the “Min” and “Max” values found in Table 2 for each parameter. We then set off 212 75-year perturbed runs of E3SMv1, one for each set of parameter values using preindustrial control forcing. In addition to perturbing the values of the parameters in Table 2, modified parameter values from (Golaz et al., 2019), which are given in Table 1, were used for all of the perturbed runs for consistency with the final model spin-up, discussed in Section 2.2. The values of all 212 perturbed sets of parameters are given in Appendix C of Peterson et al. (2020). Parameter values for the so-called “baseline” run, which was a continuation of the final spin-up run and included in our ensemble set, are given in Table 2. All of our simulations were run on 96 processors (6 nodes) of Sandia's Skybridge high-capacity cluster, described earlier in Section 2.5.

Of the N = 212 perturbed runs, a total of 138 runs made it to T_final = 75 years. The baseline run also made it to T_final = 75 years, totaling M = 139 successful runs. These 139 75-year runs took approximately 1.00 × 10⁶ CPU hours to complete on the Skybridge cluster. We estimated that repeating this study using the standard 1° configuration of the E3SM would require more than 100 times more resources (≈1.14 × 10⁸ CPU hours on Skybridge).

As described earlier in Section 2.3, in calculating the QOIs in Table 3, we performed averaging both annually and seasonally over years 51–75, so as to allow each perturbed run a spin-up/equilibration period of 50 years.

3.1 Ensemble Trajectories

Figure 8 shows the trajectories of all six QOIs considered (Table 3) for each of the 139 successful ensemble runs (runs that made it to year T_final = 75). The QOIs are averaged over each year and plotted as a function of the year since the start of each perturbed run. The baseline run is distinguished from the others by the red markers. All six QOIs are effectively in equilibrium at all times for the baseline run, as expected. A careful inspection of the trajectories in Figure 8 reveals that the relationships between the QOIs are also as expected, that is, runs giving rise to a large sea ice area also give rise to a smaller surface air temperature. Additionally, one can see from Figure 8 that most of the perturbed runs appear to have reached equilibrium by year 40. This justifies the selection of T_spin-up = 50 years. It is interesting to remark that significant changes to the QOIs are seen in the perturbed runs, with several runs resulting in a complete loss of Arctic sea ice and several runs exhibiting an apparent exponential growth in Arctic sea ice. This suggests that our parameter choices and ranges produced a sufficiently wide range of possible climate outcomes, as intended.

3.2 Ensemble Statistics

We now look at some statistics for the perturbed runs that made it to year 75. Figure 9 shows box-and-whiskers plots for each of the six QOIs considered, calculated by season. Here, the seasons are defined as follows: “Winter” is comprised of the months from January to March, “Spring” is comprised of the months from April to June, “Summer” is comprised of the months from July to September, and “Autumn” is comprised of the months from October to December. The red central mark indicates the median of the data, whereas the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to the most extreme data points not considered outliers, and the outliers are plotted using the “+” symbol. Outliers are defined as values that are more than 1.5 times the interquartile range away from the top or bottom of the box in a given box plot.

Figure 9 shows that the maximum and minimum sea ice extents are observed in the “Spring” and “Autumn” seasons, respectively. This result may seem surprising, as observational data and standard 1° resolution E3SM simulations (see Peterson et al., 2020) have shown that the maximum and minimum sea ice extent in general occur in March and September, respectively, which would correspond to the “Winter” and “Autumn” seasons based on our definition. A closer inspection reveals that, for the majority of our ULR runs, including the baseline run, the maximum and minimum sea ice extent occur in April and October (for a plot showing this, the reader is referred to Peterson et al. (2020)). Similarly, the maximum and minimum sea ice volume occur in May and October, respectively. The cause of this shift in the month of maximum and minimum sea ice extent and volume in the ULR configuration is uncertain at this time, but these results motivate follow-on work to understand the behavior in more detail.

It is interesting to look at the relative spreads of the box-and-whiskers plots in Figure 9. This spread can be viewed as a measure of uncertainty. One can see from Figure 9a that the SIE QOI has the smallest uncertainty in the melting seasons (during which it is particularly relevant for trans-Arctic shipping routes), summer and autumn. The only QOI with significant outliers is the SIV. Referring to the ensemble trajectory plots, namely Figure 8b, the reader can observe that the SIV QOI (an estimator of older, multiyear ice) is the only QOI with a significant number of trajectories so anomalous that they predict an apparent exponential growth in Arctic sea ice volume. It is likely that these trajectories translate to the outliers in the box-and-whiskers plot for SIV (Figure 9b); however, it is unclear what mechanism within the ULR E3SM is causing a feedback of this type. The SIV QOI has the same uncertainty trends as the SIE QOI if outliers are excluded; however, if outliers are included, the uncertainty in SIV is comparable across all four seasons, a result similar to the one obtained in Urrego-Blanco et al. (2019). The remaining four QOIs have the largest uncertainty during the seasons in which they are either minimal (for TS and CLDLOW) or maximal (for sea surface temperature [SST] and FLNS), on average. Certain expected correlations in uncertainties between the QOIs are observed. For example, the box-and-whisker plot spreads for the FLNS and CLDLOW mimic each other across all four seasons, which can be explained by the fact that FLNS is in general strongly determined by cloud variations and cloud cover (Schweiger et al., 2008).

3.3 Correlations in QOIs

Tables 4–7 give the correlation coefficients between our six QOIs, averaged seasonally. In general, the relationships between the QOIs are consistent with expectations. SIE and SIV, as well as SST and TS, have a strong positive correlation across all four seasons. SIE/SIV are negatively correlated with SST/TS, again as expected: larger sea ice volumes occur under lower air and SSTs. One can additionally observe a general negative correlation between CLDLOW and FLNS, especially during the warmer spring, summer, and autumn seasons. This relationship can be explained by the fact that clouds absorb and reemit the longwave radiation emitted by the surface. Since FLNS is defined as a derived quantity representing the difference between surface upwelling and downwelling longwave radiation, in scenarios with abundant low-cloud cover, one would expect greater cloud-emitted downwelling radiation at the surface, and less upwelling, as cloud cover reduces incident surface radiation. Similarly, thin low cloud coverage would result in relatively little cloud-emitted downwelling radiation, and greater surface upwelling (heat). There is virtually no correlation between the following pairs of QOIs in the winter season: (SIE and FLNS), (SIV and CLDLOW), and (TS and FLNS). The lack of correlation between (SIE and FLNS) and (SIV and CLDLOW) in winter is contrary to results obtained using higher-resolutions of the E3SM (Urrego-Blanco et al., 2019). One possible explanation for this discrepancy is the coarseness of the atmosphere grid in the ULR E3SM, resulting in differences in cloud formation relative to higher-resolution models. The reader can observe negative relationships between CLDLOW and the surface temperature QOIs (SST and TS) across all four seasons. In the spring and summer seasons, when the sun is above the horizon, clouds will generally reflect solar (shortwave) radiation, which would potentially decrease surface temperature. This interpretation is consistent with our results in all seasons but winter. In the winter season, the general expectation is that cloud coverage would increase surface temperature. This trend is not observed in our data. It is possible that the fact that our data set contains a number of runs without any sea ice coverage is biasing the results. Since, at present time, there do not exist observational data for the case of no sea ice (especially in winter), it may not be possible to interpret the CLDLOW correlations with the surface temperatures.

Table 4. Table of Correlation Coefficients Between the Six Quantities of Interests Considered (Table 3), Averaged During the Winter Season (January–March) Over the Last 25 Years, for the Successful Ensemble Runs

• Note. Large positive correlation coefficients (≥0.75) are colored blue. Large negative correlation coefficients (≤−0.75) are colored yellow.

Table 5. Table of Correlation Coefficients Between the Six Quantities of Interests Considered (Table 3), Averaged During the Spring Season (April–June) Over the Last 25 Years, for the Successful Ensemble Runs

• Note. Large positive correlation coefficients (≥0.75) are colored blue. Large negative correlation coefficients (≤−0.75) are colored yellow.

Table 6. Table of Correlation Coefficients Between the Six Quantities of Interests Considered (Table 3), Averaged During the Summer Season (July–September) Over the Last 25 Years, for the Successful Ensemble Runs

• Note. Large positive correlation coefficients (≥0.75) are colored blue. Large negative correlation coefficients (≤−0.75) are colored yellow.

Table 7. Table of Correlation Coefficients Between the Six Quantities of Interests Considered (Table 3), Averaged During the Autumn Season (October–December) Over the Last 25 Years, for the Successful Ensemble Runs

• Note. Large positive correlation coefficients (≥0.75) are colored blue. Large negative correlation coefficients (≤−0.75) are colored yellow.

3.4 Main Effects, Total Effects, and Sobol Indices

Finally, we present and discuss the results of the GSA study using the methodology and workflow described in Sections 2.4 and 2.5, respectively. Our main results are summarized in Figures 10-15 below. For each row of each figure, three plots are reported, which show the main effect, Sobol, and total effect indices (from left to right, respectively) corresponding to each of the nine parameters considered (Table 2). As discussed in more detail in Section 2.4.2, the main effect indices measure the effect of individual parameters acting alone and can sum to at most 1. As the sum approaches 1, the contribution of all parameter combinations involving two or more variables decreases. A value of 1 indicates that the function is purely additive and there is no interaction between any parameters. Total effect indices measure the total contribution of each parameter to the variance of a given QOI; specifically, they measure the contributions of all interactions involving a specific parameter. Consequently, the total effect index of a single variable will always be at least as large as the main effect index of that variable. Furthermore, the sum of all total effect indices can be greater than 1, because Sobol indices for parameter interactions involving at least two variables can be used to compute the total effects of multiple variables, that is, the Sobol index of will contribute to the total effect indices of both the ith and jth variables. Comparing main effect and total effect indices can be used to determine the strength of high-order (involving more than two parameters) interactions. For example, in Figure 12b, the main effect of clubb_c1 (z₅) is less than 3% of the total variance, yet the total effect of this variable is over 20% of the total variance. While main and total effect indices summarize the contributions of a single parameter to the variance of a QOI, Sobol indices can be used to identify the contribution of specific parameter interactions to the total variance. Sobol indices involving just one parameter are labeled “(z_k)” and indices involving two parameters are labeled “(z_i, z_j)” with i ≠ j. Contributions by miscellaneous pairs of parameters in which the percent contribution was <1% were omitted from the plots. We found that there were no strong interactions involving three or more variables. The confidence intervals provided in the plots provide a more goal-oriented means to determining the confidence in parameter rankings. Overlapping intervals of sensitivity indices suggest that we cannot rank parameters confidently.

Figures 10-15 also report the predictivity coefficient Q², which is a measure of the mean square error of the Gaussian process model using cross-validation (Marrel et al., 2008). A value of Q² = 1 is indicative of a perfect cross-validation fit for the given data. Larger values of Q² imply greater confidence can be placed in the sensitivity results; however, the value of Q² that engenders sufficient confidence is subjective.

3.4.1 Atmospheric Parameters

From Figures 10-15, we can credibly conclude that the atmospheric parameters cldfrc_dp1 (z₄), clubb_c1 (z₅), clubb_c8 (z₆), and gamma_coeff (z₇) are the most sensitive for all seasons and QOI. The minimum values (bottom whisker) of the total effects of these parameters are all larger than the maximum values (top whisker) of the other parameters. This result is consistent with results obtained in earlier sensitivity studies, namely the fully coupled study of (Urrego-Blanco et al., 2019). Although there are uncertainty bounds that make it difficult to rigorously pick the most important parameter, based on the median values of the main and total effect indices obtained from Gaussian process emulator approximations, the parameter clubb_c8 (z₆) is consistently the most important parameter for all six QOIs and across all four seasons, followed by gamma_coeff (z₇). In fact, for most seasons and QOIs, the minimum total effect values of these two parameters are greater than the maximum values for all other parameters. The main effects trends for parameters clubb_c1 (z₄) and clubb_c8 (z₅) are not as clear cut, but seem to follow similar correlation patterns for the QOI as clubb_c8 (z₆) and gamma_coeff (z₇) respectively (i.e., clubb_c1 has similar trends to clubb_c8, and clubb_c1 has similar trends to gamma_coeff).

To streamline and consolidate some of the presentation, we introduce and analyze Figure 16, which plots the seasonal variation of the median total sensitivity (total effects) indices of the four most influential (atmospheric) parameters. In this plot, the box represents 25%–75% confidence intervals, the red line denotes the median of the data, and the blue dot denotes the mean of the data. Whiskers designate the minimal and maximum values of the total effect indices.

3.4.1.1 The cldfrc_dp1 (z₄) Parameter

The cldfrc_dp1 (z₄) CLUBB parameter, which controls cumulus cloud-formation convective regimes in the E3SM (Larson, 2020; Qian et al., 2018), has a significant impact on four of the six QOIs considered here, namely SIE, SST, CLDLOW, and FLNS. Figure 16 shows that CLDLOW is most sensitive to this parameter in winter. In contrast, SIE and FLNS are most sensitive to cldfrc_dp1 in spring (Figures 10, 15 and 16). The sensitivities of SIE and SIV have strong cyclic seasonal trends. In addition, noncyclical seasonal variation is present in SIV and CLDLOW. Seasonal variations in the median values of the sensitivity indices of some other QOI are also present; due to large confidence intervals that overlap, these trends may be considered plausible but, without higher accuracy, not credible. With this being said, it is interesting to note that the seasonal trend in the median total effect indices of SIV and SIE differs significantly. These differences could reflect the difference between relatively stable multiyear ice (measured by SIV) and young, seasonal ice (measured by SIE).

3.4.1.2 The clubb_c1 (z₅) Parameter

The clubb_c1 (z₅) parameter controls the balance of cumulus versus stratocumulus clouds, as discussed in Larson (2020). Large positive values of this parameter favor cumulus clouds, while small or negative values are associated with stratocumulus clouds. Stratocumulus clouds are hybrids of the layered stratus and cellular cumuli clouds and are believed to have a planet wide surface cooling effect, but earlier investigations have hypothesized that this cloud type in the Arctic has surface warming effects over most of the year (Eastman & Warren, 2010). Figure 16 shows that the SIE, TS, and FLNS QOIs exhibit a strong sensitivity to clubb_c1 (z₅) during the autumn season. These results are consistent with previous observational and modeling studies (Eastman & Warren, 2010; Huang et al., 2019; Kay & Gettelman, 2009; Philipp et al., 2020; Taylor et al., 2015), which have reported a correlation between cloud type, Arctic surface temperature, and Arctic sea ice extent during the October–November months. Interestingly, our CLDLOW QOI does not show as strong a sensitivity to clubb_c1 (z₅) in the autumn as seen for the FLNS QOI. This indicates that while clubb_c1 (z₅) influences cloud type (cumulus or stratocumulus (Larson, 2020)), it may not strongly influence the fraction of general low cloud cover. That FLNS is responsive to clubb_c1 (z₅) in autumn is not surprising, given that this season represents the period of maximum interannual variation in SIE, which both reflects and influences the atmosphere/cloud-ocean-sea ice feedback.

3.4.1.3 The clubb_c8 (z₆) Parameter

The clubb_c8 (z₆) parameter was developed to achieve radiative balance in atmospheric models (Larson, 2020; Qian et al., 2018). Specifically, increasing clubb_c8 (z₆) brightens clouds, resulting in Earth's surface cooling as brighter clouds reflect more incoming solar radiation. Figure 16 reports that the clubb_c8 (z₆) has significant influence over all six QOIs considered across all four seasons, with a median main effect of at least 0.4. It is interesting to observe that the CLDLOW and FLNS responses to clubb_c8 (z₆) trend similarly across the four seasons. Even accounting for errors in sensitivity indices, Figure 16 suggests that FLNS has the strongest seasonal response to perturbation of clubb_c8 (z₆) in winter. The SIE QOI shows a strong response to clubb_c8 (z₆) in autumn, with a median total effect of approximately 0.6 and a lower bound of the confidence interval above 0.5. This seems to suggest that cloud brightening has the potential to control the degree to which sea ice is lost toward the end of the melting season (autumn). The impact of clubb_c8 (z₆) perturbation relative to the other atmospheric parameters, with the exception of the significantly less influential clubb_c1 (z₅) parameter on the SST QOI is difficult to separate due to overlapping uncertainty bounds for these QOIs (Figure 12). In contrast, clubb_c8 (z₆) is very clearly the most dominant parameter when it comes to its influence over the TS QOI for all seasons (Figure 13).

3.4.1.4 The gamma_coeff (z₇) Parameter

Like clubb_c8 (z₆), the gamma_coeff (z₇) parameter is a tunable parameter in the CLUBB shallow convection parameterization scheme that can brighten or dim low clouds, developed to achieve global radiative balance in E3SM (Larson, 2020). Our results show both relatively strong (SIE, SIV, CLDLOW, and FLNS) and moderate (TS and SST) seasonal responsiveness to gamma_coeff (z₇) (Figure 16). SIE shows the greatest response to gamma_coeff (z₇) in spring, the period of both the onset of melt season and the annual maximum, with mean total effects of 0.50, and minimum/maximum total effects of approximately 0.40/0.60, respectively. In spring, the season during which SIE is most responsive to gamma_coeff (z₇), the Arctic is moving into longer days, as the annual SIE maximum is reached, and the melt season is beginning. In this context, cloud brightening potentially influences surface energy balance because brighter clouds reflect more incoming solar radiation. Interestingly, SIV, an estimator of multiyear ice, shows a markedly different response to perturbation of this parameter than SIE, a proxy for seasonal and marginal ice; however, these results should be interpreted with some caution due to the large confidence intervals. While the gamma_coeff (z₇) and clubb_c8 (z₆) parameters both have ostensible control on cloud brightness, their impacts upon SIE differ markedly: the greatest mean total effects for the clubb_c8 (z₆) parameter were observed in autumn (≈0.60), compared to spring for the gamma_coeff (z₇) (≈0.40). The different responses are explained by the fact that the parameters represent distinct terms in CLUBB (Larson, 2020).

3.4.1.5 Interactions Between Atmospheric Parameters

It is important to note that while the present study reveals that significant parameter interactions generally involve the four atmospheric parameters, our study demonstrates the effect of these parameters on QOIs from E3SM components other than the atmosphere model. These results would be impossible to obtain without a global fully coupled ESM. Despite nontrivial errors in the sensitivity indices, we can also conclude that certain parameter interactions involving the four most sensitive parameters contribute more to the variability of all QOI than any of the five insensitive parameters. For example, the Sobol index labeled (z₅, z₆) in Figure 11, which quantifies the strength of the interactions between clubb_c1 and clubb_c8 for the QOI SIV in spring, is much stronger than the total effects of the five insensitive parameters. Indeed, in this case, the interaction contributes more than cldfrc_dp1 (z₄) acting alone. Additionally, for the CLDLOW and FLNS QOIs (Figures 14 and 15, respectively), a number of parameter interactions involving the various atmospheric parameters are at least comparable to the effect of clubb_c1 (z₅) acting alone.

3.4.2 Sea Ice and Ocean Parameters

While we see little impact from the sea ice and ocean parameters relative to the atmospheric parameters, there are a few cases for which the total effects of these parameters are nonzero. Of the sea ice parameters, ksno (z₁) had the largest total effect for several QOIs in several seasons. Nonzero total effect indices associated with ksno (z₁) for the SST and FLNS QOIs are shown in Figures 12 and 15, respectively. This result is consistent with the observation that the snow conductivity can affect ocean temperature since it impacts the amount of heat flux (solar radiation) that reaches the ocean in ice-covered waters. During the late spring, summer, and early autumn seasons, this solar radiation input would primarily come from short-wave solar radiation. The reader can observe by examining Figures 10-15 that the effects of the two ocean parameters over the range of parameters tested, as well as their interaction with each other and other parameters, on our six QOIs are negligible.

3.5 Marginalized Main Effect Indices

In this section, we present the univariate marginalized main effect functions (Equation 6) described in Section 2.4.2. These main effect functions enable us to determine a priori whether increasing/decreasing a given parameter will increase or decrease a given QOI. These results are particularly useful for model spin-up/tuning, which can be an ad hoc trial-and-error process. For the sake of brevity, we provide the marginalized main effects results for only two of our QOIs averaged annually, SIE and TS (Figures 17 and 18, respectively), as these are the QOIs most relevant for model spin-ups. Identical conclusions were obtained from the analogous seasonal plots.

The results presented below demonstrate that, as expected, the four atmospheric parameters considered herein have the greatest influence when it comes to model spin-up/tuning. The reader can observe by examining Figures 17 and 18 that there are clear-cut parameter-QOI correlations for the clubb_c8 (z₆) and gamma_coeff (z₇) parameters. The parameter clubb_c8 (z₆) has a strong positive correlation with SIE and a strong negative correlation with TS, whereas the parameter gamma_coeff (z₇) has a strong negative correlation with SIE and a strong positive correlation with TS. The fact that SIE and TS have opposite trends is consistent with the QOI correlations uncovered earlier (Section 3.3). It is interesting that the marginalized main effects plots for the remaining two atmospheric parameters, cldfrc_dp (z₄) and clubb_c1 (z₅), have inflection points and some level of convexity/concavity, meaning that determining whether increasing/decreasing these parameters will increase/decrease a QOI depends on the parameter value. In our manual spin-up of the ULR E3SMv1, we found by trial-and-error that cldfrc_dp1 (z₄) had a significant effect on tuning the model, in particular, increasing cldfrc_dp1 within the range [0.075, 0.5] decreased TS and increased SIE (Peterson et al., 2020). This provides some corroboration of the results in Figures 17 and 18.

Reconciling the results discussed above with the relevant physical processes requires discussion of the physical effects of our four atmospheric parameters. Without loss of generality, we will focus on the surface air temperature, or TS, QOI. From Table 2, clubb_c1 (z₅) and clubb_c8 (z₆) have an effect on the skewness of the probability density function of the vertical velocity w′ within the CLUBB parameterization (Guo et al., 2014; Larson, 2020; Qian et al., 2018). High skewness in the vertical velocity causes the production of cumulus-like layers of clouds with a low cloud fraction, whereas low skewness results in stratocumulus clouds having a high cloud fraction. Increasing clubb_c8 (z₆) is known to lead to cloud brightening and cooling at the Earth's surface (Larson, 2020). This result is consistent with our analysis. Additionally, with low values of clubb_c1 (z₅), which favor insolation-reducing stratiform clouds, SIE is relatively high and TS is low, a result consistent with observational studies on the general surface-cooling effects of this cloud type. Like stratocumulus clouds, cumuli can reflect incident solar radiation, or trap heat, depending on the cloud height and optical density. Since SIE is relatively low and TS is relatively high for larger values of clubb_c1 (z₅), our results point to the heat-trapping effects of the cumulus species. The parameter gamma_coeff (z₇), which controls the width of the individual Gaussians within the CLUBB parameterization (Larson, 2020), has broad effects within CLUBB, influencing not only shallow convection but also stratocumulus cloud formulation. As discussed in Qian et al. (2018), increasing gamma_coeff (z₇) has a similar effect to increasing skewness, which leads to a smaller cloud fraction. Thus, the parameter gamma_coeff (z₇) is expected to have a similar effect on the surface air temperature as clubb_c1 (z₅), which is in general consistent with our results. Finally, we turn our attention to the last atmospheric parameter, cldfrc_dp1 (z₄), CLUBB's deep convection cloud parameter. Increasing this parameter results in the movement (convection) of hotter and therefore less dense material upward, causing colder and denser material to sink under the gravity, cooling the Earth's surface. Yet again, the negative cldfrc_dp1 (z₄)-TS correlation uncovered by our results is consistent with this physical mechanism.

While the subplots in Figures 17 and 18 corresponding to the ocean and sea ice parameters are flat compared to the subplots corresponding to the atmospheric parameters, the reader can observe a slight curvature in the plots for sea ice parameters ksno (z₁) and dragio (z₃). It is interesting to remark that the trends present in these parameter-QOI correlations are similar to the trends uncovered using an alternate marginalization technique for the stand-alone sea ice model GSA of Urrego-Blanco et al. (2016) (see Figure 11 in this reference).

4 Summary

We performed a GSA involving nine parameters and six QOIs spanning three climate components (atmosphere, ocean, and sea ice) using a fully coupled ULR configuration of E3SMv1, which is more than 100 times faster than the standard 1° resolution E3SM. To the best of our knowledge, this is the first GSA involving the fully coupled E3SMv1. A study of this scope would be intractable with higher-resolution, scientifically validated configurations of E3SM, such as the 1° configuration, due to the computational cost of running numerous E3SM ensembles at high resolution.

Before beginning our analysis, we performed a spin-up of the ULR configuration E3SM with preindustrial control forcing to achieve an equilibrium climate state. Comparisons of the ULR configuration simulation output with 1° resolution E3SM simulation output as well as observational (CERES-EBAF and ERA-Interim) data demonstrated that the ULR E3SM reproduced large-scale patterns in top of atmosphere radiation, precipitation, zonal mean temperature, and zonal mean wind. In order to perform the sensitivity analysis, we created a fast Gaussian process emulator from 139 75-year runs of the ULR E3SMv1, which included preindustrial control forcing and were initialized from a spun-up initial condition developed for the purpose of this study. The runs exhibited a great deal of variability, spanning the gamut from a complete loss of Arctic sea ice to an apparent exponential growth in Arctic sea ice. Our Gaussian process emulator was used to determine Sobol indices, main effect indices, and total effect indices for each QOI-parameter combination and provided uncertainty bounds for each set of indices. While the sometimes large uncertainty bounds made it difficult to rigorously pick out the most influential parameter for each QOI, the study enabled a definitive ranking of the most dominant parameters affecting each QOI annually and seasonally. We found the atmospheric parameters related to cloud physics within the CLUBB model in EAM (and their interactions) had by far the greatest impact on the Arctic climate state. While our study demonstrated that the most significant parameter-parameter interactions involved the atmospheric parameters with each other, it enabled us to investigate the effect of these parameters on QOIs from E3SM components different than the atmosphere model. The fact that this investigation would not be possible with a stand-alone atmosphere model reinforces the need for coupled analyses when studying climate model uncertainties/sensitivities. We performed a careful study of QOI-QOI correlations and parameter-parameter interactions using our sensitivity indices, and were able to reconcile these relationships with several well-known Arctic feedback processes. By approximating univariate main effect functions (Oakley & O’Hagan, 2004), we were able to determine the sensitivity of individual QOIs on individual parameters, thereby inferring QOI-parameter correlations, useful for model spin-up/tuning. We also performed a careful study of the marginalized main effect functions for the most influential (atmospheric) parameters, and demonstrated that the trends uncovered by the study are consistent with both our manual spin-up of the ULR E3SMv1 as well as the physical processes underlying the CLUBB parameterization (e.g., the formation of cumulus vs. stratocumulus clouds, the relative amount of shortwave cloud forcing/cloud brightening).

The study discussed in this paper is significant for several reasons. As stated earlier, the computational cost of running higher-resolution models makes sensitivity analyses using such models intractable at the present time. Our results suggest that the ULR configuration is a plausible surrogate when compared to existing and up-and-coming data-driven machine learning surrogate construction approaches, which require a tremendous amount of training data, are not physics-based, and do not in general possess rigorous accuracy bounds when used in the predictive regime. Additionally, this study can serve as a baseline for and guide future studies with higher-resolution models when algorithmic developments and advancements in computational power enable their use. Finally, our results can be used to: (a) show the number of samples needed to get even moderate accuracy in a sensitivity analysis with a variety of different parameters, which is useful for predicting the computational budget to run future GSA studies; and (b) investigate the impact of resolution on sensitivity indices when the computational resources to run higher-resolution GSA studies become available. One avenue for near-term future work is to augment the present study with higher-fidelity ensemble data (e.g., using a medium-low resolution, or MLR, of the E3SMv1 having a resolution of approximately 2.7° for the atmosphere component (Peterson et al., 2020)), toward a multifidelity GSA.

Acknowledgments

This work was funded by the Laboratory Directed Research & Development (LDRD) program at the Sandia National Laboratories. The writing of this manuscript was funded by the I. Tezaur's Presidential Early Career Award for Scientists and Engineers (PECASE). Sandia National Laboratories is a multimission laboratory managed and operated by the National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the U.S. Government. The authors would like to thank Drs. Luke van Roekel and Xylar Asay-Davis for helping us with the selection of appropriate oceanic parameters to include in our study. The authors also wish to thank two anonymous reviewers for their insightful comments and suggestions, which helped to improve this manuscript.

Open Research

Data Availability Statement

Per the Enabling FAIR Data Project guidelines, we have made the data used in the global sensitivity analysis performed herein publicly available. These data can be downloaded from the following Zenodo repository: https://zenodo.org/record/6321483 (https://doi.org/10.5281/zenodo.6321483). The following fork of Energy Exascale Earth System Model was used to generate the ensembles analyzed in this study: https://github.com/karapeterson/E3SM (add_namelist_params branch). The PyApprox toolkit used in the analysis described in this paper is also available on GitHub: https://github.com/sandialabs/pyapprox.