Volume 7, Issue 3 p. 963-986
Research Article
Open Access

Large eddy simulation using the general circulation model ICON

Anurag Dipankar

Corresponding Author

Anurag Dipankar

Max Planck Institute for Meteorology, Hamburg, Germany

Correspondence to: A. Dipankar, [email protected]Search for more papers by this author
Bjorn Stevens

Bjorn Stevens

Max Planck Institute for Meteorology, Hamburg, Germany

Search for more papers by this author
Rieke Heinze

Rieke Heinze

Institute for Meteorology and Climatology, Leibniz University of Hannover, Hannover, Germany

Search for more papers by this author
Christopher Moseley

Christopher Moseley

Max Planck Institute for Meteorology, Hamburg, Germany

Search for more papers by this author
Günther Zängl

Günther Zängl

Deutscher Wetterdienst, Offenbach, Germany

Search for more papers by this author
Marco Giorgetta

Marco Giorgetta

Max Planck Institute for Meteorology, Hamburg, Germany

Search for more papers by this author
Slavko Brdar

Slavko Brdar

Deutscher Wetterdienst, Offenbach, Germany

Search for more papers by this author
First published: 16 June 2015
Citations: 140

Abstract

ICON (ICOsahedral Nonhydrostatic) is a unified modeling system for global numerical weather prediction (NWP) and climate studies. Validation of its dynamical core against a test suite for numerical weather forecasting has been recently published by Zängl et al. (2014). In the present work, an extension of ICON is presented that enables it to perform as a large eddy simulation (LES) model. The details of the implementation of the LES turbulence scheme in ICON are explained and test cases are performed to validate it against two standard LES models. Despite the limitations that ICON inherits from being a unified modeling system, it performs well in capturing the mean flow characteristics and the turbulent statistics of two simulated flow configurations—one being a dry convective boundary layer and the other a cumulus-topped planetary boundary layer.

Key Points:

  • Large eddy simulation using a general circulation model ICON is presented
  • ICON is validated against standard LES models for boundary layer simulations
  • Despite limitations, ICON performs well compared to the standard LES models

1 Introduction

Understanding climate change requires clear understanding of feedbacks due to clouds, which are the leading source of uncertainty in the existing climate models [Bony et al., 2004; Stevens and Bony, 2013]. The uncertainty in cloud fields in climate models is in turn due largely to the limitations of the shallow and deep convective parameterization schemes. Different modeling approaches like single column models [Betts and Miller, 1986; Randall et al., 1996; Ghan et al., 2000], cloud resolving modeling [Xu et al., 1992; Xu and Arakawa, 1992; Grabowski et al., 1996], and superparameterization [Grabowski, 2001; Khairoutdinov et al., 2005] have been used in the past to understand convective parameterization in a climate model.

With the increasing availability of computing resources, efforts have been taken to perform global simulations at even higher resolution to possibly reduce the uncertainty by explicitly resolving some of the scales involved in the convective motion. A good example of this approach is the Japanese model NICAM, which has been used in the recent years to perform global cloud resolving simulations [Miura et al., 2007; Satoh et al., 2014]. While one can argue about the deep convective scales that are resolved in a typical cloud resolving model with grid resolution ranging from 1 to 4 km, the grid resolution used in such models is definitely too coarse to resolve shallow convection [Miller, 1978; Bryan et al., 2003]. Moreover, the subgrid turbulence schemes used in the aforementioned cloud resolving models are typically not even designed to work in these resolution ranges. This motivated Wyngaard [2004] to refer to this range of scales, where convection is neither resolved nor so unresolved as to be representable in terms of its ensemble effects, as “gray zone.”

The only way to avoid this gray zone is by explicitly resolving shallow cumulus convection, which is defined by the depth of the atmospheric boundary layer, which is typically of the order of a kilometer. Thus, simulations on a urn:x-wiley:19422466:media:jame20180:jame20180-math-0001(100 m) grid largely obviate the need for special parameterization for organized turbulent motions such as those that define the atmospheric boundary layer and the areas of shallow convection. By crossing this threshold of urn:x-wiley:19422466:media:jame20180:jame20180-math-0002(100 m) grid resolution one can begin thinking of LES, wherein the subgrid-scale parameterizations have a sounder theoretical foundation and matter less [Deardorff, 1970; Moeng, 1984; Bryan et al., 2003]. This motivated us to work toward simulations at urn:x-wiley:19422466:media:jame20180:jame20180-math-0003(100 m) horizontal resolution, on a sufficiently large domain for sufficiently long time, which will help us to understand the parameterized convection better, and possibly, to improve it.

To take this forward, the German Federal Ministry of Education and Research (BMBF) has initiated a project named High Definition Clouds and Precipitation for advancing Climate Prediction (HD(CP)2), incorporating several institutes across Germany. The project targets limited-area LES at resolution urn:x-wiley:19422466:media:jame20180:jame20180-math-0004(100 m). There are other groups across the world also moving in this direction of ultra-high resolved simulation in a quasi-operational sense. Some of them, within our knowledge, are Chow et al. [2006], Moeng et al. [2007], and Hanley et al. [2013], who have modified existing regional models to do LES. We use ICON as the host model which is a new nonhydrostatic modeling system, developed in collaboration between the German Weather Service (DWD) and the Max Planck Institute for Meteorology (MPI-M). ICON is different from the models mentioned above because it is a unified modeling system suitable for global and limited-area applications on one hand and for climate prediction and weather forecasting on the other. After the inclusion of LES capabilities, ICON can now be refined to the spatial scales needed to resolve convection while interacting with the evolving large-scale atmosphere.

Admittedly, ICON is not the only unified modeling system known in literature. The Advanced Research WRF (AR-WRF) [Skamarock et al., 2008], for example, is a nonhydrostatic fully compressible modeling system which also has capabilities to perform global as well as limited-area simulations with suite of physics packages relevant for climate predictions, NWP, and LES. There is however an important distinction between AR-WRF and ICON in that ICON is an operational global NWP and climate model, which in this manuscript is shown to be flexible enough to be run as a LES model.

The purpose of this paper is to describe the new physics package (LES physics) in the ICON framework and compare its performance against well-established standard LES models for idealized boundary layer flows. We do not present ICON as an alternative to standard LES models that are designed primarily for boundary layer studies. Rather, we aim to utilize the unified nature of ICON by using its eddy-resolving abilities to better understand the processes that are parameterized in the climate or numerical weather forecast configurations of ICON. The model equations and the necessary changes made in ICON to get the new LES physics package are described in the following section. It is important to realize that ICON inherits some compromises from being a unified model that standard LES models do not. For example, while standard LES models use structured orthogonal grids which easily allow to implement higher-accuracy schemes, ICON uses an unstructured icosahedral grid, suitable for spherical geometry, which puts a restriction on the use of a higher-accuracy scheme at a reasonable computational cost. These challenges are apparent in section 3 where the conservative discretization of the three-dimensional turbulence scheme on the ICON grid and its coupling to the model dynamics are explained. The model is then validated against two standard LES models for the following boundary layer flow types: (a) dry convective boundary layer in section 4.1, and (b) cloud topped boundary layer in section 4.2. Concluding remarks are presented in section 5.

2 Model Description

ICON is a fully compressible model that uses geodesic Delaunay grids with C-type staggering, and has the ability to locally refine a region using the classical nesting approach. Much effort has been put in designing the code for high performance on massively parallel computing architectures. The model presently hosts two basic physics packages: one for weather predictions (DWD package) and the second for climate modeling applications (MPI-M package). These two packages are designed for subgrid-scale processes operative on scales of hundreds of kilometers to tens of kilometers. In order to use ICON at urn:x-wiley:19422466:media:jame20180:jame20180-math-0005(100 m) scales, a new LES physics package has been added to ICON incorporating the following modifications.

Some of the parameterizations in the DWD and MPI-M packages are invalid at urn:x-wiley:19422466:media:jame20180:jame20180-math-0006(100 m) scales and must be turned off. This pertains to the schemes for convection, subgrid-scale orographic effects (blocking and gravity wave drag) and nonorographic gravity wave drag. On the other hand, new approaches for representing the subgrid-scale turbulence and more complex microphysical process must be introduced. Therefore, a new subgrid-scale turbulence scheme based on the classical Smagorinsky scheme has been implemented. The scheme uses the modifications by Lilly [1962] to account for thermal stratification. A double-moment microphysics scheme based on Seifert and Beheng [2001] has also been implemented for the LES package. Furthermore, instead of the default diagnostic cloud fraction scheme, a simple all-or-nothing scheme [Sommeria and Deardorff, 1977] is used that assumes that the cloud fraction within a grid box is either 1 or 0. The default artificial numerical dissipation for LES studies is reduced to fourth order for the momentum equations.

Full details of the set of equations used in ICON and its numerical treatment are beyond the scope of the present work and can be found in Zängl et al. [2014] and Wan et al. [2013]. Only information relevant to the implementation of the turbulence scheme in ICON are discussed here. Information about the time integration scheme, the spatial discretization schemes, and a few other numerical details are also given in section 4.

The equations employed in ICON are based on the prognostic variables suggested by Gassmann and Herzog [2008], and it uses a vector-invariant form for the nonlinear momentum advection terms. The prognostic variables are the horizontal velocity component normal to the triangle edges vn, the vertical wind component w, density ρ, the density potential temperature θρ [Stevens, 2007] (also referred to as virtual potential temperature θv [Zängl et al., 2014]), and the specific masses and number densities of tracers qi (i = 1,2,3…,Nt). The tracers include water vapor (qv), liquid water (ql), snow (qs), ice (qi), etc. Nt is the total number of tracers which depends on the microphysics scheme used. Here ρ is the full air density including liquid and solid condensates, and
urn:x-wiley:19422466:media:jame20180:jame20180-math-0007(1)
In the above equation, p00 is the reference pressure, π is the Exner function, and Tρ =  is the density temperature with
urn:x-wiley:19422466:media:jame20180:jame20180-math-0008(2)
where urn:x-wiley:19422466:media:jame20180:jame20180-math-0009 is the contribution due to liquid and solid condensates, Rd and Rv are the gas constants for dry air and water vapor, respectively, and cpd is the isobaric specific heat capacity of dry air.
The turbulent fields in the model are filtered as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0010(3)
where urn:x-wiley:19422466:media:jame20180:jame20180-math-0011 is Favre-filtered [Hinze, 1975]
urn:x-wiley:19422466:media:jame20180:jame20180-math-0012(4)
The overbar urn:x-wiley:19422466:media:jame20180:jame20180-math-0013 indicates the filtering in the traditional manner as used in incompressible flows [Sagaut, 1998, p. 16]. For better readability, we henceforth use xi and vi to indicate the three ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0014) orthogonal axes and the respective velocity components. Here x1 denotes the horizontal axis normal to the triangle edge and x2 denotes the horizontal axis parallel to the triangle edge (see unit vectors 1, 2 in Figure 1). The vertical coordinate is denoted by x3, which points in the upward direction as indicated by urn:x-wiley:19422466:media:jame20180:jame20180-math-0015 in Figure 1. The velocity vector (v1,v2,v3) forms a right-handed system. Following this convention, the Navier-Stokes equations used in the model can be written as:
urn:x-wiley:19422466:media:jame20180:jame20180-math-0016(5)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0017(6)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0018(7)
where, v urn:x-wiley:19422466:media:jame20180:jame20180-math-0019 and v are the horizontal and full three-dimensional (3-D) velocity components, respectively, ζ is the vertical vorticity component, f is the Coriolis parameter. It is to be noted that the model only solves for v1, and the tangential velocity component (v2) is diagnosed using the radial basis function (RBF) reconstruction [Narcovich and Ward, 1994].
Details are in the caption following the image

(left) Schematic showing the primal (black triangles) and dual (red hexagons) cells, and the associated local coordinate system used in this manuscript. Unit vectors 1, 2, and 3 point in the direction of edge normal, tangent, and vertically upward (indicated by urn:x-wiley:19422466:media:jame20180:jame20180-math-0020), respectively. (right) Schematic of two adjacent triangles in ICON grid identifying the various locations used to discretize the turbulent diffusion term. urn:x-wiley:19422466:media:jame20180:jame20180-math-0021 are the triangle vertices, e is the center of the edge cd about which all the strain rates are calculated, and p, q are the cell-centers.

The turbulent parameterization terms, urn:x-wiley:19422466:media:jame20180:jame20180-math-0022, are calculated as the divergence of the subgrid-scale stress tensor τkl
urn:x-wiley:19422466:media:jame20180:jame20180-math-0023(8)
Herein, the Einstein summation convention applies to repeated indices. The subgrid-scale stress tensor is then parameterized following Lilly [1962] as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0024(9)
where Km is the subgrid viscosity
urn:x-wiley:19422466:media:jame20180:jame20180-math-0025(10)
Here λ is the subgrid length scale proportional to the grid volume, Δ = ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0026) urn:x-wiley:19422466:media:jame20180:jame20180-math-0027, and the Smagorinsky constant Cs. It is varied in vertical following Mason and Brown [1999] as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0028(11)
In the case that the grid is vertically stretched, the maximum grid volume in the first model level near the surface is used for Δ. The other terms in equations 9-11 are the von Karman constant κ, the turbulent Prandtl number Prt, and urn:x-wiley:19422466:media:jame20180:jame20180-math-0029 where urn:x-wiley:19422466:media:jame20180:jame20180-math-0030 is the Favre-filtered rate of strain tensor given by
urn:x-wiley:19422466:media:jame20180:jame20180-math-0031(12)
The Richardson number, Ri, in equation 10 is calculated as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0032(13)
where N and Nm are the dry and the moist Brunt Väisälä frequency defined, following Durran and Klemp [1982], as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0033(14)

Here Γd and Γm are the dry and the moist adiabatic lapse rate, respectively, Lv is the latent heat of vaporization, and θ is the potential temperature.

The thermodynamic equation expressed in terms of the density potential temperature and the budget equations for the tracers in ICON are expressed as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0034(15)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0035(16)

Here urn:x-wiley:19422466:media:jame20180:jame20180-math-0036 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0037 represent the forcing from the slow-physics (i.e., radiation) and the fast-physics parameterizations (i.e., saturation adjustment, cloud microphysics, and turbulence). Note that urn:x-wiley:19422466:media:jame20180:jame20180-math-0038 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0039 in equations 5-7 are also categorized as fast-physics. As the name suggests, the slow-physics are called less frequently compared to the fast-physics and therefore their tendencies are stored to be integrated with the governing equation. The fast-physics, on the other hand, are called every physics time step to sequentially update the prognostic variables and therefore do not provide tendencies to the governing equations. The sequential coupling between the turbulence parameterization and the dynamics is explained in Appendix Appendix A.

Equations 5-7 and equation 15, together with the slow-physics tendencies, are integrated in time using the two-time level predictor-corrector scheme except for the terms corresponding to the vertical sound-wave propagation, which are integrated implicitly. Tracers in equation 16 are integrated using a flux-form semi-Lagrangian scheme for its better conservation properties. As rather small time steps are required for the dynamics to maintain numerical stability, it is substepped several times between successive calls to the physical parameterizations. In the default configuration, physics time step is 5 times larger than the dynamics time step.

Radiation and the cloud microphysics have been turned off because of the idealized nature of the test cases. Therefore, urn:x-wiley:19422466:media:jame20180:jame20180-math-0040 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0041 (for urn:x-wiley:19422466:media:jame20180:jame20180-math-0042) only represent the effects of subgrid turbulent diffusion and condensation. That is,
urn:x-wiley:19422466:media:jame20180:jame20180-math-0043
Effects of condensation are calculated using the standard saturation adjustment scheme that assumes a uniform distribution of temperature and humidity within a grid box [Sommeria, 1976]. Using equations 1 and 2, the subgrid turbulent term can be expressed in terms of the prognostic variables as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0044
where
urn:x-wiley:19422466:media:jame20180:jame20180-math-0045(17)

Here urn:x-wiley:19422466:media:jame20180:jame20180-math-0046 is the eddy diffusivity coefficient, which is assumed to be same for urn:x-wiley:19422466:media:jame20180:jame20180-math-0047 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0048. urn:x-wiley:19422466:media:jame20180:jame20180-math-0049 is set to 1/3.

We note that the governing equations in ICON differ from standard LES models in many ways. For example, in ICON only one of the horizontal velocity components (v1) is prognosed while the other component (v2) is diagnosed, and both the thermodynamic and the tracer equations use nonconservative variables. Standard LES models, on the other hand, typically prognose both the horizontal velocity components and use conserved (in nonprecipitating convective motions [Deardorff, 1980]) thermodynamic variables like liquid water potential temperature (θl) and total liquid water (qt). These differences are discussed in more detail in section 4 when the other two LES models are introduced.

3 Numerical Implementation

In the following, we discuss in detail the implementation of the subgrid-scale turbulence term on the ICON grid. Details on the coupling between the dynamics and the turbulence parameterization, and the time integration of the turbulent diffusion terms are given in Appendix Appendix A. Note that the use of tilde urn:x-wiley:19422466:media:jame20180:jame20180-math-0050 and overbar urn:x-wiley:19422466:media:jame20180:jame20180-math-0051 on filtered variables are dropped henceforth for notational simplicity. The implementation of the diffusion terms presented here assumes a flat surface that is suitable for the idealized runs discussed in this paper. The full formulation over nonflat surfaces will be discussed in a subsequent work.

The local coordinate system used for the discretization is indicated in Figure 1 by the unit vectors 1, 2, and 3. The (right) schematic shows two triangles, adc and dbc with common edge cd, where e is the center of the local axes. The (circum-) centers of the triangles are indicated by p and q. We follow the C-gridtype arrangement of Arakawa [1966]: v1 is located at the edge-center e of the full (main) vertical levels; v2 is located at the center of the dual-edge pq of the full levels, as indicated by the red line(s) in the figure, which by design coincides with e; v3 is located at the cell-center p of the half (interface) vertical levels; and temperature, density, and tracers are stored at the cell-centers of the full vertical levels. The cell-centers are also referred to as mass points in the text.

The various interpolation operators, for an arbitrary variable urn:x-wiley:19422466:media:jame20180:jame20180-math-0052, used in this section are:
  1. urn:x-wiley:19422466:media:jame20180:jame20180-math-0053: a RBF reconstruction from edges to vertices.
  2. urn:x-wiley:19422466:media:jame20180:jame20180-math-0054: a linear interpolation from mass points or vertices to edges.
  3. urn:x-wiley:19422466:media:jame20180:jame20180-math-0055: a bilinear interpolation from edges to mass points.
  4. urn:x-wiley:19422466:media:jame20180:jame20180-math-0056: an area-weighted interpolation from mass points to vertices.
  5. urn:x-wiley:19422466:media:jame20180:jame20180-math-0057: a linear interpolation from main levels to interface levels (quadratic extrapolation at the surface).
  6. urn:x-wiley:19422466:media:jame20180:jame20180-math-0058: a linear interpolation from interface levels to main levels.

The letters next to the overbar urn:x-wiley:19422466:media:jame20180:jame20180-math-0059 here indicate the location to which the interpolation is performed, except for urn:x-wiley:19422466:media:jame20180:jame20180-math-0060 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0061, which indicate interpolation from edge-to-vertices and cells (mass points)-to-vertices, respectively. RBF reconstruction in ICON uses the Gaussian basis function, urn:x-wiley:19422466:media:jame20180:jame20180-math-0062, with shape parameter, β, tuned for great circle distances r [Wan et al., 2013]. For the flat surfaces, the basis function uses the Cartesian distance as r, which requires changing β for better performance. For now, it has been set to twice the length of the dual-edge (pq), which gives reasonable accuracy but further investigation is required.

The discrete operators used in this section are
urn:x-wiley:19422466:media:jame20180:jame20180-math-0063(18)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0064(19)
where urn:x-wiley:19422466:media:jame20180:jame20180-math-0065 is half of the vertical spacing at a given level. In the following, it will become clear that these difference operators are intended for implementing a second-order central difference scheme. For spherical triangles, however, the horizontal differences in the edge-normal direction may not be truly centered, especially when the triangles become distorted by the nonuniform tiling of the sphere. Errors because of this are generally small, and they decrease as the resolution increases, but such distortion limits the order of the discrete operators. For the idealized test cases presented in this paper, these difference operators are truly centered because the mesh is on a plane with equilateral triangles.

3.1 Subgrid Viscosity

Different strategies are used to calculate strain rates for the subgrid viscosity (Km) and for the turbulent fluxes. While they are estimated at their natural locations for the turbulent fluxes (shown in section 3.2), for Km they are all estimated at the edge-center (e.g., e in Figure 1) of the full levels for coding convenience.
urn:x-wiley:19422466:media:jame20180:jame20180-math-0066(20)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0067(21)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0068(22)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0069(23)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0070(24)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0071(25)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0072(26)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0073(27)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0074(28)

The above set of equations give strain rates and subgrid viscosity at the edge-center on the full levels. Therefore, interpolation is required to calculate Km and Kh at the interface levels for the diffusion of v1, θ, and all the tracers. The effectiveness of the diffusive transport is found to be very sensitive to this interpolation [Stevens et al., 1999]. Some advocate the use of geometric interpolation over arithmetic interpolation [Patankar, 1980, p. 197], but the observations by Stevens et al. [1999] suggest otherwise. We also performed simulations to investigate the effect of arithmetic interpolation over geometric interpolation, and found that the latter reduces the value of Km significantly, making the model unstable. In order to further understand this sensitivity, we also calculated Km at the half level cell-centers by first interpolating |S| from edges- to the cell- centers and then vertically to the half levels, which is normally done in LES models based on regular grids. The results obtained from both these approaches for the case in section 4.2 were almost indistinguishable from each other (not shown).

3.2 Turbulent Fluxes

The location of the stress terms τ1l and τ3l (same as that of S1l and S3l) used in the calculation of turbulent fluxes are shown in Figure 2, which shows one of the triangles (acda) in Figure 1. It is advised to follow both Figures 1 and 2 to understand the discretization explained below. Also, note that the location of a variable, if required, is indicated by |() which should not be confused by the letters used as superscript indicating interpolation scheme. Detailed derivation of some of the turbulent fluxes is given in Appendix Appendix B for reference.

Details are in the caption following the image

Schematic showing the locations at which the strain rates Sij are defined for (left) horizontal and (right) vertical turbulent diffusion. The full and half vertical levels are indicated by k and urn:x-wiley:19422466:media:jame20180:jame20180-math-0075, respectively, and the triangles at the half levels are marked by dashed line. Naming convention same as that of Figure 1 is used. The direction vectors are also indicated in the middle.

The subgrid diffusion term in equation 5 is calculated at the edge-center by discretizing the right-hand side (RHS) of equation 8
urn:x-wiley:19422466:media:jame20180:jame20180-math-0076(29)
in flux-form as (see Appendix Appendix B for detailed derivation)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0077(30)

Here urn:x-wiley:19422466:media:jame20180:jame20180-math-0078 that appears in equation 9, which is essentially one-third of the total divergence. All the terms on the RHS of the above equation are integrated in time using Euler explicit except for the term urn:x-wiley:19422466:media:jame20180:jame20180-math-0079 contributing to the vertical diffusion, which is integrated using Euler implicit (see Appendix Appendix A for details).

The subgrid turbulence term for equation 6 can also be written in a similar manner at the v3 point as (written without the density term)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0080(31)
It is important to note that the terms urn:x-wiley:19422466:media:jame20180:jame20180-math-0081 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0082 are first calculated at edge-center of the half levels (see Figure 2) and then interpolated to the cell-center as indicated by urn:x-wiley:19422466:media:jame20180:jame20180-math-0083 in the equation above. Fully discrete set of equation reads as (dropping out urn:x-wiley:19422466:media:jame20180:jame20180-math-0084 for clarity)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0085
Once again, all the terms on the RHS of equation 31 are integrated in time using Euler explicit except for the term urn:x-wiley:19422466:media:jame20180:jame20180-math-0086, which is integrated using Euler implicit. The discretization for the turbulent diffusion of cell-centered variables (θ and urn:x-wiley:19422466:media:jame20180:jame20180-math-0087) is straightforward. The formulation presented below is for equation 17 but the same is applicable for tracers as well,
urn:x-wiley:19422466:media:jame20180:jame20180-math-0088(32)

Here ac is the area of the triangle, fo is the orientation factor (±1) indicating the orientation of the triangle edge [Wan et al., 2013]. The first term on the RHS of equation 32 is the horizontal diffusion which is obtained using the divergence theorem by summing the fluxes across the three edges of the triangle. Similar to the momentum equations, the first term on the RHS of equation 32 is integrated explicitly in time whereas the second term is integrated implicitly.

It is clear that the discretization of the turbulent diffusion terms is flux-conservative. Meeting this conservation requirement on a triangular grid, while maintaining the overall computational performance, is not trivial. This is primarily because of the several interpolations involved which are computationally expensive, and result in loss of accuracy, and depending on the type of interpolation function, smoothening of the interpolant. For some asymptotic descriptions this effectively implies an A-grid implementation [Gassmann and Herzog, 2008]. Despite this overhead, the preliminary study revealed that the turbulence scheme in ICON takes about 20% of the overall computational time, which is comparable to the performance of the similar turbulence scheme implemented in a structured grid in one of the standard LES model (UCLA-LES) used here.

3.3 Surface Boundary Condition

Surface fluxes in ICON are parameterized by the drag-law formulation. Surface fluxes of sensible and latent heat in that formulation are written as:
urn:x-wiley:19422466:media:jame20180:jame20180-math-0089(33)
urn:x-wiley:19422466:media:jame20180:jame20180-math-0090(34)
where the subscript s indicate the value at the surface, C is the bulk-aerodynamical transfer coefficient for turbulent heat exchange at the surface parameterized in ICON following Louis [1979], and urn:x-wiley:19422466:media:jame20180:jame20180-math-0091 is the horizontal velocity at the first model level. Surface temperature, θs, is required in equation 33 which is either prescribed, for example, for the idealized simulations, or prognosed by the land-surface model.
Since the subgrid viscosity is calculated at the full levels in the present formulation, Brunt Väisälä frequency (N and Nm) is needed to be calculated at the first model level, which in turn requires θs. For the idealized simulations, if not prescribed, it is obtained using the integrated form of the flux-profile relationships [Louis, 1979] that assumes logarithmic profile for mean quantities
urn:x-wiley:19422466:media:jame20180:jame20180-math-0092(35)
where the constant R (=0.74) is the ratio of the drag coefficients for momentum and heat in the neutral limit, z0 is the roughness length for momentum, urn:x-wiley:19422466:media:jame20180:jame20180-math-0093 is the surface layer temperature scale
urn:x-wiley:19422466:media:jame20180:jame20180-math-0094(36)
and L is the Monin-Obukhov-scale height
urn:x-wiley:19422466:media:jame20180:jame20180-math-0095(37)
where κ is the von Kárman constant, and urn:x-wiley:19422466:media:jame20180:jame20180-math-0096 is the friction velocity. The functions ψ have the same form as in Louis [1979] and are therefore not reproduced here.

3.4 Lateral Boundary Condition

Doubly periodic boundary conditions are required for the simulations performed in the present work. This is achieved by treating the grid as a pseudo 2-D torus so that the boundaries in the respective coordinate directions are joined to each other. In order to incorporate this geometry in ICON, modifications have been made in the calculation of the interpolation coefficients and the discrete operators so that it uses Cartesian coordinates instead of the default spherical coordinates. Option is also available to use open lateral boundary conditions using data from the European Center for Medium-Range Weather Forecast (ECMWF), Consortium for Small-Scale Modeling (COSMO) model, and the ICON model itself for limited-area simulations.

4 Benchmarking ICON-LES

Within the framework of HD(CP)2, two simulations are performed to benchmark ICON as a model capable of doing large eddy simulations: the dry convective boundary layer (DCBL) and the cloud topped boundary layer (CTBL). Both simulations are performed in a doubly periodic domain with prescribed surface fluxes and zero flux at the top. The model configuration of ICON capable of performing LES is termed as ICON-LES (hereafter simply ICON). The setups used here have been studied extensively in the literature, both numerically and experimentally (see e.g., Deardorff [1972]; Willis and Deardorff [1974]; Moeng [1984]; Wyngaard [1985]; Schmidt and Schumann [1989] and the references therein for DCBL, and Nitta and Esbensen [1974]; Sommeria [1976]; Nicholls and LeMone [1980]; Siebesma and Cuijpers [1995]; Stevens [2007], and the references therein for CTBL). For this reason and because they test the representation of dry convective turbulence and its coupling to the moist processes, they are good candidates for benchmarking LES models simulating atmospheric boundary layer turbulence.

To further assist the benchmarking process, simulations are also performed using two well-established and standard LES models: (a) UCLA-LES [Stevens et al., 2005] (hereafter simply UCLA) and (b) PALM [Raasch and Schröter, 2001; Maronga et al., 2015]. The numerics in ICON and the standard LES models differ significantly because of the differences in the underlying grid. Some of the major differences in the model dynamics, which we think will be helpful in understanding the differences in the subsequent sections, are listed below:
  1. ICON uses the fully compressible set of equations whereas UCLA uses the anelastic approximation [Ogura and Phillips, 1962] and PALM uses the Bousinessq approximation [Dutton and Fichtl, 1969]. Therefore, ICON is forced to use a smaller time step allowing for the sound-wave propagation. For the simulations performed here, the dynamical time step in ICON is about 0.02 times that of PALM and UCLA.
  2. ICON solves the edge-normal velocity component, therefore it has three degrees of freedom in each grid cell whereas it is four in the UCLA and PALM.
  3. ICON uses θρ as the prognostic variable for the thermodynamic equation, whereas UCLA and PALM use θl as the prognostic variable.
  4. ICON uses qv and ql as the prognostic variables to represent moist processes, whereas UCLA and PALM use qt that is a conserved quantity in the absence of precipitation.
  5. The advection scheme for momentum equations in ICON is second-order accurate in both vertical and horizontal directions. In addition, ICON uses fourth-order artificial numerical dissipation for numerical stability in the momentum equations. In UCLA and PALM, the momentum advection schemes are fourth-order central and fifth-order upwind, respectively.
  6. The advection scheme for the thermodynamic equation in ICON uses the second-order upwind [Miura, 2007] and central scheme for flux reconstruction in horizontal and vertical directions, respectively. In addition, a Smagorinsky type second-order numerical dissipation is applied on temperature fields for stability reasons. In UCLA and PALM, these are second-order (with flux limiter) and fifth-order upwind, respectively.
  7. The tracer advection scheme in ICON uses the second-order upwind scheme by Miura [2007] in horizontal (there is an option for third-order upwind also) and third-order piecewise parabolic method by Colella and Woodward [1984] in vertical direction with flux limiter. In UCLA and PALM, these are second-order (with flux limiter) and fifth-order upwind, respectively.
  8. ICON uses a second-order Predictor-Corrector time integration [Zängl et al., 2014] scheme whereas UCLA and PALM both use a third-order Runge-Kutta scheme.
  9. ICON and UCLA use the classical Smagorinsky turbulence scheme whereas PALM uses the turbulent kinetic energy based scheme of Deardorff [1980].

In addition to the artificial numerical dissipation, a three-dimensional divergence damping of fourth-order is used in ICON to damp the acoustic waves. The damping coefficient is set to 0.0025Δt. It is advisable for LES simulations to minimize the use of dissipation so that it does not interfere with the subgrid diffusion. However, the use of the artificial dissipation cannot be avoided for simulations with real orography, therefore, we decided to switch it on to see its effect on the resolved scales in the benchmark cases.

The points mentioned above clearly show the advantages that standard LES models (like UCLA and PALM) have over the general purpose models like ICON, which must be kept in mind while analyzing the results presented in the subsequent sections. A summary of the formal accuracy of the (advection) schemes used in these models is provided in Table 1.

Table 1. Formal Order of Accuracy of the Schemes Used in the Discretization of the Various Advective Terms in ICON, UCLA, and PALM
Model Momentum (Horizontal, Vertical) Thermodynamics (Horizontal, Vertical) Tracer (Horizontal, Vertical)
ICON (2,2) (2,2) (2,3)
UCLA (4,4) (2,2) (2,2)
PALM (5,5) (5,5) (5,5)

4.1 Dry Convective Boundary Layer

4.1.1 Simulation Design

The dry convective boundary layer simulation is initiated with a potential temperature profile constantly increasing with height from a surface temperature of θs = 290 K at a constant lapse rate of Γ = 0.006 Km−1. The initial wind is set to zero and the turbulence is triggered by adding random perturbations to the temperature field up to a height of 300 m in ICON and up to 1600 m in UCLA and PALM. The boundary layer then develops in time because of the fixed kinematic surface heat flux urn:x-wiley:19422466:media:jame20180:jame20180-math-0097 K ms−1. Note that the overbar urn:x-wiley:19422466:media:jame20180:jame20180-math-0098 henceforth denotes spatial average over horizontal slabs of the computational domain and the prime urn:x-wiley:19422466:media:jame20180:jame20180-math-0099 is used to indicate the deviation from this slab average. Kinematic units are used to force the models with the same surface fluxes irrespective of the way density is handled by them individually.

Simulations are performed for 3 h on a domain of size 9.6 × 9.6 km2 in horizontal and 3.2 km in the vertical with varying grid spacings to study grid convergence. Resolutions considered for the present case are ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0100): (a) 100 m, (b) 50 m, and (c) 25 m. Ascribing similar grid resolution to a triangular mesh is not possible. In practice, either the length between the two cell-centers or the length of the triangle edge or the square root of the area of the triangle is used to identify the grid resolution. Following the latter, and noting that the triangles in a doubly-periodic flat plane are equilateral, the triangle edge length (Δl) corresponding to the resolution (Δxy) of a regular grid cell is obtained by equating the area of the cells in the two grids:
urn:x-wiley:19422466:media:jame20180:jame20180-math-0101(38)

This definition ensures the same number of grid cells in both meshes. However, the spectral analysis of the simulation data revealed that the actual grid resolution is lower than the present estimation. This is discussed further in Appendix Appendix C. It is also worth noting that the same number of grid cells in both triangular and rectangular meshes implies less degrees of freedom in ICON, which has three velocity components per grid cell, compared to the rectangular mesh, which has four velocity components per grid cell.

4.1.2 Time Evolution

We start by analyzing the time evolution of the horizontal mean of boundary layer height as simulated by the three models in Figure 3. It is calculated as the height (zi) at which the vertical gradient of θ is at maximum. Different model resolutions are indicated in each figure. The jumps in zi during the first 30–75 min (progressively less for higher resolutions) indicate the spin-up phase during which turbulence is first establishing itself. The difference in UCLA and PALM as compared to ICON is because of differences in initialization (i.e., the height of initially prescribed random noise). ICON shows large oscillations for Δ = 100 m while keeping the (temporal) mean lower than the other LES models. One clear feature that is seen in these results, which has been noted previously, e.g., by Sullivan and Patton [2011], is that the entrainment rate (dzi/dt) decreases in all the models as the grid spacing Δ is reduced. The effect of the larger entrainment rate at coarse resolution is seen as enhanced warming in Figure 4.

Details are in the caption following the image

Time evolution of boundary layer height at the indicated resolutions.

Details are in the caption following the image

Vertical profile of mean potential temperature at the indicated resolutions. The dashed line is the initial profile.

4.1.3 Vertical Structure

Vertical profiles of θ, averaged spatially over the horizontal domain and temporally over the last 15 min of the simulation (sampled every 30 s), are shown in Figure 4. For Δ = 100 m, ICON captures the gross features, like the superadiabatic layer near the surface and the interfacial layer, just like standard LES models. The profile changes appreciably as the resolution is increased and ICON remains trustful to what are by now familiar patterns, e.g., as shown by the UCLA and PALM. UCLA entrains more (see Figure 6 near zi), the effect of which is more heating in the mixed layer and a deeper interfacial layer. At Δ = 12.5 m, the θ profile in PALM and UCLA (not shown) does not change much indicating a nearly converged solution at Δ = 25 m. This can be also realized in Figure 5 where the convergence of horizontally averaged θ at the boundary layer height (calculated at the end of simulation) with increasing resolution is shown. From the figure it is clear that all the models approach convergence at the same rate with ICON following PALM more closely.

Details are in the caption following the image

Mean potential temperature at the boundary layer height with increasing resolution.

Details are in the caption following the image

Mean vertical heat flux profile at the indicated resolutions. The solid lines are the total and the dashed lines are the subgrid-scale part of the total flux. The zero value is indicated by dashed line.

The subgrid contribution to the total flux should decrease with increasing resolution, so that the resolved and the total flux coincide except near the surface and inversion. To study this, the total and subgrid part of the mean vertical turbulent heat flux is plotted in Figure 6 for different resolution. The resolved flux in ICON is obtained as
urn:x-wiley:19422466:media:jame20180:jame20180-math-0102(39)
which leads to some diagnostic error which is visible as a small bump near the surface in all the panels (e.g., at z = 100 m in the middle panel). The other models avoid such errors by directly using the advective fluxes as the resolved scale fluxes, because urn:x-wiley:19422466:media:jame20180:jame20180-math-0103 due to incompressibility and periodicity.

At Δ = 100 m, PALM has a small (positive) heat flux immediately above the inversion which is not seen for the other models. This is an artifact indicative of insufficient numerical damping in the PALM model. A similar pattern was found in Moeng [1984] who used a central difference scheme in the vertical. The subgrid-scale flux in ICON is almost the same as in UCLA because they use the same turbulence scheme, whereas it is slightly smaller in PALM, at all resolutions. As expected, subgrid-scale fluxes of ICON (and other models) decrease, hence the resolved scale fluxes increase, as the resolution is increased. In general, PALM shows the highest turbulent flux in the surface layer. At the inversion, urn:x-wiley:19422466:media:jame20180:jame20180-math-0104 in ICON and PALM are about 10% of the surface value (at all resolutions), whereas it is about 18% in UCLA. These values are within the expected range of 10–30% [Stull, 1988, p. 478]. The deeper and (negative) stronger inversion in UCLA explains its larger cooling rate in the inversion layer in Figure 4 [Garcia and Mellado, 2014].

Previous model intercomparison studies have shown that significant spread exists between models for vertical velocity variance ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0105) [Siebesma et al., 2003; Stevens et al., 2001]. In order to quantitatively assess urn:x-wiley:19422466:media:jame20180:jame20180-math-0106, we have used the DNS results of Garcia and Mellado [2014] for reference in Figure 7. Although the subgrid contribution to urn:x-wiley:19422466:media:jame20180:jame20180-math-0107 is missing in the LES models (because of its unavailability from ICON), we have noted from UCLA and PALM that its contribution is fairly small to affect the discussions below. In order to dimensionalize the DNS results, zi = 700 m and a convective velocity scale urn:x-wiley:19422466:media:jame20180:jame20180-math-0108 ms−1 from ICON results at Δ = 25 m are used.

Details are in the caption following the image

Mean vertical velocity variance profile (resolved) at the indicated resolutions together with a reference DNS profile.

urn:x-wiley:19422466:media:jame20180:jame20180-math-0109 is somewhat underestimated by all the models at Δ = 100 m, except for ICON and UCLA which overestimate it above z = 400 m and z = 600 m, respectively. At Δ = 50 m, all models underestimate the variance below z urn:x-wiley:19422466:media:jame20180:jame20180-math-0110 200 m. Beyond z = 400 m, ICON and UCLA overestimate the variance whereas PALM underestimates it. At the highest resolution, PALM correctly captures the variance throughout the boundary layer, except with an overestimation near the top. At this resolution, both ICON and UCLA show an improvement for z < 200 m but they overestimate the DNS above 200 m. This consistent overestimation by ICON and UCLA can be understood by noting that in the absence of mean (horizontal) wind and subsidence,
urn:x-wiley:19422466:media:jame20180:jame20180-math-0111(40)
for the present case. By closely inspecting Figure 6, we note that urn:x-wiley:19422466:media:jame20180:jame20180-math-0112 (= urn:x-wiley:19422466:media:jame20180:jame20180-math-0113 for DCBL) is also overestimated by ICON and UCLA in a similar manner.

Results in this section suggest that ICON performs satisfactorily in comparison to UCLA and PALM. For Δ = 100 m, ICON captures the salient features of the boundary layer better than the models run at a few km resolution without parameterized boundary layers (e.g., cloud resolving models). ICON also shows grid convergence which is necessary to ensure the consistency of the discrete system. We focused on the vertical heat flux to tune the value of the Smagorinsky constant, and urn:x-wiley:19422466:media:jame20180:jame20180-math-0114 was found to give the best results. The model became unstable for urn:x-wiley:19422466:media:jame20180:jame20180-math-0115. We also performed simulations using urn:x-wiley:19422466:media:jame20180:jame20180-math-0116, instead of equation 11, to get reduced values of urn:x-wiley:19422466:media:jame20180:jame20180-math-0117 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0118 near the surface. The fluxes did improve near the surface but we also saw wiggles near the surface in the mean profile of temperature, so we decided to stay with equation 11.

4.1.4 Spatial Structure

It is also instructive to look at the spatial structure of the flow field for better assessment of the model. The most commonly known spatial feature in convective boundary layers is the presence of a spoke-like pattern near the surface [Mason, 1989; Schmidt and Schumann, 1989]. In order to find out whether ICON can resolve such a pattern on its triangular grid, contours of the fluctuations of the virtual potential temperature and vertical velocity in a horizontal cross section are plotted in Figure 8 for Δ = 25 m. The figures clearly show those spoke-like patterns in both urn:x-wiley:19422466:media:jame20180:jame20180-math-0119 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0120 contours. By visually inspecting the figure, we find that the typical length of the segments forming this pattern is about 1.3zi, similar to the value reported in Schmidt and Schumann [1989]. As expected, urn:x-wiley:19422466:media:jame20180:jame20180-math-0121 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0122 are strongly correlated as required by the sign of vertical heat flux.

Details are in the caption following the image

Contour plots of (left) urn:x-wiley:19422466:media:jame20180:jame20180-math-0123 (K) and (right) urn:x-wiley:19422466:media:jame20180:jame20180-math-0124 (ms−1) in a horizontal plane at z = 137 m drawn at the end of the simulation for Δ = 25 m.

4.2 Cloud Topped Boundary Layer

4.2.1 Simulation Design

For the benchmarking of ICON as an LES model it is necessary to ensure that it correctly represents the strong nonlinearity of moist processes. This is achieved by including (nonprecipitating) clouds in a setup otherwise similar to that for the dry convective boundary layer. We follow the idealized setup of cumulus convection described in Stevens [2007]. The simulation has been initialized with the same conditions as the DCBL case to carefully examine the effect of moist convection as opposed to the dry case. In addition, moisture is initialized with an analytic profile for specific humidity given by
urn:x-wiley:19422466:media:jame20180:jame20180-math-0125(41)

The factor 0.0088 is roughly 0.74 times the saturation specific humidity at the surface. This profile corresponds to an exponentially decreasing relative humidity, which in turn ensures an initial equivalent potential temperature profile decreasing with height which is necessary for the growth of a conditionally unstable cloud layer [Stevens, 2007]. The initial temperature and humidity profiles are shown in Figures 10a and 10b as dotted lines. The initial wind is set to zero as in the DCBL case.

Similar to the DCBL, the boundary layer is set to grow in time due to a fixed forcing at the surface. Instead of fixing the sensible and the latent heat flux at the surface, an extra degree of freedom is provided to the system by allowing these fluxes to change in time while maintaining a constant surface buoyancy flux urn:x-wiley:19422466:media:jame20180:jame20180-math-0126 = 0.0007 m2 s−3 for the reference temperature Θ = 290 K. The surface thermodynamic variables, θs and urn:x-wiley:19422466:media:jame20180:jame20180-math-0127 are obtained by iterating the following equation:
urn:x-wiley:19422466:media:jame20180:jame20180-math-0128(42)
where urn:x-wiley:19422466:media:jame20180:jame20180-math-0129 is the saturation humidity at surface pressure ps = 1020 hPa. Here urn:x-wiley:19422466:media:jame20180:jame20180-math-0130 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0131 are the horizontally averaged potential temperature and specific humidity at the first model level, and urn:x-wiley:19422466:media:jame20180:jame20180-math-0132. The turbulent exchange coefficient, urn:x-wiley:19422466:media:jame20180:jame20180-math-0133, has been fixed to 0.02 ms−1 in all the models. For a typical value of the transfer coefficient C (∼0.001–0.002) over sea surface at low wind speed [Beljaars, 1994], urn:x-wiley:19422466:media:jame20180:jame20180-math-0134 = 0.02 ms−1 implies very high wind speed of order 10 ms−1. Such large wind speeds are unrealistic, but because they only serve to determine the surface fluxes, this only effects the strength of the near surface gradients of temperature and moisture.

Results in the previous section clearly show convergence of ICON as the grid resolution is increased. The intermediate resolution of Δ = 50 m was found to be in good agreement with Δ = 25 m. Similar conclusion was drawn for the CTBL case with a base simulation using Δ = 50 m and another simulation with double the number of points in the vertical. It is for this reason that we only use the results from the base simulation in this section. The domain size is kept the same as in the dry case with doubly periodic boundary conditions and the solution is integrated for 25 h.

4.2.2 Time Evolution

Figure 9a shows the time evolution of the boundary layer height as simulated by the three models. As noted earlier in Figure 3, UCLA and PALM start with a big jump due to the random initialization. Before the cloudy layer develops, the evolution of zi in all the models is consistent with the DCBL case at Δ = 50 m in the sense that ICON has lowest zi and PALM has the highest. UCLA and PALM match well until t = 20 h after which they depart slowly. ICON simulates a shallower boundary layer (by 150–200 m). The rate of boundary layer growth, urn:x-wiley:19422466:media:jame20180:jame20180-math-0135, in ICON (and in both other models) increases due to the developing cloud layer after urn:x-wiley:19422466:media:jame20180:jame20180-math-013612 h. This is consistent with earlier findings [Stevens, 2007] that urn:x-wiley:19422466:media:jame20180:jame20180-math-0137 varies as t1∕2 when the boundary layer is dry, and as t when the cloud layer develops.

Details are in the caption following the image

Time evolution of the indicated quantities for the cloud topped boundary layer simulation. The black solid line in Figure 9b is the mean over all the three models used in the present paper, and the spread of 50% [from Siebesma et al., 2003] about this ensemble mean is indicated by the bars. The magenta lines in Figures 9c and 9d are from ICON after doubling the number of vertical levels.

Details are in the caption following the image

Mean vertical profiles of (a) urn:x-wiley:19422466:media:jame20180:jame20180-math-0138, (b) urn:x-wiley:19422466:media:jame20180:jame20180-math-0139, (c) urn:x-wiley:19422466:media:jame20180:jame20180-math-0140, and (d) urn:x-wiley:19422466:media:jame20180:jame20180-math-0141. The initial conditions are indicated by the black dashed lines, and the black solid line in Figure 10c is the mean over the three models and the uncertainty range from Siebesma et al. [2003].

Clouds are triggered at t urn:x-wiley:19422466:media:jame20180:jame20180-math-0142 5 h in ICON and PALM, and at t urn:x-wiley:19422466:media:jame20180:jame20180-math-0143 10 h in UCLA as seen in Figure 9b. All models maintain a near constant cloud fraction after t = 15 h. ICON saturates at a cloud fraction of 0.2, PALM at 0.13, and UCLA at 0.1. In order to ensure that the spread in cloud fraction lies within the known limits of uncertainty due to different model configurations, an uncertainty range is added in the figure corresponding to the spread found in the LES intercomparison study of cumulus convection as observed during the Barbados Oceanographic and Meteorological Experiment (BOMEX) by Siebesma et al. [2003]. The range shows a spread of 50% about the ensemble mean which is marked as a thin solid line. It is clear that all the models stay within this range despite all the differences.

The surface temperatures in all the models match quite well, except for some initial oscillations in ICON (see Figure 9e). These oscillations are more apparent in the surface fluxes in Figures 9d and 9e for t < 10 h. At first we thought that the oscillations are due to the fact that ICON has lesser degree of freedom than UCLA and PALM, therefore we did simulations with higher horizontal resolution so that the degree of freedom in ICON becomes equal to the other models (see discussions in Appendix Appendix C) and also with doubled vertical resolution (Δ3 = 25 m). Simulation with higher horizontal resolution had minimal effect (not shown). Simulation with doubled vertical resolution, as indicated by ICON–HI in Figures 9c and 9d, show that the oscillations are greatly reduced in ICON suggesting that the oscillations are mainly due to grid-locking of the inversion height to model levels.

Besides these initial oscillations, the surface fluxes in the models evolve in the same fashion. ICON simulates higher urn:x-wiley:19422466:media:jame20180:jame20180-math-0144 (by 5 W m−2) throughout the simulation and smaller urn:x-wiley:19422466:media:jame20180:jame20180-math-0145 (by 20 W m−2) after the formation of the clouds, thereby maintaining a constant value of the buoyancy flux.

4.2.3 Vertical Structure

In this section we look at the vertical structure of the thermodynamic variables and the turbulent fluxes in ICON. The vertical profiles are obtained by averaging spatially in the horizontal direction and temporally over the last 30 min (sampled every 30 s). The liquid water potential temperature ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0146) profile in Figure 10a shows that ICON is about 0.2 K cooler compared to UCLA in the subcloud layer. Similar behavior was seen in the DCBL case. We believe that this additional cooling in ICON is due to the fact that a fraction of the heat transported from the surface is used in the expansion work in ICON, thereby generating mean vertical wind (see Figure 10d), which does not take place in the other two models because of the Bousinessq and anelastic assumptions. Furthermore, the subcloud layer top in ICON is slightly less than 700 m where UCLA and PALM converge. This implies a shallower cloud base in ICON which can also be seen in the ql profile in Figure 10c. The specific humidity profile from ICON lies exactly between that of PALM and UCLA as seen in Figure 10b.

As previously mentioned, ICON prognoses both ql and qv whereas PALM and UCLA only prognose qt. qc is then diagnosed using the condensation scheme of Sommeria and Deardorff [1977]. Despite this difference, the mean cloud liquid water in ICON stays within the range noted earlier by Siebesma et al. [2003], as indicated by the bars in Figure 10c. The reason for a lower cloud base in ICON can be understood by realizing that the air near the surface in ICON is moister than in the other models (by 0.2g kg−1) whereas the temperature is a little lower (by 0.2 K). Therefore, an air parcel ascending from the surface saturates much faster in ICON in comparison to PALM and UCLA. Furthermore, the slightly lower cloud top in ICON suggests enhanced mixing with the dry atmosphere above, in comparison to other models. This extra mixing near the cloud top in ICON is either due to the implicit diffusion in the tracer advection scheme or the flux limiter which gets more active near the cloud boundary in ICON because of the use of nonconservative tracers.

The lower maximum (negative) urn:x-wiley:19422466:media:jame20180:jame20180-math-0147 flux in ICON as seen Figure 11a is because of lower liquid water vertical flux ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0148) (see Figure 11c), which can be understood by noting that
urn:x-wiley:19422466:media:jame20180:jame20180-math-0149
Details are in the caption following the image

Turbulent flux (resolved) profiles of the indicated quantities in the cloud topped boundary layer simulation. The zero value is indicated by black dashed line.

Figure 11a also shows a small kink at the cloud base in ICON. This is probably due to the use of the nonconservative thermodynamic and moisture variables in ICON which creates a discontinuity at the cloud boundary. This effect, although very small in the present case, can generate significant errors in cases with strong convection.

The turbulent buoyancy flux in the subcloud layer of a CTBL is known to follow the DCBL case in a nondimensional sense [Stevens, 2007]. The maximum resolved buoyancy flux in DCBL case for Δ = 50 m was found to be around 84% of the surface flux prescribed in all models. Looking at the similar maxima in Figure 11b (near the surface) it appears that ICON resolves slightly less, around 76% of the prescribed value ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0150 = 25 W m−2) whereas UCLA and PALM resolve nearly 84%. However, as was mentioned during the discussion of Figure 6, this is simply a diagnostic error which arises due to the vertical averaging of the fluxes in ICON from main levels to the interface levels for plotting purposes. Also, Figure 11b shows two (positive) peaks as one would expect in a CTBL. The flux decreases linearly in the subcloud layer up to the cloud base where it resolves about 10% (negative) of urn:x-wiley:19422466:media:jame20180:jame20180-math-0151 in all models.

The resolved vertical flux of specific humidity in ICON is found to be smaller than in the other two models (see Figure 11c). In the subcloud layer it is a consequence of its lower surface moisture flux, whereas in the cloud layer it is probably due to the enhanced mixing near the cloud top in ICON. The liquid water vertical flux ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0152) in ICON compares very well with other models (see Figure 11d) despite the aforementioned differences in the way cloud liquid water is handled.

Vertical profiles of resolved vertical velocity variance by the three models are shown in Figure 12. Similar to the dry case in Figure 7 for Δ = 50 m, urn:x-wiley:19422466:media:jame20180:jame20180-math-0153 in ICON and UCLA are nearly the same till the lowest peak at urn:x-wiley:19422466:media:jame20180:jame20180-math-0154. At the peak and beyond that, the variance in ICON is slightly smaller than in UCLA in the subcloud layer, which is different from the dry case. In the cloud layer and above, the differences between the two is reduced again.

Details are in the caption following the image

Vertical velocity (resolved) variance in the cloud-topped boundary layer simulation.

Overall, the results of the CTBL simulation show that ICON follows the expected trend and agrees well with the standard LES models with some differences in and around the cloud layer. We believe that such differences are bound to show up because of the inherent differences in the model and experimental designs. Some of the differences in both DCBL and CTBL cases are also because of the artificial initial profiles of temperature and moisture that produces very thin boundary layer in the early phase of the simulation, which cannot be resolved with the prescribed grid resolution. While it is hard to argue how long and by how much this affects the simulation, we think that in the presented simulations such effects are relatively smaller after the first hour of the simulation. To confirm this, we also performed the BOMEX simulation [Siebesma et al., 2003] that does not suffer from similar initialization issues. The results from this case do not differ in any important way from those from the simulation of the CTBL case, and are therefore not presented here.

As already mentioned, several interpolations are required in ICON for spatial discretization which can lead to (numerical) errors and smoothening of the results. On top of that, unlike standard LES models that use high-order schemes for spatial discretization, the triangular grid in ICON poses a challenge on the use of higher-order schemes in the horizontal. It is, however, to be noted that the overall (spectral) accuracy of the models using such higher-order schemes decreases as the complexity of the experiment increases. For complex cases, the use of artificial second-order numerical dissipation becomes necessary to keep the model stable, which reduces the order of the leading truncation error term of the spatial discretization scheme to second order, thereby reducing the overall accuracy of the spatial discretization to first order [Ghosal, 1996].

5 Conclusion

This paper describes the extension of a unified modeling system for climate and weather forecast ICON (ICOsahedral Nonhydrostic) to a large-eddy simulation framework and its comparison against two well-established LES models.

The first part details the implementation of the turbulence scheme on the triangular grid used in ICON and the numerical technique used to integrate the diffusion equations. Conservative implementation of a three-dimensional turbulence scheme in a triangular grid is nontrivial and involves several interpolation operations. Despite these computational overheads, the turbulence scheme in ICON-LES takes less than 20% of the overall computational time in case of a dry convective boundary layer, which is comparable to the performance of the similar turbulence scheme implemented in the quadrilateral grid of UCLA-LES.

The second part of the manuscript deals with the validation of ICON-LES against two standard LES models (UCLA-LES and PALM) for two test cases: dry convective boundary layer, and cloud-topped boundary layer. Considering the compromises that ICON undergoes from being a unified modeling system, and that it is mostly second-order accurate in space whereas UCLA-LES is fourth and PALM is fifth-order accurate, the eddy-resolving capability of ICON-LES is fairly good. The turbulence scheme in ICON-LES captures the mean flow characteristics quite satisfactorily, and produces similar turbulent structure and statistics in comparison to the standard LES models. In particular,
  1. it was shown in section 4.1 that ICON-LES approaches toward a converged solution as the grid resolution is increased.
  2. the profiles of resolved vertical fluxes for both DCBL and CTBL indicate that the eddy-resolving capabilities of ICON-LES is comparable to that of the standard LES models.
  3. the lower cloud top in ICON-LES for the CTBL case suggests enhanced (numerical) mixing near the cloud top which could have been activated due to the use of nonconservative variables in ICON which creates a discontinuity at the cloud boundary.

Acknowledgments

The authors thank the German Federal Ministry of Education and Research (BMBF) for their funding to support the HD(CP)2 project (01LK1202E). We also thank Jade Garcia for providing us with the DNS data, the whole ICON development team at DWD and MPI-M for their excellent work, and the anonymous reviewers for their constructive suggestions. We specially thank Leonidas Linardakis for his continuous help during the course of the model development. The data used in this paper are available at the German Climate Computing Center and can be obtained from the corresponding author upon request.

    Appendix A: Coupling of Dynamics and Turbulence

    In this section, the sequential coupling of the dynamics with the turbulence parameterization in ICON is explained. When marching in time from nΔt to urn:x-wiley:19422466:media:jame20180:jame20180-math-0155 being the physics time step, let us indicate the variables after being updated from dynamics by urn:x-wiley:19422466:media:jame20180:jame20180-math-0156. Subgrid-scale diffusion is applied on equation 5 in three stages as following:
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0157(A1)
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0158(A2)
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0159(A3)

    The RHS of equation (A1) is composed of the first two terms in equation 30, representing the horizontal diffusion, all evaluated using variables from dynamics. This equation is integrated in time using Euler explicit and the tendency is stored in urn:x-wiley:19422466:media:jame20180:jame20180-math-0160. Vertical diffusion is then performed using equation (A2) where all the terms in the RHS are also evaluated using the urn:x-wiley:19422466:media:jame20180:jame20180-math-0161 variables, except for the term involving the vertical derivative of v1, which is evaluated implicitly using Euler implicit method. This is indicated in equation (A2) by the use of an intermediate time step urn:x-wiley:19422466:media:jame20180:jame20180-math-0162. The tendency from this equation is then combined with the horizontal tendency in the final equation to get the velocities at urn:x-wiley:19422466:media:jame20180:jame20180-math-0163.

    Subgrid-scale diffusion for the vertical momentum equation is similarly performed by splitting the RHS of equation 31 into the horizontal and vertical components. As for the thermodynamic and tracer variables, the implementation is described for θ following equation 32. That is,
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0164(A4)
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0165(A5)
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0166(A6)

    Here again, equation (A4) is integrated explicitly using the variables from dynamics and the tendency is stored in urn:x-wiley:19422466:media:jame20180:jame20180-math-0167, and equation (A5) is integrated implicitly storing the tendency in urn:x-wiley:19422466:media:jame20180:jame20180-math-0168. It is to be noted that Km and Kh are always evaluated using the variables from dynamics.

    Appendix B: Discretization of Turbulent Fluxes

    Here we derive some of the turbulent fluxes on ICON grid for the interested readers.

    Following steps are taken to derive equation 30. The first term on the RHS of above equation, which is due to the first term in the RHS of equation 29, is obtained as
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0169
    The second term in RHS of equation 30 involves some manipulation which can be realized by noting that
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0170

    At this point, if we make the assumption that urn:x-wiley:19422466:media:jame20180:jame20180-math-0171 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0172, and that v2 at nodes urn:x-wiley:19422466:media:jame20180:jame20180-math-0173 are also arithmetic averages of their neighboring vertices (e.g., urn:x-wiley:19422466:media:jame20180:jame20180-math-0174), after some manipulation we end up with the second term of the RHS of equation 30. The third term on the RHS of equation 30 is a straightforward expansion of urn:x-wiley:19422466:media:jame20180:jame20180-math-0175 utilizing the strain rates located at urn:x-wiley:19422466:media:jame20180:jame20180-math-0176 and urn:x-wiley:19422466:media:jame20180:jame20180-math-0177 in Figure 2.

    The first term on RHS equation 31, which is due to urn:x-wiley:19422466:media:jame20180:jame20180-math-0178, is evaluated at urn:x-wiley:19422466:media:jame20180:jame20180-math-0179 as (dropping out urn:x-wiley:19422466:media:jame20180:jame20180-math-0180 for clarity)
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0181

    Rest of the terms can be derived in a similar manner and are therefore not derived here.

    Appendix C: Grid Resolution in ICON

    It is mentioned at the outset that the results in this section are relevant for flat meshes only.

    One of the difficulties with the triangular mesh is an a-priori estimation of its grid resolution (should not be confused with the effective grid resolution). For the present experiments, we have defined the resolution as square root of the area of the triangle, which gives
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0182(C1)
    The other two definitions which are frequently used are the distance between the two cell-centers, which gives
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0183(C2)
    and, the triangle edge length that gives Δ3 = Δ l. Of course, urn:x-wiley:19422466:media:jame20180:jame20180-math-0184. An advantage of using the first definition is that it gives the same number of grid points as a regular mesh with an added disadvantage of having to deal with lesser degree of freedom per grid cell for velocity components. If one chooses to define a resolution such that the degrees of freedom remains the same as in a regular mesh, then for Ng number of total grid points in a regular quadrilateral mesh, urn:x-wiley:19422466:media:jame20180:jame20180-math-0185 grid points will be required in the ICON grid. Noting that the total number of grid points in an ICON grid is urn:x-wiley:19422466:media:jame20180:jame20180-math-0186, for a square domain of length L, we can estimate the required resolution to be
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0187(C3)

    In order to see how the estimation from equation (C1) compares with a regularly spaced grid model, we have compared the horizontal spectra of vertical velocity from ICON to that of PALM and UCLA for the DCBL case. These one-dimensional spectra are obtained by Fourier transforming the field along a horizontal line and averaging over all parallel lines at a fixed height and time. ICON spectrum is obtained by interpolating its velocity field to a regular grid with the same number of grid points as the standard LES models using distance weighted averaging technique. That is, the Δ1 = 50 m simulation results are interpolated to 192 × 192 grid points and the Δ1 = 25 m to 384 × 384 grid points. The corresponding spectra at z = 500 m are shown in Figure 13.

    Details are in the caption following the image

    The horizontal spectra of the vertical velocity at indicated resolutions in log-log scale. The data have been multiplied by some factors in order to avoid overlap. Also shown is the expected Kolmogorov's spectrum ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0188 k urn:x-wiley:19422466:media:jame20180:jame20180-math-0189) (black line), for urn:x-wiley:19422466:media:jame20180:jame20180-math-0190 50, 25 m.

    Figure 13 shows that the Kolmogorov's spectrum ( urn:x-wiley:19422466:media:jame20180:jame20180-math-0191 k urn:x-wiley:19422466:media:jame20180:jame20180-math-0192) is followed by all the models, and the representation of the larger scales improves as the resolution is increased to Δ1 = 25 m. The peculiarity of the ICON data is that it shows a small spectral bump toward the high wave numbers (see also Figure 14 where the ICON spectrum for Δ1 = 50 m is redrawn). This bump is indicative of aliasing in ICON data which lead to the accumulation of energy at the scales near the Nyquist limit. It means that 192 × 192 grid points is more than the actual number of points required to properly represent ICON data on a regular grid. After some trials with less number of grid points, we found that when interpolated to a 148 × 148 grid the bump in the spectrum almost disappears (see the blue colored line in Figure 14). This implies that 148 × 148 number of grid points in a regular grid is a fairly good representative of the ICON data at Δ1 = 50 m. We use this to infer that the spectral resolution of ICON grid is
    urn:x-wiley:19422466:media:jame20180:jame20180-math-0193(C4)
    Details are in the caption following the image

    The horizontal spectra of vertical velocity at z = 500 m for two different grids.

    The estimate from equation (C4) cane be generalized further, accounting for the interpolation and measurement errors, to deduce that Δ5 lies is the range of 0.8Δl − 0.9Δl, which, at the best, is a good approximation. Interestingly, the resolution Δ4 based on the degrees of freedom lies within the range of Δ5. However, more investigation is required in this direction.