Large eddy simulation using the general circulation model ICON
Abstract
ICON (ICOsahedral Nonhydrostatic) is a unified modeling system for global numerical weather prediction (NWP) and climate studies. Validation of its dynamical core against a test suite for numerical weather forecasting has been recently published by Zängl et al. (2014). In the present work, an extension of ICON is presented that enables it to perform as a large eddy simulation (LES) model. The details of the implementation of the LES turbulence scheme in ICON are explained and test cases are performed to validate it against two standard LES models. Despite the limitations that ICON inherits from being a unified modeling system, it performs well in capturing the mean flow characteristics and the turbulent statistics of two simulated flow configurations—one being a dry convective boundary layer and the other a cumulus-topped planetary boundary layer.
Key Points:
- Large eddy simulation using a general circulation model ICON is presented
- ICON is validated against standard LES models for boundary layer simulations
- Despite limitations, ICON performs well compared to the standard LES models
1 Introduction
Understanding climate change requires clear understanding of feedbacks due to clouds, which are the leading source of uncertainty in the existing climate models [Bony et al., 2004; Stevens and Bony, 2013]. The uncertainty in cloud fields in climate models is in turn due largely to the limitations of the shallow and deep convective parameterization schemes. Different modeling approaches like single column models [Betts and Miller, 1986; Randall et al., 1996; Ghan et al., 2000], cloud resolving modeling [Xu et al., 1992; Xu and Arakawa, 1992; Grabowski et al., 1996], and superparameterization [Grabowski, 2001; Khairoutdinov et al., 2005] have been used in the past to understand convective parameterization in a climate model.
With the increasing availability of computing resources, efforts have been taken to perform global simulations at even higher resolution to possibly reduce the uncertainty by explicitly resolving some of the scales involved in the convective motion. A good example of this approach is the Japanese model NICAM, which has been used in the recent years to perform global cloud resolving simulations [Miura et al., 2007; Satoh et al., 2014]. While one can argue about the deep convective scales that are resolved in a typical cloud resolving model with grid resolution ranging from 1 to 4 km, the grid resolution used in such models is definitely too coarse to resolve shallow convection [Miller, 1978; Bryan et al., 2003]. Moreover, the subgrid turbulence schemes used in the aforementioned cloud resolving models are typically not even designed to work in these resolution ranges. This motivated Wyngaard [2004] to refer to this range of scales, where convection is neither resolved nor so unresolved as to be representable in terms of its ensemble effects, as “gray zone.”
The only way to avoid this gray zone is by explicitly resolving shallow cumulus convection, which is defined by the depth of the atmospheric boundary layer, which is typically of the order of a kilometer. Thus, simulations on a
(100 m) grid largely obviate the need for special parameterization for organized turbulent motions such as those that define the atmospheric boundary layer and the areas of shallow convection. By crossing this threshold of
(100 m) grid resolution one can begin thinking of LES, wherein the subgrid-scale parameterizations have a sounder theoretical foundation and matter less [Deardorff, 1970; Moeng, 1984; Bryan et al., 2003]. This motivated us to work toward simulations at
(100 m) horizontal resolution, on a sufficiently large domain for sufficiently long time, which will help us to understand the parameterized convection better, and possibly, to improve it.
To take this forward, the German Federal Ministry of Education and Research (BMBF) has initiated a project named High Definition Clouds and Precipitation for advancing Climate Prediction (HD(CP)2), incorporating several institutes across Germany. The project targets limited-area LES at resolution
(100 m). There are other groups across the world also moving in this direction of ultra-high resolved simulation in a quasi-operational sense. Some of them, within our knowledge, are Chow et al. [2006], Moeng et al. [2007], and Hanley et al. [2013], who have modified existing regional models to do LES. We use ICON as the host model which is a new nonhydrostatic modeling system, developed in collaboration between the German Weather Service (DWD) and the Max Planck Institute for Meteorology (MPI-M). ICON is different from the models mentioned above because it is a unified modeling system suitable for global and limited-area applications on one hand and for climate prediction and weather forecasting on the other. After the inclusion of LES capabilities, ICON can now be refined to the spatial scales needed to resolve convection while interacting with the evolving large-scale atmosphere.
Admittedly, ICON is not the only unified modeling system known in literature. The Advanced Research WRF (AR-WRF) [Skamarock et al., 2008], for example, is a nonhydrostatic fully compressible modeling system which also has capabilities to perform global as well as limited-area simulations with suite of physics packages relevant for climate predictions, NWP, and LES. There is however an important distinction between AR-WRF and ICON in that ICON is an operational global NWP and climate model, which in this manuscript is shown to be flexible enough to be run as a LES model.
The purpose of this paper is to describe the new physics package (LES physics) in the ICON framework and compare its performance against well-established standard LES models for idealized boundary layer flows. We do not present ICON as an alternative to standard LES models that are designed primarily for boundary layer studies. Rather, we aim to utilize the unified nature of ICON by using its eddy-resolving abilities to better understand the processes that are parameterized in the climate or numerical weather forecast configurations of ICON. The model equations and the necessary changes made in ICON to get the new LES physics package are described in the following section. It is important to realize that ICON inherits some compromises from being a unified model that standard LES models do not. For example, while standard LES models use structured orthogonal grids which easily allow to implement higher-accuracy schemes, ICON uses an unstructured icosahedral grid, suitable for spherical geometry, which puts a restriction on the use of a higher-accuracy scheme at a reasonable computational cost. These challenges are apparent in section 3 where the conservative discretization of the three-dimensional turbulence scheme on the ICON grid and its coupling to the model dynamics are explained. The model is then validated against two standard LES models for the following boundary layer flow types: (a) dry convective boundary layer in section 4.1, and (b) cloud topped boundary layer in section 4.2. Concluding remarks are presented in section 5.
2 Model Description
ICON is a fully compressible model that uses geodesic Delaunay grids with C-type staggering, and has the ability to locally refine a region using the classical nesting approach. Much effort has been put in designing the code for high performance on massively parallel computing architectures. The model presently hosts two basic physics packages: one for weather predictions (DWD package) and the second for climate modeling applications (MPI-M package). These two packages are designed for subgrid-scale processes operative on scales of hundreds of kilometers to tens of kilometers. In order to use ICON at
(100 m) scales, a new LES physics package has been added to ICON incorporating the following modifications.
Some of the parameterizations in the DWD and MPI-M packages are invalid at
(100 m) scales and must be turned off. This pertains to the schemes for convection, subgrid-scale orographic effects (blocking and gravity wave drag) and nonorographic gravity wave drag. On the other hand, new approaches for representing the subgrid-scale turbulence and more complex microphysical process must be introduced. Therefore, a new subgrid-scale turbulence scheme based on the classical Smagorinsky scheme has been implemented. The scheme uses the modifications by Lilly [1962] to account for thermal stratification. A double-moment microphysics scheme based on Seifert and Beheng [2001] has also been implemented for the LES package. Furthermore, instead of the default diagnostic cloud fraction scheme, a simple all-or-nothing scheme [Sommeria and Deardorff, 1977] is used that assumes that the cloud fraction within a grid box is either 1 or 0. The default artificial numerical dissipation for LES studies is reduced to fourth order for the momentum equations.
Full details of the set of equations used in ICON and its numerical treatment are beyond the scope of the present work and can be found in Zängl et al. [2014] and Wan et al. [2013]. Only information relevant to the implementation of the turbulence scheme in ICON are discussed here. Information about the time integration scheme, the spatial discretization schemes, and a few other numerical details are also given in section 4.











(left) Schematic showing the primal (black triangles) and dual (red hexagons) cells, and the associated local coordinate system used in this manuscript. Unit vectors 1, 2, and 3 point in the direction of edge normal, tangent, and vertically upward (indicated by
), respectively. (right) Schematic of two adjacent triangles in ICON grid identifying the various locations used to discretize the turbulent diffusion term.
are the triangle vertices, e is the center of the edge cd about which all the strain rates are calculated, and p, q are the cell-centers.












Here Γd and Γm are the dry and the moist adiabatic lapse rate, respectively, Lv is the latent heat of vaporization, and θ is the potential temperature.


Here
and
represent the forcing from the slow-physics (i.e., radiation) and the fast-physics parameterizations (i.e., saturation adjustment, cloud microphysics, and turbulence). Note that
and
in equations 5-7 are also categorized as fast-physics. As the name suggests, the slow-physics are called less frequently compared to the fast-physics and therefore their tendencies are stored to be integrated with the governing equation. The fast-physics, on the other hand, are called every physics time step to sequentially update the prognostic variables and therefore do not provide tendencies to the governing equations. The sequential coupling between the turbulence parameterization and the dynamics is explained in Appendix Appendix A.
Equations 5-7 and equation 15, together with the slow-physics tendencies, are integrated in time using the two-time level predictor-corrector scheme except for the terms corresponding to the vertical sound-wave propagation, which are integrated implicitly. Tracers in equation 16 are integrated using a flux-form semi-Lagrangian scheme for its better conservation properties. As rather small time steps are required for the dynamics to maintain numerical stability, it is substepped several times between successive calls to the physical parameterizations. In the default configuration, physics time step is 5 times larger than the dynamics time step.






Here
is the eddy diffusivity coefficient, which is assumed to be same for
and
.
is set to 1/3.
We note that the governing equations in ICON differ from standard LES models in many ways. For example, in ICON only one of the horizontal velocity components (v1) is prognosed while the other component (v2) is diagnosed, and both the thermodynamic and the tracer equations use nonconservative variables. Standard LES models, on the other hand, typically prognose both the horizontal velocity components and use conserved (in nonprecipitating convective motions [Deardorff, 1980]) thermodynamic variables like liquid water potential temperature (θl) and total liquid water (qt). These differences are discussed in more detail in section 4 when the other two LES models are introduced.
3 Numerical Implementation
In the following, we discuss in detail the implementation of the subgrid-scale turbulence term on the ICON grid. Details on the coupling between the dynamics and the turbulence parameterization, and the time integration of the turbulent diffusion terms are given in Appendix Appendix A. Note that the use of tilde
and overbar
on filtered variables are dropped henceforth for notational simplicity. The implementation of the diffusion terms presented here assumes a flat surface that is suitable for the idealized runs discussed in this paper. The full formulation over nonflat surfaces will be discussed in a subsequent work.
The local coordinate system used for the discretization is indicated in Figure 1 by the unit vectors 1, 2, and 3. The (right) schematic shows two triangles, adc and dbc with common edge cd, where e is the center of the local axes. The (circum-) centers of the triangles are indicated by p and q. We follow the C-gridtype arrangement of Arakawa [1966]: v1 is located at the edge-center e of the full (main) vertical levels; v2 is located at the center of the dual-edge pq of the full levels, as indicated by the red line(s) in the figure, which by design coincides with e; v3 is located at the cell-center p of the half (interface) vertical levels; and temperature, density, and tracers are stored at the cell-centers of the full vertical levels. The cell-centers are also referred to as mass points in the text.

-
: a RBF reconstruction from edges to vertices.
-
: a linear interpolation from mass points or vertices to edges.
-
: a bilinear interpolation from edges to mass points.
-
: an area-weighted interpolation from mass points to vertices.
-
: a linear interpolation from main levels to interface levels (quadratic extrapolation at the surface).
-
: a linear interpolation from interface levels to main levels.
The letters next to the overbar
here indicate the location to which the interpolation is performed, except for
and
, which indicate interpolation from edge-to-vertices and cells (mass points)-to-vertices, respectively. RBF reconstruction in ICON uses the Gaussian basis function,
, with shape parameter, β, tuned for great circle distances r [Wan et al., 2013]. For the flat surfaces, the basis function uses the Cartesian distance as r, which requires changing β for better performance. For now, it has been set to twice the length of the dual-edge (pq), which gives reasonable accuracy but further investigation is required.



3.1 Subgrid Viscosity









The above set of equations give strain rates and subgrid viscosity at the edge-center on the full levels. Therefore, interpolation is required to calculate Km and Kh at the interface levels for the diffusion of v1, θ, and all the tracers. The effectiveness of the diffusive transport is found to be very sensitive to this interpolation [Stevens et al., 1999]. Some advocate the use of geometric interpolation over arithmetic interpolation [Patankar, 1980, p. 197], but the observations by Stevens et al. [1999] suggest otherwise. We also performed simulations to investigate the effect of arithmetic interpolation over geometric interpolation, and found that the latter reduces the value of Km significantly, making the model unstable. In order to further understand this sensitivity, we also calculated Km at the half level cell-centers by first interpolating |S| from edges- to the cell- centers and then vertically to the half levels, which is normally done in LES models based on regular grids. The results obtained from both these approaches for the case in section 4.2 were almost indistinguishable from each other (not shown).
3.2 Turbulent Fluxes
The location of the stress terms τ1l and τ3l (same as that of S1l and S3l) used in the calculation of turbulent fluxes are shown in Figure 2, which shows one of the triangles (acda) in Figure 1. It is advised to follow both Figures 1 and 2 to understand the discretization explained below. Also, note that the location of a variable, if required, is indicated by |() which should not be confused by the letters used as superscript indicating interpolation scheme. Detailed derivation of some of the turbulent fluxes is given in Appendix Appendix B for reference.

Schematic showing the locations at which the strain rates Sij are defined for (left) horizontal and (right) vertical turbulent diffusion. The full and half vertical levels are indicated by k and
, respectively, and the triangles at the half levels are marked by dashed line. Naming convention same as that of Figure 1 is used. The direction vectors are also indicated in the middle.


Here
that appears in equation 9, which is essentially one-third of the total divergence. All the terms on the RHS of the above equation are integrated in time using Euler explicit except for the term
contributing to the vertical diffusion, which is integrated using Euler implicit (see Appendix Appendix A for details).









Here ac is the area of the triangle, fo is the orientation factor (±1) indicating the orientation of the triangle edge [Wan et al., 2013]. The first term on the RHS of equation 32 is the horizontal diffusion which is obtained using the divergence theorem by summing the fluxes across the three edges of the triangle. Similar to the momentum equations, the first term on the RHS of equation 32 is integrated explicitly in time whereas the second term is integrated implicitly.
It is clear that the discretization of the turbulent diffusion terms is flux-conservative. Meeting this conservation requirement on a triangular grid, while maintaining the overall computational performance, is not trivial. This is primarily because of the several interpolations involved which are computationally expensive, and result in loss of accuracy, and depending on the type of interpolation function, smoothening of the interpolant. For some asymptotic descriptions this effectively implies an A-grid implementation [Gassmann and Herzog, 2008]. Despite this overhead, the preliminary study revealed that the turbulence scheme in ICON takes about 20% of the overall computational time, which is comparable to the performance of the similar turbulence scheme implemented in a structured grid in one of the standard LES model (UCLA-LES) used here.
3.3 Surface Boundary Condition








3.4 Lateral Boundary Condition
Doubly periodic boundary conditions are required for the simulations performed in the present work. This is achieved by treating the grid as a pseudo 2-D torus so that the boundaries in the respective coordinate directions are joined to each other. In order to incorporate this geometry in ICON, modifications have been made in the calculation of the interpolation coefficients and the discrete operators so that it uses Cartesian coordinates instead of the default spherical coordinates. Option is also available to use open lateral boundary conditions using data from the European Center for Medium-Range Weather Forecast (ECMWF), Consortium for Small-Scale Modeling (COSMO) model, and the ICON model itself for limited-area simulations.
4 Benchmarking ICON-LES
Within the framework of HD(CP)2, two simulations are performed to benchmark ICON as a model capable of doing large eddy simulations: the dry convective boundary layer (DCBL) and the cloud topped boundary layer (CTBL). Both simulations are performed in a doubly periodic domain with prescribed surface fluxes and zero flux at the top. The model configuration of ICON capable of performing LES is termed as ICON-LES (hereafter simply ICON). The setups used here have been studied extensively in the literature, both numerically and experimentally (see e.g., Deardorff [1972]; Willis and Deardorff [1974]; Moeng [1984]; Wyngaard [1985]; Schmidt and Schumann [1989] and the references therein for DCBL, and Nitta and Esbensen [1974]; Sommeria [1976]; Nicholls and LeMone [1980]; Siebesma and Cuijpers [1995]; Stevens [2007], and the references therein for CTBL). For this reason and because they test the representation of dry convective turbulence and its coupling to the moist processes, they are good candidates for benchmarking LES models simulating atmospheric boundary layer turbulence.
- ICON uses the fully compressible set of equations whereas UCLA uses the anelastic approximation [Ogura and Phillips, 1962] and PALM uses the Bousinessq approximation [Dutton and Fichtl, 1969]. Therefore, ICON is forced to use a smaller time step allowing for the sound-wave propagation. For the simulations performed here, the dynamical time step in ICON is about 0.02 times that of PALM and UCLA.
- ICON solves the edge-normal velocity component, therefore it has three degrees of freedom in each grid cell whereas it is four in the UCLA and PALM.
- ICON uses θρ as the prognostic variable for the thermodynamic equation, whereas UCLA and PALM use θl as the prognostic variable.
- ICON uses qv and ql as the prognostic variables to represent moist processes, whereas UCLA and PALM use qt that is a conserved quantity in the absence of precipitation.
- The advection scheme for momentum equations in ICON is second-order accurate in both vertical and horizontal directions. In addition, ICON uses fourth-order artificial numerical dissipation for numerical stability in the momentum equations. In UCLA and PALM, the momentum advection schemes are fourth-order central and fifth-order upwind, respectively.
- The advection scheme for the thermodynamic equation in ICON uses the second-order upwind [Miura, 2007] and central scheme for flux reconstruction in horizontal and vertical directions, respectively. In addition, a Smagorinsky type second-order numerical dissipation is applied on temperature fields for stability reasons. In UCLA and PALM, these are second-order (with flux limiter) and fifth-order upwind, respectively.
- The tracer advection scheme in ICON uses the second-order upwind scheme by Miura [2007] in horizontal (there is an option for third-order upwind also) and third-order piecewise parabolic method by Colella and Woodward [1984] in vertical direction with flux limiter. In UCLA and PALM, these are second-order (with flux limiter) and fifth-order upwind, respectively.
- ICON uses a second-order Predictor-Corrector time integration [Zängl et al., 2014] scheme whereas UCLA and PALM both use a third-order Runge-Kutta scheme.
- ICON and UCLA use the classical Smagorinsky turbulence scheme whereas PALM uses the turbulent kinetic energy based scheme of Deardorff [1980].
In addition to the artificial numerical dissipation, a three-dimensional divergence damping of fourth-order is used in ICON to damp the acoustic waves. The damping coefficient is set to 0.0025Δt. It is advisable for LES simulations to minimize the use of dissipation so that it does not interfere with the subgrid diffusion. However, the use of the artificial dissipation cannot be avoided for simulations with real orography, therefore, we decided to switch it on to see its effect on the resolved scales in the benchmark cases.
The points mentioned above clearly show the advantages that standard LES models (like UCLA and PALM) have over the general purpose models like ICON, which must be kept in mind while analyzing the results presented in the subsequent sections. A summary of the formal accuracy of the (advection) schemes used in these models is provided in Table 1.
Model | Momentum (Horizontal, Vertical) | Thermodynamics (Horizontal, Vertical) | Tracer (Horizontal, Vertical) |
---|---|---|---|
ICON | (2,2) | (2,2) | (2,3) |
UCLA | (4,4) | (2,2) | (2,2) |
PALM | (5,5) | (5,5) | (5,5) |
4.1 Dry Convective Boundary Layer
4.1.1 Simulation Design
The dry convective boundary layer simulation is initiated with a potential temperature profile constantly increasing with height from a surface temperature of θs = 290 K at a constant lapse rate of Γ = 0.006 Km−1. The initial wind is set to zero and the turbulence is triggered by adding random perturbations to the temperature field up to a height of 300 m in ICON and up to 1600 m in UCLA and PALM. The boundary layer then develops in time because of the fixed kinematic surface heat flux
K ms−1. Note that the overbar
henceforth denotes spatial average over horizontal slabs of the computational domain and the prime
is used to indicate the deviation from this slab average. Kinematic units are used to force the models with the same surface fluxes irrespective of the way density is handled by them individually.


This definition ensures the same number of grid cells in both meshes. However, the spectral analysis of the simulation data revealed that the actual grid resolution is lower than the present estimation. This is discussed further in Appendix Appendix C. It is also worth noting that the same number of grid cells in both triangular and rectangular meshes implies less degrees of freedom in ICON, which has three velocity components per grid cell, compared to the rectangular mesh, which has four velocity components per grid cell.
4.1.2 Time Evolution
We start by analyzing the time evolution of the horizontal mean of boundary layer height as simulated by the three models in Figure 3. It is calculated as the height (zi) at which the vertical gradient of θ is at maximum. Different model resolutions are indicated in each figure. The jumps in zi during the first 30–75 min (progressively less for higher resolutions) indicate the spin-up phase during which turbulence is first establishing itself. The difference in UCLA and PALM as compared to ICON is because of differences in initialization (i.e., the height of initially prescribed random noise). ICON shows large oscillations for Δ = 100 m while keeping the (temporal) mean lower than the other LES models. One clear feature that is seen in these results, which has been noted previously, e.g., by Sullivan and Patton [2011], is that the entrainment rate (dzi/dt) decreases in all the models as the grid spacing Δ is reduced. The effect of the larger entrainment rate at coarse resolution is seen as enhanced warming in Figure 4.

Time evolution of boundary layer height at the indicated resolutions.

Vertical profile of mean potential temperature at the indicated resolutions. The dashed line is the initial profile.
4.1.3 Vertical Structure
Vertical profiles of θ, averaged spatially over the horizontal domain and temporally over the last 15 min of the simulation (sampled every 30 s), are shown in Figure 4. For Δ = 100 m, ICON captures the gross features, like the superadiabatic layer near the surface and the interfacial layer, just like standard LES models. The profile changes appreciably as the resolution is increased and ICON remains trustful to what are by now familiar patterns, e.g., as shown by the UCLA and PALM. UCLA entrains more (see Figure 6 near zi), the effect of which is more heating in the mixed layer and a deeper interfacial layer. At Δ = 12.5 m, the θ profile in PALM and UCLA (not shown) does not change much indicating a nearly converged solution at Δ = 25 m. This can be also realized in Figure 5 where the convergence of horizontally averaged θ at the boundary layer height (calculated at the end of simulation) with increasing resolution is shown. From the figure it is clear that all the models approach convergence at the same rate with ICON following PALM more closely.

Mean potential temperature at the boundary layer height with increasing resolution.

Mean vertical heat flux profile at the indicated resolutions. The solid lines are the total and the dashed lines are the subgrid-scale part of the total flux. The zero value is indicated by dashed line.


At Δ = 100 m, PALM has a small (positive) heat flux immediately above the inversion which is not seen for the other models. This is an artifact indicative of insufficient numerical damping in the PALM model. A similar pattern was found in Moeng [1984] who used a central difference scheme in the vertical. The subgrid-scale flux in ICON is almost the same as in UCLA because they use the same turbulence scheme, whereas it is slightly smaller in PALM, at all resolutions. As expected, subgrid-scale fluxes of ICON (and other models) decrease, hence the resolved scale fluxes increase, as the resolution is increased. In general, PALM shows the highest turbulent flux in the surface layer. At the inversion,
in ICON and PALM are about 10% of the surface value (at all resolutions), whereas it is about 18% in UCLA. These values are within the expected range of 10–30% [Stull, 1988, p. 478]. The deeper and (negative) stronger inversion in UCLA explains its larger cooling rate in the inversion layer in Figure 4 [Garcia and Mellado, 2014].
Previous model intercomparison studies have shown that significant spread exists between models for vertical velocity variance (
) [Siebesma et al., 2003; Stevens et al., 2001]. In order to quantitatively assess
, we have used the DNS results of Garcia and Mellado [2014] for reference in Figure 7. Although the subgrid contribution to
is missing in the LES models (because of its unavailability from ICON), we have noted from UCLA and PALM that its contribution is fairly small to affect the discussions below. In order to dimensionalize the DNS results, zi = 700 m and a convective velocity scale
ms−1 from ICON results at Δ = 25 m are used.

Mean vertical velocity variance profile (resolved) at the indicated resolutions together with a reference DNS profile.





Results in this section suggest that ICON performs satisfactorily in comparison to UCLA and PALM. For Δ = 100 m, ICON captures the salient features of the boundary layer better than the models run at a few km resolution without parameterized boundary layers (e.g., cloud resolving models). ICON also shows grid convergence which is necessary to ensure the consistency of the discrete system. We focused on the vertical heat flux to tune the value of the Smagorinsky constant, and
was found to give the best results. The model became unstable for
. We also performed simulations using
, instead of equation 11, to get reduced values of
and
near the surface. The fluxes did improve near the surface but we also saw wiggles near the surface in the mean profile of temperature, so we decided to stay with equation 11.
4.1.4 Spatial Structure
It is also instructive to look at the spatial structure of the flow field for better assessment of the model. The most commonly known spatial feature in convective boundary layers is the presence of a spoke-like pattern near the surface [Mason, 1989; Schmidt and Schumann, 1989]. In order to find out whether ICON can resolve such a pattern on its triangular grid, contours of the fluctuations of the virtual potential temperature and vertical velocity in a horizontal cross section are plotted in Figure 8 for Δ = 25 m. The figures clearly show those spoke-like patterns in both
and
contours. By visually inspecting the figure, we find that the typical length of the segments forming this pattern is about 1.3zi, similar to the value reported in Schmidt and Schumann [1989]. As expected,
and
are strongly correlated as required by the sign of vertical heat flux.

Contour plots of (left)
(K) and (right)
(ms−1) in a horizontal plane at z = 137 m drawn at the end of the simulation for Δ = 25 m.
4.2 Cloud Topped Boundary Layer
4.2.1 Simulation Design

The factor 0.0088 is roughly 0.74 times the saturation specific humidity at the surface. This profile corresponds to an exponentially decreasing relative humidity, which in turn ensures an initial equivalent potential temperature profile decreasing with height which is necessary for the growth of a conditionally unstable cloud layer [Stevens, 2007]. The initial temperature and humidity profiles are shown in Figures 10a and 10b as dotted lines. The initial wind is set to zero as in the DCBL case.









Results in the previous section clearly show convergence of ICON as the grid resolution is increased. The intermediate resolution of Δ = 50 m was found to be in good agreement with Δ = 25 m. Similar conclusion was drawn for the CTBL case with a base simulation using Δ = 50 m and another simulation with double the number of points in the vertical. It is for this reason that we only use the results from the base simulation in this section. The domain size is kept the same as in the dry case with doubly periodic boundary conditions and the solution is integrated for 25 h.
4.2.2 Time Evolution
Figure 9a shows the time evolution of the boundary layer height as simulated by the three models. As noted earlier in Figure 3, UCLA and PALM start with a big jump due to the random initialization. Before the cloudy layer develops, the evolution of zi in all the models is consistent with the DCBL case at Δ = 50 m in the sense that ICON has lowest zi and PALM has the highest. UCLA and PALM match well until t = 20 h after which they depart slowly. ICON simulates a shallower boundary layer (by 150–200 m). The rate of boundary layer growth,
, in ICON (and in both other models) increases due to the developing cloud layer after
12 h. This is consistent with earlier findings [Stevens, 2007] that
varies as t1∕2 when the boundary layer is dry, and as t when the cloud layer develops.

Time evolution of the indicated quantities for the cloud topped boundary layer simulation. The black solid line in Figure 9b is the mean over all the three models used in the present paper, and the spread of 50% [from Siebesma et al., 2003] about this ensemble mean is indicated by the bars. The magenta lines in Figures 9c and 9d are from ICON after doubling the number of vertical levels.

Clouds are triggered at t
5 h in ICON and PALM, and at t
10 h in UCLA as seen in Figure 9b. All models maintain a near constant cloud fraction after t = 15 h. ICON saturates at a cloud fraction of 0.2, PALM at 0.13, and UCLA at 0.1. In order to ensure that the spread in cloud fraction lies within the known limits of uncertainty due to different model configurations, an uncertainty range is added in the figure corresponding to the spread found in the LES intercomparison study of cumulus convection as observed during the Barbados Oceanographic and Meteorological Experiment (BOMEX) by Siebesma et al. [2003]. The range shows a spread of 50% about the ensemble mean which is marked as a thin solid line. It is clear that all the models stay within this range despite all the differences.
The surface temperatures in all the models match quite well, except for some initial oscillations in ICON (see Figure 9e). These oscillations are more apparent in the surface fluxes in Figures 9d and 9e for t < 10 h. At first we thought that the oscillations are due to the fact that ICON has lesser degree of freedom than UCLA and PALM, therefore we did simulations with higher horizontal resolution so that the degree of freedom in ICON becomes equal to the other models (see discussions in Appendix Appendix C) and also with doubled vertical resolution (Δ3 = 25 m). Simulation with higher horizontal resolution had minimal effect (not shown). Simulation with doubled vertical resolution, as indicated by ICON–HI in Figures 9c and 9d, show that the oscillations are greatly reduced in ICON suggesting that the oscillations are mainly due to grid-locking of the inversion height to model levels.
Besides these initial oscillations, the surface fluxes in the models evolve in the same fashion. ICON simulates higher
(by 5 W m−2) throughout the simulation and smaller
(by 20 W m−2) after the formation of the clouds, thereby maintaining a constant value of the buoyancy flux.
4.2.3 Vertical Structure
In this section we look at the vertical structure of the thermodynamic variables and the turbulent fluxes in ICON. The vertical profiles are obtained by averaging spatially in the horizontal direction and temporally over the last 30 min (sampled every 30 s). The liquid water potential temperature (
) profile in Figure 10a shows that ICON is about 0.2 K cooler compared to UCLA in the subcloud layer. Similar behavior was seen in the DCBL case. We believe that this additional cooling in ICON is due to the fact that a fraction of the heat transported from the surface is used in the expansion work in ICON, thereby generating mean vertical wind (see Figure 10d), which does not take place in the other two models because of the Bousinessq and anelastic assumptions. Furthermore, the subcloud layer top in ICON is slightly less than 700 m where UCLA and PALM converge. This implies a shallower cloud base in ICON which can also be seen in the ql profile in Figure 10c. The specific humidity profile from ICON lies exactly between that of PALM and UCLA as seen in Figure 10b.
As previously mentioned, ICON prognoses both ql and qv whereas PALM and UCLA only prognose qt. qc is then diagnosed using the condensation scheme of Sommeria and Deardorff [1977]. Despite this difference, the mean cloud liquid water in ICON stays within the range noted earlier by Siebesma et al. [2003], as indicated by the bars in Figure 10c. The reason for a lower cloud base in ICON can be understood by realizing that the air near the surface in ICON is moister than in the other models (by 0.2g kg−1) whereas the temperature is a little lower (by 0.2 K). Therefore, an air parcel ascending from the surface saturates much faster in ICON in comparison to PALM and UCLA. Furthermore, the slightly lower cloud top in ICON suggests enhanced mixing with the dry atmosphere above, in comparison to other models. This extra mixing near the cloud top in ICON is either due to the implicit diffusion in the tracer advection scheme or the flux limiter which gets more active near the cloud boundary in ICON because of the use of nonconservative tracers.




Turbulent flux (resolved) profiles of the indicated quantities in the cloud topped boundary layer simulation. The zero value is indicated by black dashed line.
Figure 11a also shows a small kink at the cloud base in ICON. This is probably due to the use of the nonconservative thermodynamic and moisture variables in ICON which creates a discontinuity at the cloud boundary. This effect, although very small in the present case, can generate significant errors in cases with strong convection.
The turbulent buoyancy flux in the subcloud layer of a CTBL is known to follow the DCBL case in a nondimensional sense [Stevens, 2007]. The maximum resolved buoyancy flux in DCBL case for Δ = 50 m was found to be around 84% of the surface flux prescribed in all models. Looking at the similar maxima in Figure 11b (near the surface) it appears that ICON resolves slightly less, around 76% of the prescribed value (
= 25 W m−2) whereas UCLA and PALM resolve nearly 84%. However, as was mentioned during the discussion of Figure 6, this is simply a diagnostic error which arises due to the vertical averaging of the fluxes in ICON from main levels to the interface levels for plotting purposes. Also, Figure 11b shows two (positive) peaks as one would expect in a CTBL. The flux decreases linearly in the subcloud layer up to the cloud base where it resolves about 10% (negative) of
in all models.
The resolved vertical flux of specific humidity in ICON is found to be smaller than in the other two models (see Figure 11c). In the subcloud layer it is a consequence of its lower surface moisture flux, whereas in the cloud layer it is probably due to the enhanced mixing near the cloud top in ICON. The liquid water vertical flux (
) in ICON compares very well with other models (see Figure 11d) despite the aforementioned differences in the way cloud liquid water is handled.
Vertical profiles of resolved vertical velocity variance by the three models are shown in Figure 12. Similar to the dry case in Figure 7 for Δ = 50 m,
in ICON and UCLA are nearly the same till the lowest peak at
. At the peak and beyond that, the variance in ICON is slightly smaller than in UCLA in the subcloud layer, which is different from the dry case. In the cloud layer and above, the differences between the two is reduced again.

Vertical velocity (resolved) variance in the cloud-topped boundary layer simulation.
Overall, the results of the CTBL simulation show that ICON follows the expected trend and agrees well with the standard LES models with some differences in and around the cloud layer. We believe that such differences are bound to show up because of the inherent differences in the model and experimental designs. Some of the differences in both DCBL and CTBL cases are also because of the artificial initial profiles of temperature and moisture that produces very thin boundary layer in the early phase of the simulation, which cannot be resolved with the prescribed grid resolution. While it is hard to argue how long and by how much this affects the simulation, we think that in the presented simulations such effects are relatively smaller after the first hour of the simulation. To confirm this, we also performed the BOMEX simulation [Siebesma et al., 2003] that does not suffer from similar initialization issues. The results from this case do not differ in any important way from those from the simulation of the CTBL case, and are therefore not presented here.
As already mentioned, several interpolations are required in ICON for spatial discretization which can lead to (numerical) errors and smoothening of the results. On top of that, unlike standard LES models that use high-order schemes for spatial discretization, the triangular grid in ICON poses a challenge on the use of higher-order schemes in the horizontal. It is, however, to be noted that the overall (spectral) accuracy of the models using such higher-order schemes decreases as the complexity of the experiment increases. For complex cases, the use of artificial second-order numerical dissipation becomes necessary to keep the model stable, which reduces the order of the leading truncation error term of the spatial discretization scheme to second order, thereby reducing the overall accuracy of the spatial discretization to first order [Ghosal, 1996].
5 Conclusion
This paper describes the extension of a unified modeling system for climate and weather forecast ICON (ICOsahedral Nonhydrostic) to a large-eddy simulation framework and its comparison against two well-established LES models.
The first part details the implementation of the turbulence scheme on the triangular grid used in ICON and the numerical technique used to integrate the diffusion equations. Conservative implementation of a three-dimensional turbulence scheme in a triangular grid is nontrivial and involves several interpolation operations. Despite these computational overheads, the turbulence scheme in ICON-LES takes less than 20% of the overall computational time in case of a dry convective boundary layer, which is comparable to the performance of the similar turbulence scheme implemented in the quadrilateral grid of UCLA-LES.
- it was shown in section 4.1 that ICON-LES approaches toward a converged solution as the grid resolution is increased.
- the profiles of resolved vertical fluxes for both DCBL and CTBL indicate that the eddy-resolving capabilities of ICON-LES is comparable to that of the standard LES models.
- the lower cloud top in ICON-LES for the CTBL case suggests enhanced (numerical) mixing near the cloud top which could have been activated due to the use of nonconservative variables in ICON which creates a discontinuity at the cloud boundary.
Acknowledgments
The authors thank the German Federal Ministry of Education and Research (BMBF) for their funding to support the HD(CP)2 project (01LK1202E). We also thank Jade Garcia for providing us with the DNS data, the whole ICON development team at DWD and MPI-M for their excellent work, and the anonymous reviewers for their constructive suggestions. We specially thank Leonidas Linardakis for his continuous help during the course of the model development. The data used in this paper are available at the German Climate Computing Center and can be obtained from the corresponding author upon request.
Appendix A: Coupling of Dynamics and Turbulence





The RHS of equation (A1) is composed of the first two terms in equation 30, representing the horizontal diffusion, all evaluated using variables from dynamics. This equation is integrated in time using Euler explicit and the tendency is stored in
. Vertical diffusion is then performed using equation (A2) where all the terms in the RHS are also evaluated using the
variables, except for the term involving the vertical derivative of v1, which is evaluated implicitly using Euler implicit method. This is indicated in equation (A2) by the use of an intermediate time step
. The tendency from this equation is then combined with the horizontal tendency in the final equation to get the velocities at
.



Here again, equation (A4) is integrated explicitly using the variables from dynamics and the tendency is stored in
, and equation (A5) is integrated implicitly storing the tendency in
. It is to be noted that Km and Kh are always evaluated using the variables from dynamics.
Appendix B: Discretization of Turbulent Fluxes
Here we derive some of the turbulent fluxes on ICON grid for the interested readers.


At this point, if we make the assumption that
and
, and that v2 at nodes
are also arithmetic averages of their neighboring vertices (e.g.,
), after some manipulation we end up with the second term of the RHS of equation 30. The third term on the RHS of equation 30 is a straightforward expansion of
utilizing the strain rates located at
and
in Figure 2.
Rest of the terms can be derived in a similar manner and are therefore not derived here.
Appendix C: Grid Resolution in ICON
It is mentioned at the outset that the results in this section are relevant for flat meshes only.






In order to see how the estimation from equation (C1) compares with a regularly spaced grid model, we have compared the horizontal spectra of vertical velocity from ICON to that of PALM and UCLA for the DCBL case. These one-dimensional spectra are obtained by Fourier transforming the field along a horizontal line and averaging over all parallel lines at a fixed height and time. ICON spectrum is obtained by interpolating its velocity field to a regular grid with the same number of grid points as the standard LES models using distance weighted averaging technique. That is, the Δ1 = 50 m simulation results are interpolated to 192 × 192 grid points and the Δ1 = 25 m to 384 × 384 grid points. The corresponding spectra at z = 500 m are shown in Figure 13.

The horizontal spectra of the vertical velocity at indicated resolutions in log-log scale. The data have been multiplied by some factors in order to avoid overlap. Also shown is the expected Kolmogorov's spectrum (
k
) (black line), for
50, 25 m.




The horizontal spectra of vertical velocity at z = 500 m for two different grids.
The estimate from equation (C4) cane be generalized further, accounting for the interpolation and measurement errors, to deduce that Δ5 lies is the range of 0.8Δl − 0.9Δl, which, at the best, is a good approximation. Interestingly, the resolution Δ4 based on the degrees of freedom lies within the range of Δ5. However, more investigation is required in this direction.