A comparative copula‐based bivariate frequency analysis of observed and simulated storm events: A case study on Bartlett‐Lewis modeled rainfall

Abstract

[1] Because of a lack of historical rainfall time series of considerable length, one often has to rely on simulated rainfall time series, e.g., in the design of hydraulic structures. One way to simulate such time series is by means of stochastic point process rainfall models, such as the Bartlett‐Lewis type of model. For the evaluation of model performance, with a focus on the reproduction of extreme rainfall events, often a univariate extreme value analysis is performed. Recently developed concepts in statistical hydrology now offer other means of evaluating the overall performance of such models. In this study, a copula‐based frequency analysis of storms is proposed as a tool to evaluate differences between the return periods of several types of observed and modeled storms. First, this study performs an analysis of several storm variables, which indicates a problem with the modeling of the temporal structure of rainfall by the models. Thereafter, the bivariate frequency analysis of storms, defined by their duration and volume, illustrates the underestimation and overestimation of the return period of the storms simulated by the models, which is partially explained by a large difference in the marginal distribution functions of the storm duration and storm volume, the difference in the degree of association between the latter, and a different mean storm interarrival time. The proposed methodology allows for the identification of some problems with the rainfall simulations from which recommendations for possible improvements to rainfall models can be made.

1. Introduction

[2] In many hydrological applications, such as urban drainage design or the design of hydraulic structures, there is often a high demand for extensive rainfall time series at different timescales. In reality, however, one often lacks rainfall time series of desirable length or with an appropriate timescale. For this reason a variety of rainfall models has been developed throughout the years. One type are the stochastic point process models, for generating rainfall time series at a specific location or in space [Onof et al., 2000; Cowpertwait, 1995]. A lot of research has been done on the improvement of stochastic point process models, and a wide range of models can be found in literature [Cowpertwait et al., 2007; Burton et al., 2008]. Although more complex models exist, the focus of this paper will be on two older Bartlett‐Lewis type models: the modified Bartlett‐Lewis (MBL) model [Rodriguez‐Iturbe et al., 1988] and the modified Bartlett‐Lewis gamma (MBLG) model [Onof and Wheater, 1994], which are based on the original Bartlett‐Lewis (OBL) model developed by Rodriguez‐Iturbe et al. [1987a].

[3] To evaluate the performance of these stochastic rainfall models, one often analyzes the reproduction of extreme rainfall through an extreme value or frequency analysis. Traditionally, univariate methods such as the annual maximum (AM) method or peak‐over‐threshold (POT) method are used to obtain intensity‐duration‐frequency (IDF) curves. Often, an underestimation of the extremes by the models is observed [Onof et al., 2000; Verhoest et al., 1997; Velghe et al., 1994]. These analyses have steered the development in point process modeling of rainfall in the direction of an incorporation of the third‐order moment of rainfall in the fitting procedure in order to better reproduce extremes [see, e.g., Cowpertwait, 1998; Cowpertwait et al., 2002; Cowpertwait, 2004; Cowpertwait et al., 2007; Burton et al., 2008].

[4] Instead of these univariate methods, multivariate methods for the analysis of extremes could provide a more powerful tool for assessing the performance of rainfall models. Recently, a copula‐based frequency analysis has been developed, permitting a more flexible and multivariate analysis of extremes [Salvadori et al., 2007; Salvadori, 2004; Salvadori and De Michele, 2004]. Copulas are in fact multivariate distribution functions with uniform marginals on the unit interval, which allow the description of the dependence structure between random variables independently of their marginal behavior. Pioneering work in copula‐based bivariate frequency analysis are the studies of De Michele and Salvadori [2003] and Favre et al. [2004], followed by several other contributions [e.g., De Michele et al., 2005; Genest and Favre, 2007; Kao and Govindaraju, 2007; Zhang and Singh, 2007a]. Also, a trivariate frequency analysis has been proposed [Genest et al., 2007; Serinaldi and Grimaldi, 2007; Zhang and Singh, 2007b; Grimaldi and Serinaldi, 2006a], in which the issue of mixed association, i.e., both positive and negative or different pairwise associations among the variables involved, is of great importance for the choice of the copula construction method.

[5] In this study a bivariate copula‐based frequency analysis is presented as a tool to evaluate the performance of stochastic rainfall models. This approach, however, differs in several ways from the univariate extreme value analysis. In the latter, often a generalized extreme value (GEV) distribution is used to model the maxima of long and finite sequences of random variables. For a generalization of the GEV distribution to more than one dimension, extreme value theory and copula theory show that multivariate extreme value distributions can be composed of univariate GEV distributions and of an extreme value copula (see, e.g., Salvadori et al. [2007] for a detailed theoretical discussion of multivariate extreme value distributions). A bivariate extreme value distribution could be of interest when an association between the maxima of two random variables is suspected [see, e.g., Tawn, 1998; Genest and Segers, 2009, and references therein]. In this study, the focus is on a complete characterization of the joint distribution function of two storm variables, instead of on the joint distribution function of their extremes. As such, when referring to extreme events this should not be seen in the context of extreme value theory.

[6] An event‐based approach is conducted, which means that independent storms are selected out of rainfall time series. For each storm, different rainfall variables are calculated and a frequency analysis is performed on the basis of the copula‐based joint distribution function of storm duration and storm volume. A detailed comparison is performed using, on the one hand, a real‐world time series of 27 years of 10 min rainfall data and, on the other hand, two simulated time series of 27 years of 10 min rainfall data. This allows for a detailed analysis of the reproduction of storm‐based rainfall characteristics, through which the shortcomings of the rainfall models can be revealed. Recommendations for relevant model improvement are then made.

[7] The paper is structured as follows. Section 2 starts with a description of the time series used, the models considered, and the methodology for deriving the storms, which then form the basis for the further analysis. Section 3 explains how a bivariate distribution function of the storm duration and storm volume can be built by using a copula that describes the dependence structure between them. Once the distribution function is defined, the frequency analysis is conducted in section 4. An attempt is made to explain the observed differences in the return period of several bivariate (extreme) events, in a physically meaningful way. Finally, section 5 concludes this study, bringing up some recommendations for further research in copula‐based stochastic rainfall modeling.

2. The Data

2.1. The 10 min Rainfall Time Series

[8] Several time series are used throughout this study. On the one hand, a 27 year long 10 min rainfall time series from Uccle is used. Rainfall observations with a time resolution of 10 min, from 1 January 1967 to 31 December 1993, were measured by a Hellmann‐Fuess pluviograph in the climatological park of the Royal Meteorological Institute at Uccle, near Brussels, Belgium [Demaree, 2003]. This time series of Uccle, albeit with varying record length, has already been the subject of several studies [Verhoest et al., 1997; Vaes et al., 2002; De Jongh et al., 2006; Ntegeka and Willems, 2008; Vandenberghe et al., 2010a]. On the other hand, synthetic time series of 27 years of 10 min rainfall are used to compare with the Uccle data. Two types of rectangular pulses models, more specifically the modified Bartlett‐Lewis (MBL) model and the modified Bartlett‐Lewis gamma (MBLG) model, were used for the generation of the synthetic time series [Verhoest et al., 1997].

[9] The MBL model is an adapted version of the OBL model [Rodriguez‐Iturbe et al., 1987a], built to better reproduce adequate zero‐depth probabilities (ZDPs) [Rodriguez‐Iturbe et al., 1987b, 1988]. In the OBL model, model storm arrivals occur randomly according to a Poisson distribution with parameter λ. Each of these model storm origins is linked to another Poisson process of rate β of cell origins. During a given time, which is exponentially distributed with parameter γ, rectangular cells are generated within each model storm. The height of each cell, expressing the rainfall intensity of that cell, follows an exponential distribution with parameter 1/μ_x. The duration of the cells is exponentially distributed with parameter 1/η, which is the mean cell duration. Rodriguez‐Iturbe et al. [1987a] then introduced two dimensionless parameters κ = β/η and ϕ = γ/η. The number of cells C per model storm follows a geometric distribution with mean μ_c = 1 +κ/ϕ, as it is assumed that a cell is located at each model storm origin. Thus, the OBL is a five‐parameter (λ, β, γ, μ_x, η) model, for which mathematical properties, such as the analytical expressions for the first‐ and second‐order moments, are given by Rodriguez‐Iturbe et al. [1987a].

[10] The adjustment of the OBL model then consists in allowing for a randomly varying mean cell duration, 1/η. The parameter η now follows a two‐parameter gamma distribution with shape parameter α and scale parameter ν. Also the mean cell interarrival time and the mean storm duration time, namely, β⁻¹ and γ⁻¹, vary randomly, however, in such a way that κ and ϕ remain constant. This results in the six‐parameter (λ, κ, ϕ, μ_x, α, ν) MBL model. Mathematical details and empirical evidence of better model performance can be found in studies by Rodriguez‐Iturbe et al. [1988], Islam et al. [1990], Onof and Wheater [1993], Velghe et al. [1994], and Verhoest et al. [1997].

[11] Because of a poor reproduction of extreme values in the rainfall series by the MBL model, Onof and Wheater [1994] developed the MBLG model, in which the cell intensity follows a two‐parameter gamma distribution, with shape parameter p and scale parameter δ, resulting in the seven‐parameter (λ, κ, ϕ, α, ν, p, δ) MBLG model. More mathematical details are given by Onof and Wheater [1994]. The work of Cameron et al. [2000], among others, provides an evaluation of the MBLG model.

[12] Verhoest et al. [1997] generated continuous series of 100 years of 10 min rainfall depths using the MBL and MBLG models of which a 27 year subset will be used in this study. They used the method of moments to calibrate the model for each month separately, given a 27 year long 10 min rainfall time series at Uccle between 1967 and 1993, which was assumed to be stationary per month. It should be noted that this calibration period corresponds with the data set used in current study. The difference between the observed and the modeled mean, variance, covariance, and zero‐depth probability at different aggregation levels is thus minimized. A 10 min aggregation level for all moments and also a 24 h aggregation level for the covariance and zero depth probability was considered for the MBL model. For the MBLG model fitting, the 24 h variance was also included.

[13] Table 1 shows detailed information on the correspondence between observed, simulated, and analytical statistics, considering the month of July for both the MBL and MBLG model, where the analytical statistics refer to the statistics obtained by substituting the estimated parameters in the analytical expressions for the model [Onof and Wheater, 1994, 1993]. Both the absolute values of the statistics and their relative differences are given. The 24 h mean and variance are also given, although they were not used in the calibration. It is clear that quite large positive and negative relative differences (expressed as percentages) occur. To put these relative errors more in perspective, an analysis based on an ensemble generation was performed. For a set of 1000 simulated time series with a length of 27 years, considering both the MBL and MBLG models, the statistics can be calculated on a monthly basis. So for a specific model, month, aggregation level and statistic, a distribution of the values of this statistic is obtained, which represents the sampling variability of the model. The empirical cumulative distribution function can then be compared with the values of the observed, simulated, and analytical statistics. When these statistics are within the range of the ensemble, it indicates a good performance of the model. The analysis learns (no details shown in the paper) that although relative errors could be quite large, the performance of the model is good in almost all cases. For example, the observed 24 h variance is 60.74% larger than the one simulated by the MBL model (see Table 1). Nevertheless, they are both within the range of the ensemble.

Table 1. Overview of Observed, Analytical, and Simulated Statistics for 1/6 and 24 h Aggregation Levels and Their Relative Differences for the Month of July, Considering Both the MBL and MBLG Models^aa MBL, modified Bartlett‐Lewis; MBLG, modified Bartlett‐Lewis gamma; ZDP, zero‐depth probability; obs., observed; anal., analytical; sim., simulated.

	Absolute Values
	Mean		Variance		Autocovariance		ZDP
	1/6 h (mm)	24 h (mm)	1/6 h (mm^2)	24 h (mm^2)	1/6 h (mm^2)	24 h (mm^2)	1/6 h	24 h
Observed	0.0178	2.5683	0.0324	31.4968	0.0177	5.3889	0.9498	0.4576
MBL anal.	0.0161	2.3189	0.0242	19.5951	0.0127	4.7447	0.9677	0.5127
MBLG anal.	0.0161	2.3192	0.0249	23.6747	0.0130	4.7492	0.9763	0.5123
MBL sim.	0.0167	2.4074	0.0110	16.5637	0.0062	3.6909	0.9415	0.4170
MBLG sim.	0.0195	2.8086	0.0416	54.2762	0.0258	2.5338	0.9759	0.4970
	Relative Differences (%)
	Mean		Variance		Autocovariance		ZDP
	1/6 h	24 h	1/6 h	24 h	1/6 h	24 h	1/6 h	24 h
MBL
1 − obs./anal.	−10.76	−10.76	−33.61	−60.74	−39.61	−13.58	1.85	10.75
1 − sim./obs.	6.27	6.27	66.00	47.41	64.93	31.51	0.88	8.88
1 − sim./anal.	−3.82	−3.82	54.58	15.47	51.04	22.21	2.72	18.67
MBLG
1 − obs./anal.	−10.74	−10.74	−30.23	−33.04	−36.08	−13.47	2.71	10.67
1 − sim./obs.	−9.35	−9.35	−28.59	−72.32	−45.64	52.98	−2.74	−8.62
1 − sim./anal.	−21.10	−21.10	−67.47	−129.26	−98.18	46.65	0.05	2.97

• a MBL, modified Bartlett‐Lewis; MBLG, modified Bartlett‐Lewis gamma; ZDP, zero‐depth probability; obs., observed; anal., analytical; sim., simulated.

[14] Of course, the relative errors between observed and analytical statistics do indicate that there is some room for improvement of the model fitting. A new calibration of the model, with the newest optimization techniques and with other objective functions, could result in an even better model. However, the effect of an improved calibration on the better representation of storm characteristics is quite ambiguous, as it is not obvious to establish a clear link between aggregated rainfall statistics and variables at a storm level. Nevertheless, such links could be very useful in future calibration studies and will be the subject of further research.

[15] Finally, it should be noted that the selected rainfall models do not take into account the nonstationarity of the rainfall process. A Kwiatkowski‐Phillips‐Schmidt‐Shin (KPSS) test [Kwiatkowski et al., 1992] for stationarity indicates that the hypothesis of stationary daily rainfall amounts in Uccle (105 years) cannot be rejected at a significance level of 1% (p value is 0.03875). This test, however, assumes normality of the rainfall aggregates, which is not valid here. A nonparametric Mann‐Kendall test [Van Gelder et al., 2007] shows that no trend is present in the daily rainfall amounts at a significance level of 5% (p value is 0.05025).

2.2. Storms

[16] The further analysis in this study will make use of individual storms, which are selected out of the 27 year Uccle and 27 year MBL and MBLG 10 min rainfall time series. Therefore, storms are selected and postprocessed in an identical way as carried out by Vandenberghe et al. [2010a]. They used a 24 h dry spell in between storms as independence criterion, on the basis of the work by Restrepo‐Posada and Eagleson [1982]. However, one could consider using another independence criterion dictated by the application in which the storms are to be used, e.g., the concentration time of the catchment under study [Vandenberghe et al., 2010b]. In this paper, the 24 h criterion was applied for both observed and simulated time series, although Verhoest et al. [1997] found that the independence criterion for the simulated time series is of the order of 3–10 h. We thus propose that when applying the methodology as presented in the paper, simulated storms (or another type of events) should be separated on the basis of the criterion obtained by analysis of the observed time series. If the models were perfect, no difference in independence criterion would exist. When applying different independence criteria to the observed and simulated time series, it would no longer be clear what is compared to what and how the results should be interpreted.

[17] In order to circumvent the problem of abundant ties in the data (equal values associated with the resolution of the recording instrument), a small noise was introduced to the storm variables, as proposed by Vandenberghe et al. [2010a], rather than applying a threshold on the data as done for example by De Michele and Salvadori [2003]. Eventually, for each storm a number of features are calculated: the storm volume V, the storm duration W, the dry spell after a storm D, the mean storm intensity I, the maximum 10 min storm intensity I_max, the timing of the maximum storm intensity t_max, the fraction of dry spells within a storm p_d, and the number n_c of showers in a storm, which are separated by dry spells shorter than 24 h. To comply with the assumption of stationarity, only the summer storms of the months June, July, and August are further considered [Vandenberghe et al., 2010a].

[18] In this way, 562 observed Uccle (observed), 793 MBL, and 707 MBLG summer storms are selected. Yearly on average, there are 20.8 observed, 29.4 MBL, and 26.2 MBLG summer storms, which indicates that the models generate more summer storms than what is observed. It should be noted that the concept of storm, as defined here, differs from a model storm as described in section 2.1. In fact, one selected storm can consist of several model storms. In what follows, the term “storm” indicates a selected storm unless otherwise stated.

[19] Table 2 gives an overview of some important statistics for different storm characteristics: the minimum, the maximum, the mean (m), the median, the standard deviation (s), the skewness (g₁), and the kurtosis (g₂). Figure 1 visualizes this information. The results indicate that the MBL and MBLG models generate considerably more and shorter storms, with consequently on average fewer showers during one storm and a lower mean rainfall volume per storm. Synthetic storms are also separated by a dry period that is on average shorter than for the observed storms. The models thus tend to underestimate the dry period between two successive events with the considered configuration of parameters. However, the maxima in rainfall volume are higher for the MBL and MBLG storms, whereas the maximum storm durations are a lot shorter compared to the observed storms. The mean intensity of the MBL storms is on average almost equal to the mean intensity of the Uccle storms, but considerably larger for the MBLG storms. Also, the maxima of the mean storm intensity follow the same trend. This could indicate that the fitted Gamma distribution of the cell intensity in the MBLG model increases the mean storm intensity. It could also be attributed to the generation of too many cells, for which the summed intensities result in a too high mean storm intensity. The skewness and kurtosis indicate that in all cases there is a strong deviation from normality. Note that the existence of these higher‐order moments is not always guaranteed, which should be considered when interpreting them. The presence of several outliers and asymmetrical boxes in the box plots also illustrates this strong deviation from normality.

Figure 1

Visualization of the information contained in Table 2 by means of box plots. A large deviation from normality for all variables of observed (O), modified Bartlett‐Lewis (MBL), and modified Bartlett‐Lewis gamma (MBLG) storms is present.

Table 2. Statistics of Storm Characteristics for Observed, MBL, and MBLG Storms^aa O, observed; m, mean; s, standard deviation; g₁, skewness; g₂, kurtosis.

		V (mm)	W (h)	I (mm/h)	D (h)	Imax (mm/h)	tmax (h)	pd	nc
Minimum	O	0.00	0.15	0.01	24.15	0.01	0.15	0.00	1
	MBL	0.01	0.15	0.02	24.16	0.07	0.15	0.00	1
	MBLG	0.02	0.15	0.02	24.16	0.14	0.15	0.00	1
Maximum	O	99.36	360.35	11.19	528.82	92.33	130.51	0.98	133
	MBL	185.01	229.16	10.03	289.85	37.16	202.66	0.98	89
	MBLG	99.72	151.65	17.52	414.83	46.22	100.32	0.99	44
m	O	9.95	30.21	0.61	76.76	9.94	13.33	0.65	10.81
	MBL	6.88	21.15	0.70	54.66	4.83	11.05	0.58	8.38
	MBLG	7.81	16.16	1.50	65.81	10.83	7.67	0.63	5.30
Median	O	4.33	16.50	0.32	46.33	5.31	2.34	0.79	6
	MBL	3.54	10.98	0.44	45.34	3.94	2.34	0.68	6
	MBLG	4.08	8.50	0.69	50.17	9.14	1.68	0.82	4
s	O	13.81	41.24	1.06	74.20	13.00	21.27	0.32	14.82
	MBL	12.25	27.95	0.95	30.84	4.06	20.02	0.32	9.17
	MBLG	10.47	21.24	2.32	47.35	8.37	12.83	0.36	5.11
g1	O	2.46	2.69	5.62	2.58	2.62	2.35	−1.06	4.16
	MBL	7.91	2.72	4.82	2.09	2.82	3.80	−0.63	2.96
	MBLG	3.38	2.35	3.54	2.52	1.30	2.72	−0.86	2.52
g2	O	10.83	13.83	43.67	10.90	11.69	9.17	2.73	28.72
	MBL	95.90	14.32	34.05	10.11	15.84	24.08	2.05	17.26
	MBLG	20.06	10.73	18.32	12.00	4.98	12.40	2.09	12.99

• a O, observed; m, mean; s, standard deviation; g₁, skewness; g₂, kurtosis.

[20] Because the models generate more and shorter storms, being separated by shorter dry periods, compared to the observed storms, one may deduce that too severe a clustering is induced by the model structure. A more detailed look at the differences in the distribution functions of the fraction of dry spells p_d for simulated and observed storms (Figure 2) also points in this direction. The probability of p_d ≤ 0.6 is always larger for simulated storms. The probability of 0.8 ≤ p_d ≤ 0.9 is around 22% and 20% for MBL and MBLG storms, respectively, but 28% for observed storms. For the MBL storms and specifically for the MBLG storms, a lot more storms are found without a dry spell, compared to what is observed.

Figure 2

The differences in the cumulative distribution functions of the fraction of dry spells within a storm p_d for simulated (MBL and MBLG) and observed (O) storms support the constatation of problems with the modeling of the temporal structure of storms.

3. Constructing a Bivariate Distribution Function

3.1. Choosing the Pair of Random Variables

[21] Several couples of random variables can be used to characterize a storm in order to perform a bivariate frequency analysis. The most important random variables characterizing the general storm structure (not the internal storm structure) are probably the duration W, the mean storm intensity I, and the volume V. Because of practical considerations we will focus here on the couple (W, V) for the further analysis. The other couples (I, W) and (I, V) show some degree of asymmetry, which makes the fitting of a copula more difficult [Vandenberghe et al., 2010a]. Moreover, in the context of tail dependence and interpretation of extreme events, the couple (W, V) makes more sense, as is discussed further.

3.2. Bivariate Copulas

[22] In order to perform a copula‐based frequency analysis, a bivariate distribution function of the storm duration W and the storm volume V needs to be constructed. Therefore, a bivariate copula will be used to model the dependence structure between W and V, independently of their marginal distribution functions. For all mathematical details, the reader is referred to Nelsen [2006] and Salvadori et al. [2007].

[23] To build the bivariate distribution function F_WV(w, v) of W and V, the theorem of Sklar [Sklar, 1959] is of utmost importance:

The first part of equation (1) states that a bivariate copula C couples the marginal distribution functions of W and V, i.e., F_W (w) and F_V (v), resulting in their bivariate distribution function. The second part of equation (1) is based on the invariance property of copulas, where the following transformation is made:

where U and Z are uniformly distributed on the unit interval = [0,1].

[24] In order to obtain (u_i, z_i) for each couple (w_i, v_i) in the data set, both theoretical or empirical cumulative distribution functions of W and V can be used. The latter nonparametric approach is preferred here:

with n the number of data points and R_i and S_i the ranks of w_i and v_i among w₁,…, w_n and v₁,…, v_n.

[25] A copula is in fact a multivariate distribution function with uniform marginals, allowing for the calculation of joint probabilities without the knowledge of marginal distribution functions.

[26] Vandenberghe et al. [2010a] provide an in‐depth overview of the different steps necessary for the fitting of bivariate copulas to the dependence structure between storm characteristics. Of all copula families considered, including asymmetrical copula families, the A12 copula family was evaluated as the best and most flexible to model the observed positive dependence between W and V and will therefore be used here for the frequency analysis. The A12 copula family is given by

This Archimedean family could also be expressed in terms of the additive generator ϕ:

where

The copula parameter θ is estimated using the relation between θ and Kendall's tau τ_K, a nonparametric association measure that equals 1 or −1 in the case of complete positive or negative association, respectively [Nelsen, 2006]:

For the A12 family, θ ∈ [1, +∞] and consequently τ_K ∈ [, 1].

[27] Table 3 gives an overview of Kendall's tau and corresponding parameter values for the three different data sets as well as some goodness of fit (GOF) measures, such as the root‐mean‐square error (RMSE), S_n, and T_n (more details can be found in the studies by, e.g., Genest and Favre [2007] and Genest et al. [2009]). In the latter, S_n and T_n as used here are denoted by S_n^(k) and T_n^(k). These three measures are based on the difference between the empirical and fitted copula and should thus be as small as possible. The fitted copulas clearly provide a good model for the dependence between W and V at a significance level of 5% considering the p values of S_n and T_n. As stated by Genest et al. [2009], the S_n statistic provides a powerful statistic in the evaluation of the GOF of Archimedean copulas, such as the A12 family.

Table 3. Kendall's tau τ_K and the Estimated A12 Copula Parameter θ With 95% Confidence Bounds and Corresponding Goodness of Fit Measures With Their p Values for the Observed, MBL, and MBLG Storms^aa LB, lower bound; UB, upper bound. RMSE, root‐mean‐square error.

	Observed	MBL	MBLG
τK	0.6313	0.6177	0.5797
θ	1.8082	1.7438	1.5860
LB	1.7001	1.6380	1.4837
UB	1.9163	1.8496	1.6883
RMSE	0.0059	0.0063	0.0051
Sn	0.0759	0.0335	0.0480
p value, Sn	0.0856	0.6550	0.4022
Tn	0.6263	0.5334	0.6458
p value, Tn	0.2890	0.5997	0.2905

• a LB, lower bound; UB, upper bound. RMSE, root‐mean‐square error.

[28] From Table 3, it can also be concluded that the positive association between W and V for the synthetically generated storms is slightly different. This, of course, results in a difference in the copula parameter values. With respect to the latter, it should be noted that for other pairs of storm variables even larger relative differences between the observed and simulated association are present. This might point to some problems with the Bartlett‐Lewis simulation of rainfall storms with the considered configuration of parameters.

3.3. Marginal Distribution Functions

[29] In order to be able to perform the transformations from ² to ² as defined in equation (2), the marginal cumulative distribution functions of W and V, i.e., F_W and F_V, need to be defined. Both commonly used parametric models, such as the generalized Pareto (GP), exponential and Gamma distribution functions, and a nonparametric model are considered to obtain the CDF of W and V with the best goodness of fit in the data range. The kernel smoothing function, available in the distribution fitting toolbox of Matlab, is used as the nonparametric model. In Appendix A, details can be found on the parametric models and goodness of fit measures used.

[30] Table 4 lists the values of the Anderson‐Darling A_n and Kolmogorov K_n statistics for the three parametric and one nonparametric (kernel) distribution functions fitted to the storm duration W and the volume V of the observed, MBL, and MBLG storms.

Table 4. Values of the A_n and K_n Test Statistics for the Generalized Pareto, Exponential, Gamma, and Nonparametric (Kernel) Distribution Function Fitted to the Storm Duration W and the Storm Volume V of the Observed, MBL, and MBLG Storms^aa The best fits are provided by the nonparametric models (kernel). GP, generalized Pareto; Exp, exponential.

	Observed		MBL		MBLG
	W	V	W	V	W	V
An
GP	23.497	16.732	23.675	0.900	30.228	0.944
Exp	66.377	57.666	65.716	21.835	84.311	13.058
Gamma	2.776	1.282	4.782	3.999	7.339	2.1623
Kernel	0.608	0.047	0.324	0.089	0.783	0.073
Kn
GP	0.132	0.109	0.114	0.031	0.129	0.033
Exp	0.192	0.193	0.186	0.111	0.220	0.102
Gamma	0.058	0.046	0.065	0.054	0.082	0.048
Kernel	0.031	0.012	0.017	0.013	0.034	0.013

• a The best fits are provided by the nonparametric models (kernel). GP, generalized Pareto; Exp, exponential.

[31] Both the storm duration W and the volume V are best modeled by the kernel function. Moreover, the fit of the parametric distribution functions is significantly different from the underlying distribution, as p values calculated with a bootstrap method are (very close to) zero. The same bootstrap method for the nonparametric method results in p values that indicate an appropriate fit at a significance level of 5%. As we are not interested in extrapolations outside the data range, we consider the nonparametric distribution functions for W and V to be of most practical interest in the further analysis. Furthermore, the use of one specific way of modeling all marginal distribution functions allows for a consistency throughout the comparison analysis.

[32] Figure 3 illustrates that storms with a short duration W are more likely in the case of MBLG and MBL storms compared to the observed storms. The observed storms are more likely to have a larger volume compared to the MBL and MBLG storms. To focus on the fit in the upper tail, a probability plot (for a normal distribution) is given in Figure 4, which shows a very good agreement of the fit with the observations.

Figure 3

A good agreement of the empirical cumulative distribution function (ecdf) and the fitted cumulative nonparametric distribution function for (a) W and (b) V per set of storms (observed, MBL, or MBLG) is obtained, but the CDF greatly differs between simulated and observed storms.

Figure 4

Probability plot showing the good fit of the cumulative nonparametric distribution function to the extremes of both (a) the storm duration W and (b) volume V of observed (Uccle) and simulated (MBL and MBLG) storms.

4. Frequency Analysis

[33] In the frequency analysis, the copula‐based bivariate distribution function will be used. First, some types of (extreme) events will be defined for which different return periods can be calculated (section 4.1). Thereafter, differences in the copula‐based frequency analysis results between observed, MBL, and MBLG storms will be analyzed.

4.1. Extreme Events and Return Periods

[34] Recent literature has provided a nice theoretical framework for frequency analysis of rainfall based on copulas [Salvadori et al., 2007], in which the concepts of primary and secondary return periods arise. These will be summarized here.

4.1.1. Primary Return Periods

[35] To conduct a bivariate frequency analysis, first, different (extreme) storm types are defined in terms of their duration W and storm volume V with thresholds w and v, respectively:

[36] From a hydrological point of view, these events are interesting as they cover, on the one hand, extreme rainfall events with a long duration and a high volume (and thus a moderate mean intensity), which saturate the watershed and cause destructive events at watershed scale, and, on the other hand, those events with a short duration and a high volume (and thus a high mean intensity), which cause flash floods.

[37] It is now possible to formulate an expression for the return periods T of these specific events (given in subscript), which are called primary return periods [Salvadori et al., 2007]:

[38] In these definitions, ω_T is the mean value of the interarrival time of the storms in the data set. Therefore, for each storm the dry duration D after a storm and the wet duration W are summed and this sum W + D is then averaged. For the observed, MBL, and MBLG storms, ω_T is 106.97 h (0.0122 years), 75.81 h (0.0087 years), and 81.97 h (0.0094 years), respectively. It is clear that the average storm interarrival time is shorter for the MBL and MBLG storms than for the observed storms. This is, of course, strongly related to the findings in sections 2.2 and 3, i.e., more and shorter synthetic storms, separated by shorter dry periods compared to the observed storms. The parameter ω_T is fairly closely related to the parameter λ in the MBL and MBLG models, which is calibrated at 0.0180 and 0.0222 h⁻¹, respectively, in July. Thus, in the respective models, a model storm origin will occur on average every 55.6 and 45.0 h during the month of July, which forms a part of the summer storms. Taking into account the relations between model parameters and storm variables during the calibration process could improve the results. For example, if the dry period in between storms were incorporated in the calibration process, than the simulated mean storm interarrival time would improve and the marginal distributions of storms characteristics will alter. This will be the subject of further research.

[39] It should be noted that several inequalities between these return periods exist for any given copula:

[40] The storm type with the larger return period is thus always more extreme than the one with the smaller return period. Equation (19) has been stated several times in literature in the context of a danger for underdimensioning and overdimensioning of structures [Yue and Rasmussen, 2002; Shiau, 2003; Salvadori and De Michele, 2004]. Equation (20) can be understood by analysis of equations (15) and (17). It always holds that for a given u and z:

as u ≥ C_UZ (u, z). Equation (21) can also be explained mathematically since

This means that the difference between the latter two return periods only depends on the marginal distribution function of the storm duration W. Since u and z, being marginal probability levels, correspond to certain quantiles of W and V, it is clear that one should be very aware of which storm type (the AND case, OR case, etc.) is considered in the frequency analysis and which return period one will use for engineering purposes. The extremity of a storm in this context is defined, on the one hand, by the storm type (one of the five cases) and, on the other hand, by the choice of the thresholds for the storm duration and the storm volume. It could be of great interest to further investigate the mutual relations between these return periods, together with the effect of different copulas and the influence of tail dependence, to properly evaluate the possible dangers of underdimensioning and overdimensioning.

[41] The expressions listed in the denominators of equations (14)–(18) are in fact probabilities of occurrence of the events listed in equations (9)–(13), in terms of the copula C_UZ. It should be clear that no marginal distributions need to be known for the calculation of these return periods, as they are expressed in terms of u and z. To find combinations of u( = F_W (w)) and z ( = F_V (v)) for which the return period is the same, one should find those u and z for which the denominator is constant. For Archimedean copulas, such as A12 copulas, one can easily obtain analytical expressions for curves combining isofrequent events (storms having the same return period). Salvadori and De Michele [2004], Salvadori [2004], and Salvadori et al. [2007] provide extensive mathematical and practical details on those curves for the AND, OR, and COND₁ cases. In the OR case, for example, one should find the curves for which C_UZ is constant, as can be understood from equation (15). Those level curves on which the copula has the same value, i.e., C_UZ = t can easily be constructed. For different t values one can calculate the corresponding return period T_OR, or by transforming the expression in equation (15) one can define the t values for different return periods, which is often more useful in practice. In the case of the Uccle storms, for a return period of 2 years, the corresponding t value is

[42] This means that when considering a return period of 2 years, which is in traditional analysis relatively small, the region of interest is already in the upper right tail of the copula (or, equivalently, the bivariate cumulative distribution function). This is explained by the way in which storms are selected, i.e., two successive storms are separated with a minimal dry period of 24 h resulting in a relatively small storm interarrival time. Therefore, storms occur quite frequently during the year. The timescale of traditionally used return periods (up to 100 years) is already a thousand times larger than the timescale of the investigated rainfall process. Thus, searching for a storm with a relatively small return period happens on a much larger temporal scale than the scale of the data set. In other words, a storm that only occurs every 100 years, when on average every, e.g., 3 days (given by ω_T) a storm occurs, should be very extreme. This aspect was already pointed out by Salvadori and De Michele [2004], and one should be aware of this in the further frequency analysis.

[43] It is also obvious that when the copula is not fitted well in the upper right tail, the results of the frequency analysis will be unreliable: It is therefore of major importance to properly model the dependence of extremes, or tail dependence [Salvadori et al., 2007; Poulin et al., 2007; Frahm et al., 2005]. The A12 copula family with parameter θ is able to model tail dependence, and the upper right and lower left tail dependence coefficients, λ_UR and λ_LL, respectively, are given as

[44] These tail dependence coefficients thus give the degree of association between the extremes in the bivariate distribution function: λ_UR considers the upper right tail and λ_LL the lower left tail. We estimate the tail dependence coefficients empirically using equation (26), with C_n the empirical copula and with limits t = 0.92 for λ_UR and t = 0.08 for λ_LL:

[45] Table 5 gives the theoretical and empirically estimated upper right and lower left tail dependence coefficients. The presence of tail dependence in the data is more or less taken into account by the fitted copulas. However, the method by which the empirical estimates are obtained could be improved, though the results obtained justify the choice of the A12 copula family for the modeling of the dependence between W and V.

Table 5. Similarities Between Lower Left (λ_LL) and Upper Right (λ_UR) Tail Dependence Coefficients of the Fitted A12 Copulas and Empirical Estimates^aa Note the similar magnitude. O, observed.

	λLL			λUR
	O	MBL	MBLG	O	MBL	MBLG
Theoretical	0.68	0.67	0.65	0.53	0.51	0.45
Empirical	0.67	0.65	0.51	0.55	0.55	0.55

• a Note the similar magnitude. O, observed.

4.1.2. Secondary Return Period

[46] Storms can alternatively be classified in subcritical, critical, and supercritical storms. In this context, the concept of a secondary return period has emerged as being conceptually more useful for design purposes [Salvadori and De Michele, 2004; Salvadori, 2004; Salvadori et al., 2007; Vandenberghe et al., 2010b]. In engineering applications one usually chooses a design storm with a certain (primary) return period for which the design should hold. Consider now the storms in the OR case. By fixing a certain design return period T*_OR, a certain level t* of the copula is fixed. In Figure 5 this is indicated with the thick level curve (corresponding with t = 0.4). A storm that lies on this curve has a return period T_OR* and is called a critical storm. In Figure 5, S* is such a critical storm and is defined by the critical thresholds u* and z*. A more extreme storm with a higher return period, and thus on a higher level curve, is then called a supercritical storm, e.g., S₁⁺ and S₂⁺. On the other hand, the storms S₁⁻ and S₂⁻ have lower return periods and are then called subcritical storms. The secondary return period is now defined as the average time between the occurrence of two supercritical storms and is expressed as follows:

Figure 5

(a) Illustration of subcritical, supercritical, and critical storms. (b) Difference in probability used for secondary and primary return period (dark gray area in Figure 5a versus Figure 5b). After Salvadori et al. [2007].

[47] The function K_C is the distribution function of the random variable Y = C(U, Z), or in other words {Y ≤ y} = K_C(y). For Archimedean copulas this function can easily be calculated:

Here ϕ′(t⁺) is the right derivative of the additive generator ϕ. For the A12 copula family this function becomes

[48] As with the calculation of the primary return periods (see equations (14)–(18)) the denominator in equation (27) expresses a probability. In this light _C (t*) can be interpreted as the probability that a supercritical storm will occur at any realization of a storm [Salvadori et al., 2007]. This probability mass is given in dark gray in Figure 5a.

[49] In first instance it might be difficult to intuitively feel the difference between a primary and a secondary return period [Salvadori et al., 2007]. In the calculation of the primary return period in the OR case, one uses the probability that the storm duration W or/and the storm volume V will exceed a respective threshold w and v. This probability can be expressed as 1 − C_UZ(u, z) (see equation (15)). However, this probability is not the same as the probability of the occurrence of a supercritical storm that is used for the calculation of the secondary return period, i.e., 1 − K_C(t). Figure 5a gives the probability mass of 1 − C(u*, z*) in dark gray. The primary return period T_OR will then be the average time between the occurrence of two successive storms in this region, which is defined by the critical thresholds (u*, z*). It is obvious that the probability mass in Figure 5a is smaller than the probability mass in Figure 5b, for the same critical thresholds. It should also be clear that for smaller t* values, these differences between probabilities and hence between return periods are smaller. In practice, it is more compelling to use this probability of supercritical events, which are in fact dangerous for the design, and hence the secondary return period is a more realistic concept.

4.2. Comparison of Return Periods

4.2.1. Differences Between Primary Return Periods of Uccle and Synthetic Storms

[50] Instead of a univariate extreme value analysis, which is often used to evaluate the adequate modeling of rainfall extremes, this study provides an alternative analysis of the observed, MBL, and MBLG storms for the specific storm events as defined in equations (9)–(13). First, the differences in the calculated return periods (see equations (14)–(18)) can be considered for specific combinations of thresholds w and v, with a maximum of 150 h and 80 mm, respectively. These maxima are within the range of the observed and simulated storms, which means that no extrapolations in terms of extreme values are made. This, of course, differs from the conventional univariate extreme value analysis, which only focuses on the distribution of the extremes in rainfall depth for specific aggregation levels.

[51] Figure 6 plots the behavior of the relative differences in return periods, i.e., the differences between the observed and simulated return period normalized by the observed return period, considering the MBL storms. Positive and negative differences can be interpreted as an underestimation and overestimation of the extremes by the models, respectively. These differences indicate a more or less steady underestimation of the extremes by the MBL model of around 30% in the AND, OR, and COND₁ cases. Also note that the relative differences are identical for both the AND and the COND₁ cases, as a consequence of equation (22). The relative differences for the COND₂ and the MAR cases are somewhat larger, but also indicate an underestimation of the extremes for all combinations of w and v, which is most severe for a large v and a short w.

Figure 6

The relative discrepancy (dimensionless) of the return periods for observed and MBL storms shows underestimations of the (extreme) storms by the MBL model.

[52] The same analysis can now be conducted considering the differences in Uccle and MBLG storms. Figure 7 shows that for the AND, OR, and COND₁ cases the relative differences indicate underestimations of the extremity of the MBLG storms in the range from 14% to 26%, which are smaller compared to the MBL storms. By contrast, the relative differences for the COND₂ and the MAR cases are larger than for the MBL storms and reach values up to 80%.

Figure 7

The relative discrepancy (dimensionless) of the primary return periods for observed and MBLG storms shows underestimations of the (extreme) storms by the MBLG model.

4.2.2. Differences in Secondary Return Periods

[53] Similarly as for the analysis of the differences in primary return periods between observed and simulated storms, we now analyze the differences in the secondary return period. Figure 8 shows the relative differences between the secondary return periods of observed and simulated storms (MBL in Figure 8 (top), and MBLG in Figure 8 (bottom)). Both MBL and MBLG storms have a smaller secondary return period, compared to observed storms. The smaller relative differences, considering the MBLG model, of which the smallest values are located in the region of the most extreme storms (Figure 8, top right), indicate a better performance of this model compared to the MBL model.

Figure 8

Relative differences in the secondary return period for observed and simulated storms indicate an underestimation of the interarrival time of supercritical storms by the models.

[54] Further, the relation between the primary return period in the OR case and the secondary return period is examined. Figure 9 shows the behavior of the survival probability function _C(t*) 1 − K_C(t*) and the secondary return period ρ(t*) as a function of the primary return period T_OR, in a double logarithmic scale considering the Uccle, MBL, and MBLG storms. The starting point of both types of curves corresponds with T_OR ≈ ω_T for t* ≈ 0. For K_C ≈ 0 it holds that t* ≈ 0 and consequently the curve of _C(t*) starts around one. For higher values of t* and thus higher primary return periods, the function _C(t*) will decrease, i.e., the probability of a supercritical storm decreases. This corresponds to a reduction of the dark gray area in Figure 5a: The probability that a randomly chosen couple (u, z) is in this region becomes smaller and smaller when the critical return period increases. Because of this decreasing probability of a supercritical storm, the secondary return period will increase. In fact, the expected value of the time between the occurrence of two supercritical storms increases. The secondary return period will always be larger than the primary return period.

Figure 9

Survival probability function _C(t*) = 1 − K_C(t*) and the secondary return period ρ(t*) as a function of the primary return period T_OR in a double logarithmic scale considering the observed (O), MBL, and MBLG storms.

[55] For a given T_OR, the probabilities of occurrence of a supercritical storm are the smallest and largest for the MBLG and Uccle storms, respectively, and the secondary return period is largest for MBLG storms, but almost equal for Uccle and MBL storms. Of course, the difference in ω_T and in the association between W and V will influence these interactions, as ω_T is used in the calculation of both the primary and the secondary return period and the difference in association influences the K_C function.

4.2.3. Identifying the Causes of the Differences

[56] The reason for the observed differences in the primary return periods, as presented in section 4.2.1, can be found by considering equations (14)–(18). From these equations, it can be noted that three different aspects may influence the results: (1) The copula C, or the degree of association between W and V, (2) the mean storm interarrival time ω_T, and (3) the marginal distribution functions F_W and F_V. The same factors also influence the differences in the secondary return period (section 4.2.2). Of course, the shortcomings of the MBL and MBLG models as an approximation of reality explain most of these differences in the representation of extreme behavior.

[57] To evaluate the effect of the difference in the degree of association between W and V, as expressed by different A12 copula parameters for the observed, MBL, and MBLG storms (see also Table 3), the differences in the probabilities of occurrence between the observed and simulated storms are considered. The probabilities are given by the denominators of equations (14)–(18) in the case of the primary return periods and by the denominators of equation (27) in the case of the secondary return period. Figure 10 demonstrates these relative differences between the observed and the MBL storms. The observed shapes of the contours are also found in the case of MBLG storms. However, differences are larger in absolute value (not shown). These differences and similarities point out a nonnegligible effect on the frequency analysis of the difference in dependence between W and V for the observed and synthetic storms. However, Figure 10 cannot explain the overestimations and underestimations of the extremes as given by the relative differences in Figures 6 and 8. The marginal distribution functions, however, do affect the absolute differences between return periods.

Figure 10

Relative differences (dimensionless) between the probabilities (observed versus MBL) used for calculating the different types of primary return periods and the secondary return period.

[58] To check the influence of a smaller mean storm interarrival time ω_T in the case of the MBL and MBLG storms, together with the difference in dependence, but independently of the marginal distributions, differences in return periods can be calculated in terms of U and Z. Figure 11 shows relative differences between the observed and the MBL storms. Again, similar contours are found for the MBLG storms, with comparable differences. The agreement of Figure 11 with Figures 6 and 8 indicates that the differences in the marginal distribution functions do not affect the relative differences between the return periods of observed and simulated storms, in the considered domain of W and V.

Figure 11

Relative differences (dimensionless) in primary and secondary return periods (observed versus MBL), considering U and Z instead of W and V.

[59] The above mentioned results indicate that both the difference in dependence and the difference in the mean storm interarrival time have a nonnegligible effect on the relative differences in the frequency analysis results considering observed and simulated rainfall time series. The differences in the marginal distribution functions seem to affect absolute differences only, while relative differences are more or less invariant.

5. Conclusion

[60] In this paper a new way of evaluating the reproduction of (extreme) storms by stochastic rainfall models was provided, as an alternative to traditional methods. A copula‐based bivariate frequency analysis was considered as a useful tool to evaluate the performance of rainfall models, with a case study focussing on two models of the Bartlett‐Lewis type, i.e., the MBL and MBLG models. Therefore, storms were considered which were selected out of a 27 year long 10 min rainfall time series at Uccle, on the one hand, and two 27 year long 10 min synthetically generated rainfall time series, on the other hand. It should be noted that we proposed to use a storm selection criterion, i.e., a 24 h dry period in between two storms, which is based on the analysis of the observed time series, and then to apply it as well on the simulated time series for reasons of consistency. The choice of a selection criterion could also be imposed by the practical application, e.g., the concentration time of a drainage system.

[61] First of all, when comparing observed and simulated storms, large differences in both the marginal distribution functions of different storm variables, and the pairwise dependences between these variables were observed, pointing at some deficiencies of the simulated storms, which could be related to a problem of too severe clustering of rainfall by the models. Because of the storm‐based approach of the analysis, it would be very interesting to incorporate links between model parameters and storm properties in the calibration process, which could improve some of the deficiencies identified during the copula analysis. This will be the subject of future research.

[62] Second, a frequency analysis was performed by considering copula‐based primary and secondary return periods of storms, expressed in terms of their duration W and volume V which were modeled by nonparametric marginal distribution functions. The fitted A12 copula provided a good fit and was able to model the tail dependence, and its hydrological relevance was stressed. In the future, also the use of nonparametric copulas could be considered. When considering relative differences, in primary return periods for specific thresholds for the considered storm events, underestimations and overestimations were present, but those were not always consistent considering the MBL and MBLG storms. The relative differences indicated that it is not obvious to choose the best model, as each model can perform best, depending on which storm type is considered. The three influencing factors we considered, i.e., the difference in dependence, the mean storm interarrival time, and the marginal distribution functions, were interrelated and all contributed to some extent to the absolute differences between observed and simulated return periods, which could in some cases be considerably large. However, for the considered domain of (W, V), the marginal distributions did not affect (much) the relative differences in return periods.

[63] The copula‐based methodology in this paper was thus able to identify key shortcomings of the models, and could certainly be used to evaluate other rainfall simulations of more advanced rainfall models. It should be noted that an evaluation of the same model at different sites would allow for an even more objective identification of these shortcomings. This knowledge offers opportunities for future research in copula‐based rainfall modeling. Copulas have already shown their advantages in hydrology, also resulting in new directions in rainfall modeling. Recently, Evin and Favre [2008] embedded copulas in a Neymann‐Scott type point process model to incorporate the dependence between the depth and duration of rainfall cells which constitute a storm event. Such research is, of course, very useful, especially from a theoretical point of view, but there is a lack of knowledge on the physical effect on the overall rainfall process, as it is not very clear in which way this modeled dependence at cell level alters the dependence between the duration and mean intensity of the total storm [Evin, 2008]. In this study it is shown that there exists already a negative dependence between the storm duration and mean storm intensity, although no dependence whatsoever is explicitly embedded in the stochastic rainfall models. In that respect, implementing copulas in stochastic rainfall models could be beneficial in the modeling of the temporal dependence of the rainfall process, including the internal storm structure. Some useful contributions in this direction are provided by Salvadori and De Michele [2006], Grimaldi and Serinaldi [2006b], Kao and Govindaraju [2008], Lennartsson et al. [2008], Sun [2008], Bárdossy and Pegram [2009], and Serinaldi [2009] and will be the subject of future research.

Acknowledgments

[67] The critical and useful comments of six anonymous reviewers greatly improved this work, for which the authors are very grateful. The first author is a doctoral research fellow of the Research Foundation Flanders (FWO). The authors would also like to thank the Royal Meteorological Institute (RMI) for providing the extensive data set of Uccle rainfall.

Appendix A:: Marginal Distribution Functions

[64] The expressions for the parametric distribution functions used in the current paper are given in equations (A1)–(A4). Generalized Pareto

ϱ ∈ [ −∞, +∞], ɛ ∈ [0, +∞] and ϕ ∈ [ −∞, +∞] are the location, scale, and shape parameters, respectively. The GP distribution converges to the exponential one when ϕ → 0. It is heavy tailed (upper bounded) when ϕ > 0 (ϕ < 0) and only moments of the order of less than 1/ϕ exist. Exponential

Gamma

Here ɛ > 0 and ϕ > 0 are the scale and shape parameters, respectively. The incomplete Gamma functions γ and Γ are defined as

[65] These functions are fitted using maximum likelihood estimation and the goodness of fit is considered using two different test statistics: the Anderson‐Darling A_n and Kolmogorov K_n test statistics, which are calculated as follows [Stephens, 1974]:

where n is the number of data points and U_(i) is the ith order statistic of U₁,…, U_n, with, e.g., U₁ = F_X (X₁). In other words, the sample X₁,…, X_n is transformed to a sample U₁,…, U_n on the unit interval by the marginal distribution function F_X. Subsequently, this sample is sorted in ascending order, generating a new sample of observations U₍₁₎ ≤ U₍₂₎ ≤ … ≤ U_(n) which are called the order statistics [Salvadori et al., 2007].

[66] The purpose of the test statistics is to evaluate the difference between the empirical and the fitted distribution function. The larger the difference, the larger the test statistics [Stephens, 1974]. The best model is thus associated with the smallest A_n and K_n test statistics.