Exploratory Analysis of Surrogate Metrics to Assess the Resilience of Water Distribution Networks
Abstract
This study compares and discusses the adequacy of surrogate resilience metrics proposed in the literature for resilience assessment of drinking water systems concerning demand increase and network redundancy. A sensitivity analysis is carried out for increasing flow rates using a conceptual case study with different layouts and demand scenarios, selecting several metrics to assess the resilience of two real network areas. Resilience metrics based on surplus energy are sensitive to network layout and demand scenarios. The network resilience index considers hydraulic reliability and network diameter uniformity. In contrast, the weighted resilience index also considers the network topology and gives importance to pipes with higher flow rates. Entropy-based resilience metrics mainly rely on the network flows' uniformity and are sensitive to pipe redundancy. The entropy metric most adequate to assess the hydraulic capacity is the diameter-sensitive flow entropy, since it is sensitive to the velocity inside the pipes. Topology metrics cannot assess the hydraulic capacity though evaluate the system redundancy (meshed-ness coefficient), robustness (central-point dominance) and water transportation efficiency (average-path length). Surrogate resilience metrics do not assess the system performance during a failure. They indicate systems which are better prepared to overcome failure events and increased demand events, providing vital information to drinking water systems management.
Key Points
-
Comparative study of surrogate resilience metrics based on surplus energy, entropy and graph-theory
-
Sensitivity analysis allowed the selection of adequate metrics to assess resilience to demand increase and pipe failure
-
The weighted resilience index is the most complete metric, capable of assessing hydraulic resilience and network redundancy
1 Introduction
Drinking water systems are crucial infrastructures worldwide, with the mission to continuously and efficiently deliver water in the desired amount and with good quality to all consumers. These systems daily face different events that may negatively impact the provided service. Resilience-related concerns have been gaining relevance over the years in urban water systems. In this context, resilience is defined as the “ability of any urban system, with its inhabitants, in a changing environment, to anticipate, prepare, respond to and absorb shocks, positively adapt and transform in the face of stresses and challenges” (International Organization for Standardization, 2021). A resilient system is, therefore, characterized by its absorptive, adaptive, and restorative capacities (Assad & Bouferguene, 2022; OECD, 2014; Shuang et al., 2019). The absorptive capacity is intrinsic to the drinking water system, while the adaptive and restorative capacities depend on the actions and management of the water utility. This work focuses in assessing the absorptive capacity of drinking water systems. It refers to the system's ability to continue operating and maintain its performance in the face of a certain event by absorbing its impact within an acceptable range.
The scientific community has largely centered its research on the absorptive capacity of resilience, focusing on reliability, robustness and redundancy aspects. Reliability refers to the ability of a system to provide the expected service in a given period and a specified environment (adapted from Mays, 2002). Similarly, robustness denotes the ability of a system to maintain a given performance level in the presence of unfavorable variations of operating conditions (Greco et al., 2012). Redundancy is related to the network topology and means that several supply paths are available.
Several metrics have been proposed to assess the resilience of drinking water systems, commonly designated as surrogate resilience metrics. The most used are hydraulic reliability metrics (Sitzenfrei et al., 2020; Zheng et al., 2014) that measure the hydraulic capacity of the network system to ensure the supply under uncertain demand conditions. The resilience index developed by Todini (2000) was the first to be developed and the most used. This metric has been subjected to several improvement attempts from different authors, introducing coefficients to account for pipe uniformity (Prasad & Park, 2004), topology and nodal importance (Sousa et al., 2022) or to extend the index to multi-source systems (Jayaram & Srinivasan, 2008). Flow entropy, based on Shannon's Entropy, is another metric widely used to assess resilience of drinking water systems, focusing on evaluating system's redundancy (Shuang et al., 2019). Several authors have developed different formulations of flow entropy (Awumah et al., 1990; Liu et al., 2014; Tanyimboh & Templeman, 1993). The calculation of these metrics requires the hydraulic simulation of the network, which may involve extensive computations in large and complex systems. To overcome this problem, topological metrics emerged as an alternative for computing resilience characteristics of a network, namely redundancy and robustness (Herrera et al., 2016; Meng et al., 2018; Pandit & Crittenden, 2016; Yazdani et al., 2011) (e.g., meshed-ness coefficient, central-point dominance, average path-length). However, topological metrics lack the verification of the hydraulic conditions and, therefore, should not be used alone to assess the resilience of a drinking water system (Assad & Bouferguene, 2022). Recent research focuses on metrics that combine both topological and hydraulic metrics (Raad et al., 2010; Sirsant & Reddy, 2020; Sousa et al., 2022).
The paper aims to compare and discuss different hydraulic and topological metrics used to assess the resilience of drinking water systems. Firstly, surrogate resilience metrics are applied to a conceptual network to understand their results for different operating scenarios, followed by a discussion of metric results. The results allow adequate metrics to assess hydraulic resilience (due to demand or water losses increase) and network redundancy (due to pipe failure) to be selected and applied to two real-life case studies. Obtained results are discussed and the main conclusions are drawn. The paper's main contribution is a detailed analysis of the different metrics and the identification of those more adequate to assess the resilience of a drinking water system from the two referred perspectives (i.e., hydraulic resilience and network's redundancy).
This paper has five main sections. First, the surrogate resilience metrics are framed and described and the respective formulations are presented. These metrics are classified as resilience metrics based on surplus energy, entropy metrics and topological metrics. In Section 2, a sensitive analysis is carried out to network redundancy and hydraulic resilience by applying the metrics to different layouts and demand scenarios of a conceptual network. This application aims to identify the most appropriate metrics to maintain the provision of water to consumers, in case of rising demands, and to discuss which metric is more sensitive to network redundancy. Lastly, the most appropriate metrics are selected and applied to two real-life drinking water network areas. Resilience should not compromise a good performance on other management dimensions (i.e., energy efficiency, infrastructure). Several energy efficiency alternatives are analyzed, and a discussion of the results from a management perspective is conducted, bringing resilience as a decision-making factor to drinking water systems management.
2 Surrogate Resilience Metrics
Table 1 presents the different metrics, and respective equations, assessed in the present study. A description of the metrics is presented in the following sections by types of metrics.
| Metric | Publication | Equation | |
|---|---|---|---|
| Resilience metrics based on surplus energy | |||
Resilience Index ( ) |
Todini (2000) |  |
(1) |
Network Resilience Index ( ) |
Prasad and Park (2004) |  |
(2) |
 |
(3) | ||
Modified Resilience Index ( ) |
Jayaram and Srinivasan (2008) |  |
(4) |
Target hydraulic resilience index ( ) |
Kongbuchakiat et al. (2022) |  |
(5) |
Weighted resilience index ( ) |
Sousa et al. (2022) |  |
(6) |
 |
(7) | ||
 |
(8) | ||
 |
(9) | ||
| Resilience metrics based on entropy | |||
Entropy ( ) |
Awumah et al. (1990) |  |
(10) |
 |
(11) | ||
Flow-entropy ( ) |
Tanyimboh and Templeman (1993) |  |
(12) |
| Diameter-sensitive flow entropy (DSFE) | Liu et al. (2014) |  |
(13) |
| Graph-theory metrics | |||
Link density ( ) |
Yazdani et al. (2011) |  |
(14) |
Central-point dominance ( ) |
Yazdani et al. (2011) |  |
(15) |
Average path-length ( ) |
Yazdani et al. (2011) |  |
(16) |
Meshed-ness coefficient ( ) |
Yazdani et al. (2011) |  |
(17) |
Spectral gap ( ) |
Yazdani et al. (2011) | The difference between the largest two eigenvalues 𝜆 of graph's adjacency matrix | |
Algebraic connectivity ( ) |
Yazdani et al. (2011) | The second smallest eigenvalue of the Laplacian matrix | |
-
Note. Where γ is the specific weight of water (9800 N/m3),
is the total number of demand nodes, 
is the total number of reservoirs, 
is the total number of pumps, 
is the demand in node 
(m3/s), 
is the flow input from reservoir 
(m3/s), 
is the flow entering node 
(m3/s), 
is the flow entering in node 
through pipe 
(m3/s), 
is the power of pump 
(kW), 
is the head in node 
(m), 
is the required head in node 
(m), 
is the uniformity coefficient of node 
for the network resilience index, 
is the number of pipes entering into node 
, 
number of pipes connected to node 
, 
is the diameter (mm) of pipe 
that is connected to node 
, 
is the minimum pressure (m), 
is the target pressure (m), 
is the pressure of node 
(m), 
is the uniformity coefficient, 
is the topological coefficient and 
is the importance of node 
for the weighted resilience index, 
is the network redundancy, 
is total flow into node 
(m3/s), 
is the sum of flows in all links of the network (m3/s), 
is the redundancy at node 
, 
is the pipe flow from node 
to node 
(m3/s), 
is the number of equivalent independent paths through the link from node 
to demand node 
, 
is the set of nodes on the upstream ends of links incident on node 
, 
is the inflow at source node 
(m3/s), 
is the number of supply nodes, 
is the total supply flow (m3/s), 
is the total flow reaching node 
(m3/s), 
is the demand at node 
(m3/s), 
is the set of all pipe flows emanating from node 
, 
is the pipe flow from node 
to node 
(m3/s), 
is the velocity constant (m/s) and 
represents the velocity of pipe 
(m/s), 
the number of edges, 
is the number of nodes of the mathematical graph, 
is the centrality of the most central-point, 
is the centrality of node 
and 
is the shortest distance 
between any pair of nodes 
and 
.
2.1 Resilience Metrics Based on Surplus Energy
The first type of metrics accounts for the energy available at the nodes above the minimum required to supply the consumers, also called surplus energy in the energy balance proposed by Mamade et al. (2017). Three other key concepts from the energy balance are also used in some metrics: the system input energy (by both reservoirs and pumps), the minimum required energy and the energy in excess (e.g., the difference between the system input energy and minimum required energy). The most used metric is the resilience index, proposed by Todini (2000). The metric aims to assess the system's capacity to respond to infrastructural (i.e., pipe bursts) and hydraulic (i.e., increase in demand) failures (Todini, 2000). The formulation, given by Equation (1), is based on the supply of more hydraulic power than the minimum required, to dissipate surplus energy in case of failure. The metric was developed initially for looped networks and not branched ones. As such, the metric's ability to assess a system's capacity to respond to pipe failures is questionable, since branched networks with considerable surplus energy can lead to high resilience values (Sousa et al., 2022).
Prasad and Park (2004) incorporated a uniformity coefficient, described by Equation (2), in the Todini's index, introducing the network resilience index, resulting in the formulation described by Equation (3). The uniformity coefficient accounts for reliable loops in the network, rewarding the uniformity of the pipes' diameter connected to a node.
Jayaram and Srinivasan (2008) proposed the modified resilience index, Equation (4), by quantifying the surplus energy available as a percentage of the minimum required energy. The authors question the ability of Todini's index to assess the resilience for multiple water sources. In the case of a double-source network with a considerate surplus energy and a reservoir with higher head, larger diameters promote a higher percentage of the total demand to be supplied by this reservoir. This leads to more energy input and can result in lower values of the resilience index. As the modified resilience index considers the surplus energy at the demand nodes rather than the surplus power available for internal dissipation, the authors suggest this metric is better for systems with multiple water sources.
More recently, Kongbuchakiat et al. (2022) proposed a change to the resilience index, similar to the modified resilience index, but by including a target value of the required variable (e.g., pressure), as described in Equation (5). The metric compares the available surplus of the variable under study to the target value. The interesting aspect of this work is the applicability of the metric to different variables, assessing different dimensions, such as the hydraulic dimension by accounting for pressure and the water quality dimension by accounting for free residual chlorine concentration. Only the hydraulic dimension will be analyzed herein.
Sousa et al. (2022) presented a new formulation, keeping the simplicity of Todini's index and adding node coefficients to account for specific network characteristics: pipe uniformity, network topology and node importance, Equations (6) and (7), respectively. The reason for these coefficients is that the developed resilience metrics tend to overestimate branched networks with good pressure regime and do not account for the importance of pipes that transport larger volumes of water. The metric developed is the weighted resilience index, see Equation (9). The topological coefficient aims to account for the network topology, penalizing the junctions with a lower number of connections. The importance coefficient ranks each pipe according to the water volume that passes through it. The uniformity coefficient accounts for the diameter uniformity of the pipes entering into a node.
The metrics consider that the system is more resilient for higher values. The resilience index and the network resilience index vary between 0 and 1, while the other metrics do not have an upper limit.
2.2 Resilience Metrics Based on the Entropy Concept
Another widely used surrogate resilience metric is flow entropy, which quantifies the network's redundancy. The use of the entropy concept is based on multiple supply paths for each node, with uniformly distributed flows, so that, when a path fails, the water is distributed through the alternative ways. These metrics consider that the system is more resilient for higher entropy results. The first application of the entropy concept to water distribution system analysis was developed by Awumah et al. (1990), setting up formulae that calculate water network entropy and node entropy, described by Equations (10) and (11). The node entropy function considers the number of equivalent paths through the link, a feature not considered in other entropy metrics. This metric is used to compute the entropy in the Water Network Tool for Resilience (WNTR) developed by the US Environmental Protection Agency (Klise et al., 2017).
Tanyimboh and Templeman (1993) presented an alternative way to calculate flow entropy in water distribution networks. The authors considered that the Awumah et al. (1990) metric did not obey the entropy theory assumption of the interdependency of the flows, solving it by establishing a conditional entropy function. The new formulation, described by Equation (12), applicable to single source networks, is a function of the source's entropy and the node’s entropy multiplied by the probability of the arriving flow rate.
Although flow entropy considers the uniformity of a network, through flows’ homogeneity, it does not consider the diameters of the pipes. Liu et al. (2014) updated the Tanyimboh and Templeman (1993) flow-entropy measure by incorporating a dimensionless parameter accounting the flow velocity in pipes. The diameter-sensitive flow entropy formulation is described by Equation (13).
2.3 Topological Metrics
Topological metrics have been applied to drinking water systems to assess the networks’ redundancy and robustness. The networks are configurations of interconnected components. Their structure can be represented as a mathematical graph composed of nodes representing elements at specific locations (nodes, reservoirs) and links representing the pipes. Yazdani et al. (2011) were the first authors to introduce a wide range of statistical and spectral metrics to assess resilience of drinking water systems based on graph-theory. Later, Pandit and Crittenden (2016) and Meng et al. (2018) applied some of these metrics to assess networks’ resilience. The most used graph-theory metrics, based on Yazdani et al. (2011) as it is the reference paper of graph-theory metrics application to drinking water systems, are revised herein.
Link density, Equation (14), is an indicator of the overall connectivity of the network's structure and is a ratio of the number of links over the maximum possible number for a given number of nodes.
Central-point dominance is a measure of concentration around a central location. This metric is limited by 0, corresponding to regular networks, and by 1, corresponding to star topology networks (see Figure S1 in Supporting Information S1). Its calculation, Equation (15), is based on the betweenness centrality of the network nodes, defined as the number of shortest geodesic paths between two given vertices that pass through that node divided by the total number of shortest geodesic paths between those two vertices. Though for some network types the best topology is the star type (like the internet and social networks), in drinking water networks higher centrality tends to decrease the resilience (Meng et al., 2018).
Average path-length, Equation (16), corresponds to the average number of steps along the shortest paths between all possible pairs of network nodes. This metric provides a view of network reachability and efficiency in water transportation. In this context, efficiency does not refer to the efficient use of resources or a measure of energy efficiency but to how efficiently the water is transported in a network, assuming it follows the shortest path. The metric infers that the system is more efficient with a smaller average path length.
The meshed-ness coefficient, Equation (17), estimates topological redundancy by finding the number of actually present independent loops as a percentage of the maximum possible loops in planar graphs. Higher values indicate a higher probability of the nodes remaining connected despite a link failure.
The first of the spectral metrics is the spectral gap. This metric reflects the robustness of the network and corresponds to the difference between the graph's adjacency matrix's first and second eigenvalues. It is used to detect networks that have optimal connectivity layouts. Smaller values may be indicative of a significant number of pipes and nodes that when removed might cause severe disruptions.
The last metric under analysis is the algebraic connectivity. This spectral metric corresponds to the second smallest eigenvalue of the network's normalized Laplacian matrix. It quantifies the network’s structural robustness and fault tolerance. Higher values indicates networks with higher fault tolerance and robustness.
3 Sensitivity Analysis of Resilience Metrics
3.1 Networks Characteristics and Simulated Demand Scenarios
-
layout L1: branched network with a single water source (reference layout);
-
layout L2: network with intermediate loops and a single water source;
-
layout L3: network with intermediate loops and two water sources.
-
Q0: equal demands in every node (reference demand scenario);
-
Q1: increased demand in nodes N01 and N02 (upstream network);
-
Q2: increased demand in nodes N05 and N09 (middle network);
-
Q3: increased demand in nodes N12 and N13 (downstream network);
-
Q4: Half of the original demand in all nodes (e.g., 2.5 l/s) and three times the original demand in node N13 (e.g., 15 l/s);
-
Q5: Half of the original demand in all nodes (e.g., 2.5 l/s) and five times the original demand in node N13 (e.g., 25 l/s).
Network and node characteristics are presented in Table 2.
| Layout | Node | Elevation (m) | Demand (l/s) | |||||
|---|---|---|---|---|---|---|---|---|
| L1 | L2 | L3 | Q0 | Q1 | Q2 | Q3 | ||
 |
 |
 |
N01 | 30 | 5 | 10 | 5 | 5 |
| N02 | 28 | 5 | 10 | 5 | 5 | |||
| N03 | 28 | 5 | 5 | 5 | 5 | |||
| N04 | 25 | 5 | 5 | 5 | 5 | |||
| N05 | 22 | 5 | 5 | 10 | 5 | |||
| N06 | 24 | 5 | 5 | 5 | 5 | |||
| N07 | 21 | 5 | 5 | 5 | 5 | |||
| N08 | 18 | 5 | 5 | 5 | 5 | |||
| N09 | 20 | 5 | 5 | 10 | 5 | |||
| N10 | 17 | 5 | 5 | 5 | 5 | |||
| N11 | 14 | 5 | 5 | 5 | 5 | |||
| N12 | 15 | 5 | 5 | 5 | 10 | |||
| N13 | 15 | 5 | 5 | 5 | 10 | |||
| N14 | 0 | 0 | 0 | 0 | 0 | |||
- Note. L1: branched network; L2: looped network with a single source; L3: looped network with multiple sources; Pipe diameters: thinner lines = 250 mm; thicker lines = 300 mm. Shaded values indicate increased demand nodes identification for each scenario.
For simplification, the pipe length and the Hazen-Williams roughness coefficient are considered the same in all pipes (1,000 m and 140 m0.37 s−1, respectively). Diameters vary from 250 mm (P04 to P11 and P15 to P18) to 300 mm in the rest of the network pipes (P01 to P03 and P12 to P14). Water sources are represented as reservoirs R1 and R2, with 70 m of constant water level.
Hydraulic simulations are carried out for the combinations of network layouts and demand scenarios, identified by their names. For instance, simulation L1_Q0 corresponds to the layout L1 run for the demand scenario Q0. The metrics' computation is done in python environment, the EPANET hydraulic simulations run via WNTR, and the topological metrics are computed through the NetworkX package.
In the present work, the required head (
) corresponds to the elevation of the node (z) added the service pressure head (
) of 20 m, that is 
. For the target hydraulic resilience index, 
equals the service pressure head and 
is a minimum admissible service pressure of 10 m.
3.2 Resilience Metrics Based on Surplus Energy
Table 3 presents the resilience metrics based on surplus energy results for the different network layouts and demand scenarios Q0 to Q3. Demand scenarios Q4 and Q5 results (Table S1 in Supporting Information S1) are redundant with scenarios Q1 to Q3, showing an increase in resilience values, since the general decrease in demand promotes lower head losses in the system. In both scenarios, the extreme higher consumption in node N13 does not promote enough water flow to counterbalance the general decrease in the system demand.
| Metric | Value | Variation | Value | Variation | Value | Variation | Value | Variation |
|---|---|---|---|---|---|---|---|---|
| L1_Q0 | L1_Q1 | L1_Q2 | L1_Q3 | |||||
 |
0.72 | 0.69 | − 4% ↘ | 0.63 | − 13% ↘ | 0.60 | − 16% ↘ | |
 |
0.70 | 0.68 | − 4% ↘ | 0.62 | − 12% ↘ | 0.59 | − 16% ↘ | |
 |
49.9 | 45.2 | − 10% ↘ | 43.7 | − 12% ↘ | 44.1 | − 12% ↘ | |
 |
3.06 | 2.91 | − 5% ↘ | 2.80 | − 8% ↘ | 2.78 | − 9% ↘ | |
 |
0.11 | 0.13 | 10% ↗ | 0.10 | − 16% ↘ | 0.10 | − 13% ↘ | |
| L2_Q0 | L2_Q1 | L2_Q2 | L2_Q3 | |||||
|---|---|---|---|---|---|---|---|---|
 |
0.77 | +7% ↗ | 0.74 | − 5% ↘ | 0.69 | − 11% ↘ | 0.69 | − 11% ↘ |
 |
0.75 | +7% ↗ | 0.72 | − 4% ↘ | 0.68 | − 11% ↘ | 0.68 | − 10% ↘ |
 |
53.4 | +7% ↗ | 48.1 | − 11% ↘ | 48.0 | − 11% ↘ | 50.5 | − 6% ↘ |
 |
3.21 | +5% ↗ | 3.04 | − 6% ↘ | 2.98 | − 7% ↘ | 3.04 | − 5% ↘ |
 |
0.15 | +35% ↗ | 0.15 | 1% ↗ | 0.13 | − 18% ↘ | 0.15 | − 5% ↘ |
| L3_Q0 | L3_Q1 | L3_Q2 | L3_Q3 | |||||
|---|---|---|---|---|---|---|---|---|
 |
0.95 | +32% ↗ | 0.94 | − 1% ↘ | 0.93 | − 3% ↘ | 0.94 | − 1% ↘ |
 |
0.93 | +32% ↗ | 0.92 | − 1% ↘ | 0.92 | − 2% ↘ | 0.93 | − 1% ↘ |
 |
66.0 | +32% ↗ | 61.4 | − 9% ↘ | 64.9 | − 2% ↘ | 68.9 | +5% ↗ |
 |
3.73 | +22% ↗ | 3.60 | − 4% ↘ | 3.68 | − 2% ↘ | 3.79 | +2% ↗ |
 |
0.28 | +141% ↗ | 0.25 | − 8% ↘ | 0.28 | 3% ↗ | 0.27 | − 4% ↘ |
-
Note.
– Resilience index; 
– Network resilience index; 
– Modified resilience index; 
– Target hydraulic resilience index; 
– Weighted resilience index; Variation of L2_Q0 and L3_Q0 ↗ improve; → same; ↘ worse the resilience relative to layout L1; Variation of Q1, Q2 and Q3 ↗ improve; → same; ↘ worse the resilience relative to scenario Q0 of the respective layout.
Variation of L2_Q0 and L3_Q0 are relative to the reference layout L1_Q0, whilst variations for Q1, Q2 and Q3 are relative to demand scenario Q0 of the respective layout.
Resilience metrics based on surplus energy demonstrate the capacity to account for network's redundancy. All metrics have lower resilience values for the branched network (L1) and higher for the looped network with multiple sources (L3). The weighted resilience index (
) is the most sensitive metric to assess changes in network topology, showing the highest variations. The target hydraulic resilience index (
) is the least sensitive metric to layout changes, having the lowest increase in resilience for the more redundant networks (L2 and L3) relative to the branched network (L1).
Resilience index (
), network resilience index (
), and modified resilience index (
) increase at a similar rate from the looped network with a single source (L2) to the network with multiple sources (L3). Todini's 
does not compromise in terms of multi-sources networks, as perceived by Jayaram and Srinivasan (2008), demonstrating that, to some extent, the 
can also assess higher resilience values in multi-sources networks.
A demand increase originates a decrease in the surplus energy in the system. These metrics are suitable for assessing the hydraulic resilience since most results demonstrate smaller resilience for higher demand scenarios (Q1 to Q3), compared with the reference scenario Q0.
The resilience index (
) and network resilience index (
) are sensitive to the energy dissipated in the system, with the latter also sensitive to the network diameters' uniformity (e.g., 
= 0.72 and 
= 0.70 for L1_Q0). Both metrics demonstrate the same tendency of decreasing resilience with demand increase (e.g., −4% for L1_Q1, −11% for L2_Q2, −1% for L3_Q3). The location of the higher demand node has implications for the resilience value, particularly in branched networks (L1): for higher demands further from the source, the water flowing through the pipes is increased. Consequently, higher head losses occur in the network. As such, higher resilience for the upstream increased demand scenario (
for L1_Q1) and lower resilience for the downstream increased demand scenario (
for L1_Q3) is obtained. In the looped network (L2), water reaches the downstream part of the network from different alternative paths, as the flow is distributed through them. The spatially distributed flows in the loop promote minor water head losses so that results from scenarios with higher demands in the middle (L2_Q2) and downstream (L2_Q3) of the network have similar results (
). In the double source network (L3), the decrease in resilience due to the demand increase, for every demand scenario, is smaller than in single-source networks (e.g., 
varies −16% for L1_Q3 and -1% for L3_Q3).
Modified resilience index (
) and target hydraulic resilience index (
) are sensitive to the elevation of nodes with higher demand. While the 
and the 
are metrics that relate the surplus energy at the nodes with the energy in excess (input energy minus the required energy), the 
and the 
are independent of the input energy. The 
is a percentage of the surplus energy by the energy necessary and the 
relates the surplus pressure to the difference between target pressure and minimum pressure, both metrics accounting mainly for the surplus energy. The surplus energy is higher in nodes with lower elevation, with a higher effect on the final resilience value. As such, the location of the higher demands concerning the source no longer affects the value of the final resilience result, but the elevation of the higher demand node. This can be perceived by the results of scenarios with higher demands downstream of the loop (lower elevation nodes with higher demands, see nodes N12 and N13 elevation for scenario Q3 in Table 2). For the single-source looped network (L2), this demand scenario (L2_Q3) has the second best value of resilience, just behind the reference equal demand scenario (L2_Q0) (
worsens −6% in L2_Q3, while it worsens −11% for L2_Q1 and L2_Q2, Table 3). In the double-source looped network (layout L3), this dependency is even more noticeable, with both metrics (
and 
) having the highest resilience values for the scenario where the higher demand is downstream of the loop (L3_Q3).
The weighted resilience index (
) demonstrates to be the metric most sensitive to topological changes (Table 3) and to demand increase. Like 
and 
, this metric is a percentage of the surplus energy and energy in excess, but has three additional coefficients that influence the nodes' importance. The topological coefficient (
), described by Equation (6) in Table 1, is the main driver for the index to be able to penalize branched networks and promote looped networks. However, this coefficient is dependent on the directionality of the flow. It may result in different topological coefficients for the same topology (i.e., multi-source network, L3, has other flow distribution for demand scenario Q2, see Figure S2 in Supporting Information S1). The importance coefficient (
), Equation (7) in Table 1, is the only coefficient that contributes differently for the different demand scenarios, particularly in layouts L1 and L2 since the topological and uniformity coefficients remain the same. This coefficient gives more importance to nodes with higher inflows rather than lower inflows, scaling the inflow in a node by the maximum node inflow in the network. This situation promotes the increased resilience value in scenarios with higher demands near the source (Q1). The uniformity coefficient (
), Equation (8) in Table 1, penalizes different diameters and promotes similar diameters in the network.
Resilience metrics based on surplus energy have demonstrated to be sensitive to both network layouts and demand scenarios, being two metrics recommended herein to assess the resilience of drinking water systems. The first metric suggested is the weighted resilience index (
), since it proves to be the most complete metric by contemplating not only the hydraulic reliability of the system, but also the network topology, the network uniformity and the importance of pipes with higher flow. The second metric is the network resilience index (
), since it is very similar to the original resilience index, by attending to the surplus energy susceptible to be dissipated in the case of demand increase, and considers the uniformity of the diameters in the network.
3.3 Resilience Metrics Based on the Entropy Concept
Table 4 presents the resilience metrics based on the entropy concept results for the sensitivity analysis. To assess the effect of the velocity constant 
in diameter-sensitive flow-entropy (
), for the different reference layouts (L1_Q0, L2_Q0, and L3_Q0), two values were considered: the original 
m/s (Liu et al., 2014), and a constant of the same order of magnitude of the lowest velocities 
m/s. 
with 
m/s has a lower result (i.e., indicating lower resilience) for the looped network (L2_Q0), which contradicts the positive tendency of improving resilience with more redundant networks. Although Liu et al. (2014) refer that the velocity parameter does not significantly affect the relationship between 
and reliability, from the results obtained in this study, the order of magnitude of the velocity constant 
has a considerable effect in the final value of the 
. If 
is higher than the smaller velocities in the pipes, the velocity term becomes considerably high and leads to inaccurate entropy results. Having 
of the same order of magnitude of the lowest velocities, the results are more coherent with those from the other indices. Onwards, 
is only shown and discussed with 
m/s.
| Metric | Value | Variation | Value | Variation | Value | Variation | Value | Variation |
|---|---|---|---|---|---|---|---|---|
| L1_Q0 | L1_Q1 | L1_Q2 | L1_Q3 | |||||
 |
2.07 | 2.04 | −1% ↘ | 2.08 | +1% ↗ | 2.09 | +1% ↗ | |
 |
2.56 | 2.52 | −2% ↘ | 2.52 | −2% ↘ | 2.52 | −2% ↘ | |
 ( = 0.1 m/s) |
2.08 | 2.00 | −4% ↘ | 1.84 | −12% ↘ | 1.97 | −6% ↘ | |
 ( = 1 m/s) |
12.6 | – | – | – | – | – | – | |
| L2_Q0 | L2_Q1 | L2_Q2 | L2_Q3 | |||||
|---|---|---|---|---|---|---|---|---|
 |
2.39 | +16% ↗ | 2.35 | −2% ↘ | 2.36 | −2% ↘ | 2.51 | +5% ↗ |
 |
2.98 | +16% ↗ | 2.88 | −4% ↘ | 2.86 | −5% ↘ | 3.05 | +3% ↗ |
 ( = 0.1 m/s) |
2.26 | +9% ↗ | 2.15 | −5% ↘ | 2.16 | −5% ↘ | 2.09 | −8% ↘ |
 ( = 1 m/s) |
11.8 | −7% ↘ | – | – | – | – | – | – |
| L3_Q0 | L3_Q1 | L3_Q2 | L3_Q3 | |||||
|---|---|---|---|---|---|---|---|---|
 |
2.43 | +17% ↗ | 2.38 | −2% ↘ | 2.52 | +5% ↗ | 2.38 | −2% ↘ |
 |
2.83 | +10% ↗ | 2.73 | −4% ↘ | 2.89 | +2% ↗ | 2.73 | −4% ↘ |
 ( = 0.1 m/s) |
3.01 | +45% ↗ | 2.93 | −4% ↘ | 2.64 | −12% ↘ | 2.93 | −4% ↘ |
 ( = 1 m/s) |
12.6 | +25% ↗ | – | – | – | – | – | – |
-
Note.
– Awumah entropy; 
– Flow-entropy; 
– Diameter-sensitive flow entropy. Variation of L2_Q0 and L3_Q0: ↗ improve; → same; ↘ worse the resilience relative to layout L1; Variation of Q1, Q2 and Q3: ↗ improve; → same; ↘ worse the resilience relative to scenario Q0 of the respective layout.
Entropy metrics are more sensitive to pipe redundancy than to source redundancy. The single-source more redundant network (looped network, L2) has higher entropy values than the lower redundant network (branched network, L1). However, higher differences were expected, since entropy is mainly used to account for the network's redundancy (Awumah et al., 1990; Tanyimboh & Templeman, 1993). As such, using this type of metrics for assessing the redundancy of branched networks is questionable, per Sirsant and Reddy (2020). Double-source looped networks (layout L3) do not have considerable higher entropy values than looped networks. In reality, flow-entropy (
) infers that the single-source looped network has better resilience (
= 2.98 for L2_Q0) than the double-source looped network (
= 2.83 for L3_Q0), contrarily to the tendency of diameter-sensitive flow entropy (
), Awumah entropy (
) and all resilience metrics based on surplus energy. This also questions the entropy metrics' applicability to assess multi-source systems' resilience.
Demand variation does not have the same effect on entropy metrics as in the resilience metrics based on surplus energy, as the variation in resilience metrics is around –10% (see Table 3) while in entropy metrics is –4% (Table 4). Diameter-sensitive flow-entropy (
) is the entropy metric most sensitive to demand variations. Due to the additional velocity term, this metric becomes more sensitive to the hydraulic capacity of the system. Higher demands (scenarios Q1 to Q3) promote higher velocities in the system, resulting in lower resilience when compared to the reference demand scenario Q0. Awumah entropy (
) and flow-entropy (
) have small variations (between 1% and 5%) as there is reasonable uniformity in the network's flows distribution. Neither metric considers the hydraulic capacity of the network, as concluded by Cimorelli et al. (2018). This can result in misleading resilience values because a system with good distributed flows has good entropy values but is unable to comply hydraulically (i.e., nodal pressure) with the consumers. In the double-source looped network (L3), upstream and downstream network higher consumptions (L3_Q1 and L3_Q3) have the same tendency for all metrics because the flows are the same but symmetric. The increase in demand in the middle of the network (L3_Q2) promotes a more uniform distribution of flows, reason for Awumah entropy (
) and flow-entropy (
) higher values, but have higher velocities in the network, the reason for the decrease in 
.
Resilience metrics based on the entropy concept have demonstrated to be sensitive to pipe redundancy, being the 
recommended to assess resilience, due to its sensitivity to demand variation.
3.4 Topological Metrics
Topological metrics do not consider the hydraulic behavior nor the existence of network multi-sources, focusing only on the network's structure. Obtained results for the branched and looped network (L1 and L2, respectively), depicted in Table 5, present an increase in every metric for the looped network. Link density (
) demonstrates an increase due to the addition of the four pipes in the looped area, increasing the connectivity between the nodes. Central-point dominance (
) shows a decrease of 45% for the looped network compared with the results for the branched network, demonstrating the higher robustness of the looped network. Average path length (
) shows a decrease from the branched network to the looped network, increasing the system's efficiency in transporting water. This result agrees with Yazdani et al. (2011) conclusion that looped networks are more efficient than branched networks. The meshed-ness coefficient (
) is null for the branched network and 0.16 for the looped network, reflecting the looped area's redundancy increase. Higher values of spectral gap (
) and algebraic connectivity (
) in layout L2 when compared with layout L1, once again reflecting the higher robustness and faultier tolerance of looped networks.
| Metric | Value | Value | Variation |
|---|---|---|---|
| Layout L1 | Layout L2 | ||
 (connectivity) |
0.13 | 0.17 | +29% ↗ |
 (robustness) |
0.53 | 0.29 | −45% ↗ |
 (efficiency) |
3.28 | 3.16 | −3% ↗ |
 (redundancy) |
0.00 | 0.16 | +16% ↗ |
 (robustness) |
4.98 | 5.87 | +18% ↗ |
 (robustness) |
0.11 | 0.15 | +42% ↗ |
- Note. LD – Link density (higher is better); CPD – Central Point Dominance (lower is better); APL – Average path length (lower is better); MC – Meshed-ness coefficient (higher is better); AC – Algebraic connectivity (higher is better); SG – Spectral gap (higher is better); Variation of layout L2: ↗ improve; → same; ↘ worse the resilience relative to reference layout L1.
Graph-theory metrics do not reflect the resilience of a drinking water system to increasing demands, but are metrics capable of assessing the system's resilience to eventual pipe bursts. These metrics assess different types of resilience aspects, and a single metric's assessment may be misleading. As such, three metrics are suggested to evaluate the network's topology: central-point dominance, assessing the robustness of the network; average-path length, accounting the transportation efficiency of water in the network and finally, the meshed-ness coefficient to infer the redundancy of the system.
4 Resilience Assessment in Real Case Studies
4.1 Case Studies' Description
The case studies presented herein refer to a drinking water system in the Algarve region (Portugal). The system comprises five network areas, depicted in Figure 1a. Network areas 3 and 5 were identified as priority areas to implement energy efficiency improvement measures, using the methodology described in Loureiro et al. (2020). Herein, a resilience analysis is carried out to assess the status quo situation and the energy efficiency improvement measures. The water utility provided the EPANET hydraulic models of each network area (Figures 1b and 1c) set for 24 hr simulations. Each metric is calculated for every time step (1h) and the average results for each time step obtain the final resilience result.
Water distribution system layout: (a) network areas, and schematic representation of the hydraulic models of (b) network area 3 and (c) network area 5.
-
Area3_M0: Statu quo, not considering interventions;
-
Area3_M1: Replacement of 900 m of pipes with larger nominal diameters;
-
Area3_M2: Replacement of the pump groups considering the opportunity to improve the pump operating points (i.e., diminishing 5 m of pump head) and replacement of 900 m of pipes with larger nominal diameters.
-
Area5_M0: Statu quo, not considering interventions;
-
Area5_M1: Network sectorization (allowing direct supply from the reservoir to the lowest area) by constructing ca. 236 m and replacement of ca. 1 155 m of water pipes, considering appropriate nominal diameters and replacement of the pump groups considering the opportunity to improve the pump operating points (i.e., diminishing flowrate in the first pumping station by ca. 60%).
Algarve region is a tourist site with high differences between summer and winter. Daily consumption is approximately six times higher in the summer, and spatially demand distributions are different along with demand patterns, presented in Figure S3 in Supporting Information S1. Part of the users' water consumption is provided by groundwater that is beginning to show signs of saltwater intrusion. Water utilities are concerned that the consumers will increase water consumption from the drinking water network. To consider these situations, a double-consumption scenario for the summer model and the summer and winter consumption scenarios are analyzed herein.
4.2 Resilience Assessment Results and Discussion
The complete results of the resilience assessment carried out for both network areas statu quo situation and respective improvement measures, and for each consumption scenario are presented in Tables S2 to S7 in Supporting Information S1. Results presented herein correspond to six selected metrics from the sensitive analysis, namely, the network resilience index (
) and weighted resilience index (
) from the resilience metrics based on surplus energy, diameter-sensitive flow-entropy (
) from entropy-based type and central-point dominance (
), average path length (
) and meshed-ness coefficient (
) from graph-theory.
4.2.1 Resilience Metrics Based on Surplus Energy
The winter scenario has smaller nodal consumption and consequently lower flow rates and lower dissipated energy, making the nodal pressure close to the maximum with low-pressure fluctuations, defined here as a good pressure performance of the system. As expected, the resilience is higher in the winter and smaller in the double-consumption scenario (Figure 2).
Sensitivity analysis of network resilience index (
) and weighted resilience index (
) for the statu quo situation and different demand scenarios: (a) area 3 and (b) area 5.
These metrics can assess non-complying pressure nodes by returning negative nodal resilience. The results, however, can be attenuated by pressure-compliant nodes of the network, resulting in a low resilient system. In dynamic simulations, the attenuation of smaller or negative resilience results of the higher demand times steps can also occur, due to higher resilience values for the lower demand time steps. This situation is verified for network areas 3 and 5 for the double-summer consumption scenario. For example, network area 5, at the peak hour demand (4h), has a lower elevation zone with pressures above 40 m and a non-complying pressure zone (red and dark-blue zones in Figure S4 in Supporting Information S1). The good resilience values attenuate the negative resilience values, resulting in smaller values of resilience (i.e., 
= 0.29 at 4h). At non-peak hours, the system has good surplus energy overall and high resilience values (i.e., 
= 0.89 at 17hr), leading to a fair final resilience value (i.e., 
= 0.67).
Another aspect worthy of note is the difference in magnitude observed in the weighted resilience index (
) results for network area 3 to network area 5 (e.g., for the summer scenario, 
= 0.19 in area 3 and 
= 0.03 in area 5). These results could lead to the discussion that network area three is considerably more resilient than network area 5. However, 
is very dependent on the importance coefficient, and the higher the difference in maximum and smaller flows is, the smaller the resilience becomes. As such, this metric cannot provide a fair comparison of resilience between different networks, particularly between systems with different magnitude of flow rates.
Table 6 presents the alternatives' results for network area 3 and area five in winter and summer scenarios. Double-consumption scenario results are not shown because they are similar to the summer scenario.
| Demand scenario | Metric | Value | Value | Variation | Value | Variation |
|---|---|---|---|---|---|---|
| Area3_M0 | Area3_M1 | Area3_M2 | ||||
| Winter |  |
0.97 | 0.94 | −2.2% ↘ | 0.94 | −2.6% ↘ |
 |
0.23 | 0.23 | 0.2% ↗ | 0.22 | −1.4% ↘ | |
| Summer |  |
0.83 | 0.90 | 8.2% ↗ | 0.84 | 1.0% ↗ |
 |
0.19 | 0.20 | 9.9% ↗ | 0.19 | 1.5% ↗ | |
| Area5_M0 | Area5_M1 | |||||
|---|---|---|---|---|---|---|
| Winter |  |
0.89 | 0.89 | 0.0% → | ||
 |
0.04 | 0.06 | 54.8% ↗ | |||
| Summer |  |
0.85 | 0.83 | −2.5% ↘ | ||
 |
0.03 | 0.04 | +39.9% ↗ | |||
-
Note.
– Resilience index; 
– Network resilience index; 
– Weighted resilience index; Variation of _M1 and _M2: ↗ improve, → same, ↘ worse the resilience relative to the status quo _M0 alternative.
Alternative Area3_M1 (pipe replacement with higher diameters) resilience increases relative to the status quo situation due to the lower dissipated energy in the system, perceived in the summer results of both metrics. In the winter scenario, the dissipated energy in the status quo situation is insignificant making the surplus energy gain insubstantial. As the uniformity coefficient of the network resilience index (
) decreases due to the diameters' difference, the final 
is also smaller.
In alternative Area3_M2 (pipe and pump replacement), the diminished 5 m of pump head leads to smaller surplus energy at the nodes, due to the smaller input energy. In the summer scenario, the gain in surplus energy from the smaller dissipated energy overcomes the loss from the smaller input energy, resulting in higher resilience systems. In the winter, the opposite occurs with the smaller input energy being the main driver for the smaller resilience result.
The network sectorization (alternative Area5_M1) changes the topology of the network through the construction of a new pipe, decreases the input energy (ca. 60% of the inflow rate is not elevated and enters the system at a head level of 53 m) and redistributes the flows throughout the network. The weighted resilience index (
) is the only metric that accounts for the new connectivity of the network and flow redistribution.
4.2.2 Resilience Metrics Based on the Entropy Concept
Diameter-sensitive flow entropy (
) capacity to account changes in consumption is perceived by 
results for the different demand scenarios of network area 3 and area 5 status quo situation (Table S3 in Supporting Information S1), with the winter scenario having the higher resilience and double-consumption the smallest. The magnitude different from the winter scenario (
, for network area 3) to the summer (
) and double-consumption (
) scenarios is due to the smaller velocities in the winter. The velocity constant (
m/s) is considered the same for the three demand scenarios to be able to compare the resilience.
Table 7 presents the 
results for the different improvement measures of network's area 3 and area 5, in the summer scenario, since the results have the same tendency for the other demand scenarios. The 
is sensitive to velocities inside the network, but not to reduction of input energy. Both improvement measures for network area 3 (Area3_M1 and Area3_M2) rely on replacing the main pipe conducting water to the southern part of the sector by increasing its diameter, consequently decreasing the velocity in the pipe and as expected increasing the resilience (Liu et al., 2014). However, the decrease in input energy (Area3_M2) does not change the flow distribution in the system, nor the velocity, promoting equal results for both improvement metrics.
| Metric | Value | Value | Variation | Value | Variation |
|---|---|---|---|---|---|
| Area3_M0 | Area3_M1 | Area3_M2 | |||
 |
5.14 | 5.17 | 0.6% ↗ | 5.17 | 0.6% ↗ |
| Area5_M0 | Area5_M1 | ||||
|---|---|---|---|---|---|
 |
7.8 | 8.5 | 9.4% ↗ | ||
-
Note.
– Diameter-sensitive flow entropy; Variation of _M1 and _M2: ↗ improve, → same, ↘ worse the resilience relative to the status quo _M0 alternative.
The 
capacity to account for the network's redundancy is verified in the network area 5 sectorization (Area5_M1), increasing its resilience from the topology change, originating the redistribution of flows and the new velocities.
4.2.3 Topological Metrics
Graph-theory metrics consider the network's topology regardless of the system's hydraulic behavior. Table 8 presents previously selected graph-theory metrics results to statu quo alternatives and the sectorization of network area 5 (Area5_M1). As expected, alternative Area5_M1 shows an improvement in the selected graph-theory metrics.
| Metric | Value | Value | Value | Variation |
|---|---|---|---|---|
| Area3_M0 | Area5_M0 | Area5_M1 | ||
 (robustness) |
0.37 | 0.39 | 0.37 | − 4% ↗ |
 (efficiency) |
7.58 | 27.50 | 26.93 | − 8% ↗ |
 (redundancy) |
0.01 | 0.02 | 0.02 | +6% ↗ |
-
Note.
– Central Point Dominance (lower is better); 
– Average path length (lower is better); 
– Meshed-ness coefficient (higher is better); Variation of _M1: ↗ improve, → same, ↘ worse the resilience relative to the status quo _M0 alternative.
Comparing the different network areas’ results, central-point dominance (
) has similar values indicating that the structural density around a central location is similar between networks. Network area five is longer than area 3 and so is the average-path length (
) due to the higher travel of the water in the network. The meshed-ness coefficient (
) is also higher for network area 5, being the values in the same order of magnitude as the results presented in Yazdani et al. (2011).
5 Conclusions
Several surrogate metrics are proposed in the literature to assess resilience. These can be divided into three main types: resilience metrics based on surplus energy, resilience metrics based on the entropy concept, and topological metrics. Given the lack of studies with comparative analyses of the results obtained by the different resilience metrics, this research carries out an extensive sensitivity analysis of the different resilience metrics for several network topologies and demand scenarios, establishes recommendations on the adequate metrics for diagnosis and decision-support and discusses the results for several measures to improve the resilience of existing networks.
Resilience metrics based on surplus energy are sensitive to network layouts and demand scenarios. In systems with good pressure performance, the resilience index, the network resilience index, the modified resilience index and the target hydraulic resilience index can have high resilience results for non-redundant branched networks. In contrast, the weighted resilience index penalizes branched networks, assessing network's redundancy. The weighted resilience index is the first metric suggested since it is the most complete considering hydraulic reliability, network topology, diameter uniformity and gives importance to pipes with higher flow rates. However, this metric should only be used to compare alternative measures for the same system. The second metric suggested is the network resilience index. It is very similar to the original resilience index but also considers the uniformity of the diameters in the network. This metric should be used with parsimony since diameter uniformity may promote contradictory resilience results in good pressure performance systems.
Resilience metrics based on the entropy concept have been demonstrated to be the most sensitive to pipe redundancy. Awumah entropy and flow entropy are incapable of accounting demand variation, relying mainly on the uniformity of the flow distribution. Systems with good flow uniformity have good resilience results, even when the system is not complying with minimum pressure requirements. The diameter-sensitive flow entropy is the metric suggested to assess resilience. The metric is sensitive to the velocity in pipes, asserting higher resilience values in systems with smaller head losses. The velocity constant should be in the same order of magnitude as the lower velocity values.
Topology metrics are unable to assess the resilience of a drinking water system to increasing demands but are capable of evaluating the system's resilience to eventual pipe bursts. To assess different aspects of resilience, three metrics are suggested to consider the network's topology: central-point dominance, assessing the robustness of the network; average-path length, accounting for the water transportation efficiency of the network; and the meshed-ness coefficient inferring on the redundancy of the system.
The assessment of a drinking water system's resilience should be able to account for hydraulic resilience and system's redundancy. The hydraulic capacity, to overcome eventual increasing demands is fundamental to a resilient system, as well as redundancy, with the presence of alternative paths in the eventuality of pipe bursts, to keep providing a good service to the consumers. The applicability of the metrics to different sizes and shapes of networks is also important to consider. So far, no metric can account for these three aspects (hydraulic resilience, system's redundancy and applicability to different networks), but a joint assessment of different resilience metrics gives a broader perspective of the drinking water system's resilience. The weighted resilience index is the final suggested metric to assess hydraulic resilience and network's redundancy.
Surrogate resilience metrics do not assess the system performance during a failure per se, but the eventual capacity of the system to overcome disturbances. From an asset management point of view, surrogate resilience metrics can indicate systems better prepared to overcome eventual failures events.
This research is a step forward in resilience assessment, providing a comparative study and use recommendations. Future work should focus on developing a surrogate resilience metric that can account for hydraulic resilience and system's redundancy and be applicable to different types of networks. The sensitivity analysis carried out herein was simple, and a more extensive analysis to demands and pipes diameters could be useful (i.e., global sensitivity analysis). In addition, the performance of the system in failure conditions was not assessed in this work. Further analysis comparing surrogate resilience metrics with quantitative approaches to reliability, in the case of pipe and pump failures, could also provide new insights into resilience assessment.
Acknowledgments
The authors would like to thank the Portuguese water utility that provided the real case study and Fundação para a Ciência e Tecnologia (FCT) for funding the research unit CERIS in the framework project UIDB/04625/2020 and the H2DOC doctoral program with Ph.D. Grant PD/BD/150694/2020.
Open Research
Data Availability Statement
Data supporting this research are available in Carneiro (2023). This data include the input data to the conceptual model and a code in Python Language to calculate the surrogate resilience metrics indicated in Table 1. Data relative to the real case study is not accessible to the public or research community.
