Climate Adaptation as a Control Problem: Review and Perspectives on Dynamic Water Resources Planning Under Uncertainty

Climate change introduces substantial uncertainty to water resources planning and raises the key question: when, or under what conditions, should adaptation occur? A number of recent studies aim to identify policies mapping future observations to actions—in other words, framing climate adaptation as an optimal control problem. This paper uses the control paradigm to review and classify recent dynamic planning studies according to their approaches to uncertainty characterization, policy structure, and solution methods. We propose a set of research gaps and opportunities in this area centered on the challenge of characterizing uncertainty, which prevents the unambiguous application of control methods to this problem. These include exogenous uncertainty in forcing, model structure, and parameters propagated through a chain of climate and hydrologic models; endogenous uncertainty in human‐environmental system dynamics across multiple scales; and sampling uncertainty due to the finite length of historical observations and future projections. Recognizing these challenges, several opportunities exist to improve the use of control methods for climate adaptation, namely, how problem context and understanding of climate processes might assist with uncertainty quantification and experimental design, out‐of‐sample validation and robustness of optimized adaptation policies, and monitoring and data assimilation, including trend detection, Bayesian inference, and indicator variable selection. We conclude with a summary of recommendations for dynamic water resources planning under climate change through the lens of optimal control.


Introduction
Water resources planners face the challenge of adapting to climate change with a portfolio of potential actions, including infrastructure, operating rules, and demand conservation to reduce vulnerability (Füssel, 2007;Hallegatte, 2009). These decisions are often supported by simulation and optimization methods tailored to long-term projections of hydroclimate. However, these projections are clouded by a "cascade" of uncertainty (Wilby & Dessai, 2010), propagated through the chain of greenhouse gas emissions, climate models and their initial conditions, regional downscaling, hydrologic models, and human-environmental systems models, only a portion of which can be captured in ensemble projections (Stainforth et al., 2007). This is particularly the case for the uncertain trends in flood and drought risk that drive infrastructure planning (Asadieh & Krakauer, 2017;Dottori et al., 2018;Trenberth et al., 2014).
Under these conditions, it is difficult to apply traditional decision-making methods such as cost-benefit analysis and expected value utility theory, which require exact probabilities and commensurate values Dennig, 2017;Lempert, 2015;Tol, 2003). In response, several new computational frameworks have emerged to support climate adaptation. Broadly, these can be grouped into two categories ( Figure 1): robust planning, with a focus on identifying alternatives that perform acceptably under a wide range of future conditions, and dynamic planning, which aims to identify adaptation policies that respond to new observations over time. While these are not mutually exclusive-a dynamic policy can also be robust, though the reverse is not necessarily true Maier et al., 2016)-they face very different challenges in experimental design and implementation. Broad classification of decision support frameworks for water resources planning under climate change. This paper reviews dynamic planning studies (highlighted) under the framing of an optimal control problem. Dynamic plans may also be robust, though the reverse is not necessarily true.

Figure 2.
Overview of dynamic water resources planning under climate change, structured as a control problem. Policies are optimized to one or more performance metrics evaluated by the system model according to the state transition equation. The system receives a combination of climate and nonclimate forcing inputs. We assume a watershed-scale planning problem and therefore omit the feedback from the system to global emissions. forecasted states and fluxes in the system. The objectives include one or more cost functions computed from the trajectory at each time step, J t (τ t ), which allows for time-variant cost functions (e.g., involving a discount factor or other dynamic changes). These objectives could represent the cost of water supply shortage, flood risk, or environmental damages, in addition to the cost of implementing adaptation actions. The optimization problem is to choose the policy π that minimizes the expected sum of costs over a finite planning horizon H: The expectation operator over the stochastic forcing variable could be replaced with a different statistical operation, such as the median or maximum. Similarly, a different statistical operator could be used in place of the inner summation of the cost function over time. The problem can also be extended to multiple objectives, J = (J 1 , J 2 , …, J M ), resulting in a Pareto-optimal set of policies. These choices in formulating the objective function(s) are crucial to the outcome of the optimization, including their combined effect with the choice of scenarios (e.g., Quinn, Reed, Giuliani, et al., 2017;Quinn, Reed, Giuliani et al., 2019). In many practical applications, these choices are system specific and determined in consultation with stakeholders.
If the forcing variables e t are deterministic or follow well-characterized probability distributions, then the optimal policy π can be found subject to several modeling assumptions depending on the solution method. This is generally true even if e t represents a nonstationary process. Optimal control problems have long been studied in other areas of water resources, particularly reservoir operations (e.g., Castelletti et al., 2008;Labadie, 2004;Yakowitz, 1982;Yeh, 1985). The shorter timescale of the operations problem (hourly to seasonal decisions) allows quantification of forcing uncertainty through a combination of hydrologic forecasts and climatology and justifies neglecting endogenous uncertainty in the human-environmental system. However, because climate adaptation implies control of an open system decades into the future, it is not possible for a modeled representation of e t to fully encompass all sources of uncertainty. This disconnect between a mathematical formulation apparently well suited to a dynamic decision problem and the intractability of satisfying its key assumptions drives much of the discussion in this paper.

Solution Methods
Numerical methods used to solve dynamic planning problems in the water resources field generally fall into three categories: open loop, dynamic programming, and policy search, all of which will identify the optimal policy subject to several modeling assumptions. The first and simplest of these, open loop control, directly optimizes the sequence of actions over the time horizon, a t = π(t). The actions are based only on time and are not updated as a function of new observations of states or forcing variables. Open loop problems can therefore be solved with any type of optimization method, including linear or nonlinear programming, heuristics, or if the action space is small enough, by enumeration.
Both dynamic programming and policy search methods are closed loop approaches in which decisions are adjusted based on observed conditions. Dynamic programming approaches have been applied extensively in the water resources literature, especially for short-term operation problems and also for long-term planning and capacity expansion; Yakowitz (1982) provides an early review. The most common variant is SDP, in which the value function Q for each state at time t can be found from the recursive Bellman equation (Bellman, 1956): where γ is a discount factor. Then the optimal policy can be found by minimizing the Q-function: The problem is typically discretized to be solved numerically, meaning that the optimal policy is limited by the precision of the state, control, and forcing variables. For example, Figure 2 shows three discretization levels x 1 1 ; x 2 1 ; x 3 The dynamic programming family of methods includes a number of approximate dynamic programming approaches, one of which is model predictive control (MPC) (Bertsekas, 2005). In an MPC approach, the sequence of actions is optimized on a finite rolling horizon, which is repeated at each time step. This implicitly results in closed loop control, because new information about the system state is included at time t +1 based on the outcome of the optimized decision and the realization of the stochastic forcing e t during the time step [t,t+1).
By contrast, a policy search approach assumes a specific structure for the function π(·) with parameters θ such that a t = π(I t , θ). The optimization problem then becomes The result is a parameterized function mapping observations to actions, where the parameters of the function are the decision variables to be optimized. In this case, the optimal policy is limited by the type of function chosen and by the numerical convergence of the optimization. Figure 2 shows a policy structured as a neural network to represent an arbitrary function. Many function types have been explored in the water resources literature, largely in the context of short-term operations, ranging from linear decision rules (e.g., Oliveira & Loucks, 1997) to neural networks (Raman & Chandramouli, 1996), radial basis functions (Giuliani et al., 2014;Quinn, Reed, & Keller, 2017), and binary trees . The relationship between the policy parameters and the objective function(s) may be multimodal or discontinuous, complicating the use of gradient-based techniques. As a result, heuristic methods such as evolutionary algorithms have been widely used to support policy search Nicklow et al., 2010;Reed et al., 2013).
These methods span the fields of optimal control and reinforcement learning, which share the goal of identifying a state-based policy by which an agent, or decision-maker, determines actions through time. Of the methods discussed above, DP methods are drawn from optimal control, while policy search aligns more with reinforcement learning (Busoniu et al., 2010;Recht, 2019), which has had some application in the water resources literature (e.g., Castelletti et al., 2010). Apart from differences in terminology, a key distinction between the two is that DP methods search an approximation of the cost function to find the optimal policy, while reinforcement learning searches the exact cost function (e.g., a simulation model) for an approximation of the optimal policy (Bertsekas, 2019). In this paper, we use the term control to refer broadly to dynamic decision problems, recognizing that policy search methods widely used in the water resources field may also be considered reinforcement learning approaches. robustness testing. The policy structure includes the indicators that are observed over time, the actions to be implemented, and the decision variables that are being optimized to define the policy. The uncertainty characterization includes the sources of uncertainty that are considered and how they are represented in the optimization problem. Table 1 uses these categories to classify recent climate adaptation studies employing a dynamic planning approach. Papers were selected based on the authors' knowledge of studies applying the three criteria for dynamic planning stated in section 1: (1) a dynamical system, (2) multistage or continuous decision making, and (3) the development of a control policy as a function of system states, indicator variables, and/or time.
We classify these studies individually rather than by framework to highlight differences between experimental components that may change even between studies employing the same framework. Several of the studies are focused on planning under deep uncertainty in general, rather than the specific question of climate change; however, the papers selected here include at least some representation of long-term climate uncertainty, even if not drawn directly from climate models. Finally, the references cited in this section are not restricted to the papers in Table 1, as a number of studies in related areas provide relevant discussion despite not performing dynamic planning directly.

Problem Formulation 3.1.1. Actions
Among the studies in Table 1, adaptation actions are represented either as a set of discrete choices or with the opportunity to optimize the magnitude of implementation as a continuous variable. While the climate adaptation problem is often framed in terms of infrastructure decisions, the success of these plans also depends on the operating rules governing any existing or new infrastructure. Operational changes are generally less costly and more flexible, as they can be reversed unlike most infrastructure investments (Raso et al., 2018). Several studies have explored the range of scenarios over which operations can be adapted before new infrastructure investments would be required (Culley et al., 2016;Giuliani, Anghileri, et al., 2016;Whateley et al., 2014), while others have addressed the more complex question of jointly optimizing infrastructure and operations (Bertoni et al., 2019;Mortazavi-Naeini et al., 2015). Holding infrastructure fixed, the choice of operating rules may impact system performance as much as hydrologic conditions (Tian et al., 2018). Additional noninfrastructure actions to mitigate climate variability include financial instruments and public policy measures such as drought conservation, which have been included alongside infrastructure decisions in several studies (e.g., Trindade et al., 2017;Zeff et al., 2016).
The irreversibility of infrastructure actions presents substantial challenges for the climate adaptation problem. The indicator variables used in the optimized policy must provide reliable information about current and future change or else risk overfitting adaptation triggers to the forcing scenarios or distributions chosen. This problem can be quantified in terms of false positives (investments that ultimately prove unnecessary) and false negatives (failure to adapt), which relates to the choice of indicator variables (Raso et al., 2019;Robinson & Herman, 2019;Rosner et al., 2014). For example, determining an adaptation based on the 50year moving average of annual streamflow is more likely to result in false negatives, whereas adapting based on the 10-year average will likely result in a higher rate of false positives. Decision-maker preference along this trade-off could be reflected in the objective function or through multiobjective optimization, including the costs of switching between actions . To reduce the risk of overadaptation, large infrastructure options might instead be treated incrementally, accepting increased marginal cost in exchange for increased flexibility, a key concept in Engineering Options Analysis (de Neufville & Smet, 2019;Jeuland & Whittington, 2014).

Indicator Variables
Indicator variables are the observations used to trigger actions, some of which will be more informative for adaptation than others (Groves et al., 2015;Haasnoot et al., 2015Haasnoot et al., , 2018. Broadly, there are many different climate and environmental indicators that might provide information about the trajectory of climate change and its impacts on a system (Kenney et al., 2018). In general, an indicator represents a combination of a variable (precipitation and temperature) observed over a certain timescale (annual, monthly, daily, and hourly) and aggregated over a moving window (5, 10, and 30 years) using a statistical transformation (mean, variance, and quantile). For example, a 30-year moving average of annual reservoir inflow may be a useful indicator for water supply adaptation; for flood risk, a 50-year estimate of the 99th percentile daily streamflow 10.1029/2019WR025502  Table 1 use long-term hydroclimatic indicators to trigger infrastructure actions (Fletcher et al., 2019;Hui et al., 2018;Kwakkel et al., 2015;Trindade et al., 2017;Zeff et al., 2016). Others rely on short-term indicators such as reservoir storage to trigger operational actions, which may be adapted over time as a response to climate change (Mortazavi-Naeini et al., 2015;Paton et al., 2014).

Implementation Decisions
Control methods aim to optimize the policy mapping indicators to actions, thus determining the optimal magnitude, timing, and sequence of actions in response to the evolution of the system. Studies in Table 1 using control approaches therefore account for all of these implementation decisions (Fletcher et al., 2019;Hui et al., 2018). Other studies select a subset of these aspects to optimize. When the sequence of actions is optimized, it is typically assumed that candidate actions are reviewed on a fixed time step (e.g., every 5 or 10 years) (Beh et al., 2015;Kwakkel et al., 2015;Mortazavi-Naeini et al., 2014). Other studies directly optimize the timing and magnitude of actions to be implemented , while still others optimize observable threshold values to be used as triggers for implementation of particular actions (Mortazavi-Naeini et al., 2015;Zeff et al., 2016).

Limitations
Distilling a control problem formulation from a real-world planning context requires several key simplifying assumptions. The formulation posed here assumes centralized planning, where a single decision-maker controls the full set of candidate actions, which is often unrealistic in a real-world planning process . The formulation may require iterative input from different sets of stakeholders (Quinn, Reed, Giuliani, et al., 2017;Wu et al., 2016) and could be revised to represent a decentralized process in which multiple agents optimize for their individual benefits (Jenkins et al., 2017). Additionally, where political realities may prevent an optimization approach-for example, due to lack of agreement over which objectives or scenarios to include (Hall & Borgomeo, 2013;Kasprzyk et al., 2015)-other approaches may prove useful, including simulation-driven scenario exploration Kingsborough et al., 2016;Lempert, 2002;Thacker et al., 2018), negotiation theory (Islam & Susskind, 2018), and game theory (Madani & Lund, 2011;Sechi et al., 2013). While the optimal control framing implies that climate adaptation problems can be solved definitively, this is not the case in practice (Kasprzyk et al., 2018) as even the most advanced optimization approaches can only identify candidate solutions to be analyzed further.

Uncertainty Characterization
Dynamic planning methods must identify a set of possible scenarios and models (whether physical or statistical) to quantify uncertainty, either as a probability distribution or an ensemble of realizations. The majority of studies in Table 1 have represented uncertainty with an ensemble of synthetic scenarios describing weather and/or streamflow, using one of two approaches: 1. Using historical observations to parameterize a stationary stochastic process, which is then perturbed according to statistics from GCM projections (e.g., Borgomeo et al., 2016;Culley et al., 2016;Haasnoot et al., 2015). 2. Using GCM projections to directly parameterize a nonstationary stochastic process (e.g., Fletcher et al., 2019)).
The structural and parametric uncertainties in either the stationary or nonstationary case usually focus on exogenous hydroclimate and also apply to endogenous uncertainties. Synthetically generated scenarios provide a computationally efficient alternative to the direct use of downscaled GCMs and allow an arbitrarily large number of scenarios that are similar to, but not limited by, variability in the observed record. While this is often done without explicit probabilistic representations, it is worthwhile to question whether it is possible to entirely avoid the concept of probability when comparing planning alternatives. All characterizations of uncertainty require a distribution to be specified, whether implicitly or explicitly, and any comparison of alternative policies based on metrics computed across an ensemble requires scenario weighting. Computational methods targeting the challenge of deep uncertainty face the contradiction of needing to sample variables from distributions that are by definition unquantifiable.
Here we review three primary sources of uncertainty and the extent to which they have been included in the dynamic planning studies in Table 1: 1. Sampling uncertainty, which represents natural variability in forcing that may not be fully captured in the historical or projected record. 2. Uncertainty in exogenous hydroclimate change, which encompasses the chain of physical and statistical models used to create downscaled streamflow projections. 3. Uncertainty in endogenous system dynamics arising from human behavior and environmental processes.
While sampling uncertainty is aleatory (i.e., occurs due to random variations in the variable of interest), the latter two sources of uncertainty contain both aleatory and epistemic components, which arise from lack of knowledge (Beven, 2016).

Sampling Uncertainty
Even assuming a stationary climate, long-term water resources planning has always been challenged by sampling uncertainty (or internal variability). There are few historical observations of the extreme flood and drought events that drive water resources planning and even fewer that could point to a long-term trend to trigger adaptation. In the context of a control problem, sampling uncertainty arises in two key places: (1) inferring parameters of the stochastic process from a finite sample of either historical observations or GCM projections and (2) training and testing an adaptation policy on a finite record (observed or synthetic), where the combination of the number of scenarios and the planning horizon represents the sample size. In the first case, an insufficient sample size will result in a poorly characterized distribution of scenarios, which no amount of sampling can overcome, although parameter uncertainty can be estimated and included in the stochastic generation process (Stedinger & Taylor, 1982). In the second case, an insufficient sample size will result in overfitting policies to the events in the observed or synthetic record and an inability of the policy to generalize to other scenarios.
These issues are especially of concern when performance metrics are driven by extreme events, such as a high or low percentile of the output distribution (e.g., Herman et al., 2014;Quinn et al., 2018) which are difficult to estimate from a small sample. The majority of studies in Table 1 aim to reduce the effects of sampling uncertainty using ensembles of synthetically generated streamflow scenarios in the optimization and/or the validation step. Notably, the sample sizes vary significantly between studies, which is partly a function of different application contexts and also suggests a lack of consensus on experimental design.

Uncertainty in Exogenous Hydroclimate Change
A cascade of structural and parametric uncertainties propagates through the modeling chain used to develop climate change scenario projections for water resources systems (Kundzewicz et al., 2018;Wilby & Dessai, 2010). This includes uncertainty from the GCMs themselves and also the following: 1. Emissions scenarios used to drive the GCMs (Lamontagne et al., 2018). 2. Downscaling approaches used to tailor GCM output for local/regional assessment (Chen et al., 2011;Pierce et al., 2014). 3. Hydrologic models used to convert precipitation and temperature projections into streamflow (Broderick et al., 2019;Fowler et al., 2018;Mendoza et al., 2016;Prudhomme et al., 2014;Steinschneider et al., 2012).
We consider these elements exogenous to the basin-scale water resources planning problem, though some feedback to the regional hydrologic system may occur. Given that GCM projections are, at present, the best available source of dynamically evolving future scenarios to support water resources planning, any or all of these uncertain factors may need to be represented in an optimal control problem.
Among these factors, climate models and downscaling approaches consistently contribute the largest uncertainty in hydrologic projections due to the high variance in projected precipitation (Steinschneider, Wi, et al., 2015;Vetter et al., 2017;Wilby & Harris, 2006). One major source of climate model uncertainty arises because the processes that govern precipitation (e.g., local convection, cloud formation) occur at spatial scales that are substantially smaller than a GCM grid cell (Randall et al., 2003). Very high resolution models (<5 km) can improve some aspects of modeled precipitation (Prein et al., 2015) but with increased computational cost, which limits ensemble simulations needed for water resources studies. Model errors also have a large impact on projections of future large-scale atmospheric dynamics (Sigmond et al., 2010;Simpson et al., 2016), which interact with parameterized subgrid processes and orographic effects (Davini et al., 2017;Stevens & Bony, 2013). This dynamical error structure severely complicates efforts to bias correct and downscale GCM output based on simulations under a baseline period, as is commonly done for water resources impact studies (Ehret et al., 2012). Further, the suite of GCMs available across global institutions is not 10.1029/2019WR025502 Water Resources Research independent, as submodules are often shared across models , resulting in clustered projections of regional climate change that do not reflect added confidence (Shortridge & Zaitchik, 2018;Steinschneider, McCrary, et al., 2015).
To illustrate the exogenous uncertainty in hydroclimate projections, Figure 4 shows ensemble and sampling uncertainty for statistics of annual streamflow (5th and 50th percentiles) using example data for the Sacramento River, California. The choice of an annual timescale and lower percentile reflects a focus on water supply risk rather than floods. These plots compare the paleo record (Meko & Woodhouse, 2005), observed data, and downscaled CMIP5 projections containing multiple emissions scenarios (Reclamation, 2014). Comparison of Figures 4a and 4b suggests a few points. First, these statistics have always shown some variability, subject to the choice of a 50-year rolling window. The confidence intervals tend to be relatively large for the lower percentiles (5th) than for the median, reflecting uncertainty in estimates of extremes. Finally, sampling uncertainty makes up a nontrivial portion of the exogenous uncertainty in future projections, reflected by the light red shaded area. These data include multiple GCMs and emissions scenarios but only one hydrologic model and downscaling procedure, both of which may introduce additional biases in certain aspects of the flow regime.

Uncertainty in Endogenous Human-Environmental Dynamics
Another significant source of uncertainty is the endogenous dynamics of the human-environmental system under consideration, particularly given the long planning horizons involved in climate adaptation problems (Haddeland et al., 2014). In the context of the control problem shown in Figure 2, this uncertainty primarily appears in the state transition equation. It therefore adds another layer of model structural and parametric uncertainty beyond those contributed by climate and hydrologic models and introduces the need to test multiple plausible assumptions for the system simulation model. Uncertainty in human behavior has been identified as one of the major knowledge gaps in the field Vogel et al., 2015) and has been found to exceed the impact of climate uncertainty in a majority of studies that have compared their relative influence (Alcamo et al., 2007;Anghileri et al., 2018;Droogers et al., 2012;Fant et al., 2016;Vogel et al., 2011). Uncertainty in human-environmental systems is not limited to the climate adaptation problem; even under a stationary climate, the dynamics and long-term outcomes of social and economic behavior are indeterminate (Ben-Haim, 2012).
Several of the studies in Table 1 include some consideration of uncertainty in the human system, such as water demand (Erfani et al., 2018;Jeuland & Whittington, 2014)  Unlike the streamflow scenarios described above, these are generally implemented as scalar parameters to be sampled rather than time series; none of the studies reviewed here considered structural uncertainty in the state transition equation (or simulation model). Several endogenous feedbacks could be relevant for a water resources planning problem, including, but not limited to, the following: Indirect impacts of climate change. Elements of the system may respond to nonstationary precipitation and temperature in ways not directly linked to the choice of adaptation policy but which create second-order effects on the objectives. For example, agricultural yields and water demands will respond to rising temperatures, which may trigger land use changes (Jafino et al., 2019;Wada et al., 2013); energy supply and demand are also likely to change, which further influences water demand (Carleton & Hsiang, 2016). Additionally, changes in hydroclimatic forcing may alter the risk attitudes of decision-makers, such as actions taken to mitigate extreme events (Aghakouchak et al., 2014;Viglione et al., 2014).
Institutional uncertainties. A planning organization may not be able to achieve its intended implementation even once a decision has been made, for example, through delays or cost overruns (Grimsey & Lewis, 2002). Adaptation actions may therefore only be effective when the planning agency has the necessary institutional capacity (Rist et al., 2013;Tompkins & Adger, 2004). The model may also need to include adaptations to climate change occurring at other institutional scales that are outside the scope of the control problem under consideration (Adger et al., 2005;Gonzales & Ajami, 2017). Response to nonclimate drivers. The system may respond to other social drivers which may or may not be linked to climate change, such as migration, urbanization (Zhao, 2018), population growth, land use, and technology changes.
Feedbacks to the hydrologic system. Any of the above changes might also influence the regional hydrologic system though land use, water withdrawals, and infrastructure impacts on the streamflow regime (Shin et al., 2019), particularly on evapotranspiration and peak flow events.
Environmental dynamics. Long-term environmental changes in water quality, geomorphology, and ecological regimes have not received much consideration in dynamic planning studies, despite their susceptibility to change on decadal timescales as a result of either cumulative "slow" processes or tipping points (Scheffer et al., 2009;Walker et al., 2004). For example, a warmer climate will directly affect water quality and species habitat (Moyle et al., 2013), as will many of the infrastructure and operational adaptations undertaken by human institutions at different scales. On longer timescales, hydroclimatic change and human adaptations alter geomorphic processes (Kondolf & Podolak, 2014), further influencing habitat. All of this could lead to ecological regime shifts, with the caveat that severe environmental degradation may change human attitudes toward water management (Elshafei et al., 2014).

Solution Methods
A range of solution methods have been used to identify dynamic adaptation plans under climate change. SDP has been used for infrastructure planning in several studies, including levee height optimization with Bayesian updates of the flood risk distribution (Hui et al., 2018), as well as planning reservoir capacity expansion and investment in desalination plants (Fletcher et al., 2019). Several studies optimize the timing of actions directly, either using an open loop approach  or MPC in which the short-term plan is updated periodically (Beh et al., 2015(Beh et al., , 2017. Finally, a common policy search formulation involves optimizing thresholds of observed or projected variables to trigger actions within a predefined rule structure (Kwakkel et al., 2015;Zeff et al., 2016).
While the choice among solution methods is problem specific, there are a few general advantages and disadvantages of each approach. If a well-characterized model of the stochastic forcing variable(s) can be specified, SDP identifies the exact solution, subject to the discretization scheme, and often the assumption of forcing variables being uncorrelated in time. Given the difficulty of identifying probability distributions for forcing variables in the climate adaptation problem, this advantage may be limited-the ability to find

10.1029/2019WR025502
Water Resources Research exact (or even approximate) optimal solutions is less relevant when the problem formulation itself is uncertain (Jeuland & Whittington, 2014). Furthermore, the practical implementation of SDP faces limitations from the curse of dimensionality and the requirement of a mathematical model rather than numerical simulation (Bertsekas, 2019).
Open loop methods are computationally efficient but present a high risk of overfitting to the scenarios or distributions of forcing used in the optimization if not coupled with a validation scheme. These shortcomings could be abated by repeating the open loop optimization as new information becomes available, as in MPC. However, this approach does not identify adaptation rules in response to evolving conditions as in policy search and SDP. Policy search has the additional advantage of flexibly incorporating multiple objectives and multiple indicator variables, which can be included as state variables in SDP but at the cost of significant added computational complexity . However, policy search methods also introduce additional assumptions and challenges in the subjective choice of function type and number of parameters-a poor choice risks either insufficient flexibility in approximating the optimal policy if the function is too simple or overfitting to the training data if the function is too complex. Importantly, none of these approaches avoid the challenge of characterizing uncertainty in forcing variables and system dynamics.

Validation/Robustness
Finally, many of the studies in Table 1 conclude with some form of robustness assessment by simulating the performance of the optimized policy in a new set of scenarios. This idea borrows from robust planning methods in which alternatives generated through optimization are subjected to a wider range of uncertainty . Here we distinguish between testing the robustness of an adaptive policy to (1) more realizations from the same uncertainty characterization used in the optimization, versus (2) scenarios in which new uncertain variables are sampled, or the same variables are sampled from different distributions. The former only tests against sampling uncertainty and can be used to determine whether a policy is overfit to a particular set of scenario realizations. The latter approach could test robustness to other forms of uncertainty, provided that the sampling is informed by some knowledge of the ensemble or endogenous uncertainties. In either case, the ideal outcome is either minimal degradation relative to optimized performance or maintaining acceptable performance in a wide range of scenarios, as discussed in prior studies (Lempert & Collins, 2007).

Perspectives: Research Gaps and Opportunities
Based on the concepts reviewed in the previous section, we propose several research gaps and opportunities to improve the use of control methods for dynamic planning under climate change. These research gaps align with the subsections that follow: 1. Process-based insight for synthetic generation: Many studies characterize uncertainty in future scenarios based on coarse-timescale GCM statistics, such as annual precipitation. There are opportunities to leverage insight into climate processes and model errors to inform finer-scale uncertainty characterization in synthetic scenarios. 2. Uncertainty classification: Depending on the nature, level, and potential for learning of the uncertainties included, they may be treated differently in the experimental design. 3. Endogenous uncertainty: Relatively few planning studies have considered endogenous uncertainty; those that do typically only consider parametric rather than structural uncertainty. These uncertainties should be considered when relevant, especially feedbacks in response to adaptation actions. 4. Policy validation: There is currently not a unified approach to policy validation and robustness testing, such as whether this step should include more realizations or a different uncertainty characterization altogether and what sample size is adequate. 5. Computational complexity: More rigorous comparisons of solution methods could consider their efficiency and effectiveness, scalability as a function of the number of state or indicator variables, and tendency to overfit to training scenarios. 6. Indicator variables: Finally, there are significant opportunities to include more observed and projected information as indicator variables for the adaptation policy; only a few of the studies in Table 1 consider more than one indicator variable. This choice can be informed by several monitoring and data assimilation methods.

Process-Based Insight to Improve Uncertainty Characterization
The characterization of uncertainty in the dynamic planning problem can be informed by the physical causes of uncertainties in ensemble projections, for example, whether the projected changes are thermodynamic or dynamic in nature (Emori & Brown, 2005;Seager et al., 2010). Thermodynamic changes relate directly to the increased surface warming of the Earth under anthropogenic forcing and cause more frequent and intense temperature extremes, glacial retreat, reduced snowpack and earlier snowmelt, sea level rise, and the increased moisture holding capacity of the atmosphere (i.e., Clausius-Clapeyron scaling). These trends are consistent with theory and robust in both observations and model projections (Fischer & Knutti, 2016;Collins et al., 2015), leading to high confidence in their future direction, albeit with residual uncertainty in their magnitude. By contrast, dynamic climate change relates to changes in atmospheric circulation (e.g., jet stream dynamics, storm tracks, and seasonal monsoon progression), which play a large role in determining regional precipitation. Dynamic changes are significantly more uncertain than thermodynamic change (Pfahl et al., 2017;Shepherd, 2014;Woollings, 2010) and are difficult to distinguish from internal atmospheric variability especially on timescales (10-30 years) relevant to water resource investment decisions (Knutti & Sedláček, 2013). To illustrate the relative uncertainty in dynamic processes, Figure 5 shows example projections of differences in temperature (thermodynamic) and streamflow (both thermodynamic and dynamic change).
Leveraging such process-based insights, there is an opportunity for new synthetic scenario generation methods to create plausible dynamic projections of climate change for water resources planning studies. The key questions in generating nonstationary scenarios are as follows: what variables should be perturbed, according to what distributions, and along what transient trajectories? In principle, any parameter of a stochastic generator can be perturbed based on climate information, for example, streamflow seasonality (Nazemi et al., 2013;Prudhomme et al., 2010), interannual variance and persistence (Borgomeo et al., 2013;Quinn et al., 2018), or the frequency and severity of drought events . By contrast, stochastic weather generators are commonly used to alter daily weather characteristics, including the likelihood and persistence of wet and dry days, the intensity and seasonality of precipitation, its interannual persistence, and the magnitude and range of minimum and maximum temperature (Guo et al., 2018; Steinschneider Streamflow across the ensemble typically shows a wider spread of projected changes relative to temperature. Distributions are shown for different timescales and quantiles, indicating that streamflow projections tend to be most variable for droughts (annual, 1%) and floods (daily, 99%), which presents the most difficulty for adaptation planning. . These approaches must maintain realistic persistence and covariance structures across space and time and potentially between multiple variables (e.g., precipitation, temperature, and wind speed) (Allard & Bourotte, 2015;Kwon et al., 2009;Steinschneider & Brown, 2013;Verdin et al., 2018), which can be supported by GCM output.

Water Resources Research
The distribution of perturbations to apply is as important as the choice of variables to perturb. The range of stochastic scenarios can be informed by a process-level understanding of different types of climate change. For instance, hypothesized thermodynamic change can be tested by adjusting the tails of the distribution of local precipitation via a temperature-dependent Clausius-Clapeyron scaling. There are well-documented constraints on the thermodynamically driven increase in extreme precipitation, which is expected to mirror the increase in atmospheric moisture holding capacity (~7% C −1 ) (Fischer & Knutti, 2016;Trenberth, 2011) but at hourly timescales could increase faster due to latent heat release during intense precipitation that further enhances convection and precipitation rates (Bao et al., 2017;Guerreiro et al., 2018). Dynamic changes can be imposed by altering the frequency of different types of storm events (Knighton et al., 2017;Steinschneider et al., 2019), although these perturbations could be conditioned on climate model experiments designed to explore the consistency of such (inherently uncertain) signals against the backdrop of model and parameter uncertainty (e.g., multimodel and perturbed physics ensembles; see Glenis et al., 2015). This type of approach provides a promising avenue to better link stochastic scenario development with process-level insights inferred from GCM ensembles, which is especially important in the case of extreme events (e.g., Serinaldi & Kilsby, 2015).
These insights can be used in a variety of ways to generate plausible scenarios of future change and to distinguish these from implausible ones. Broadly, we propose the following recommendations: 1. A large variance in projected climate (particularly precipitation) should not immediately mark the ensemble as uninformative, as it likely represents the combination of internal variability and different thermodynamic and dynamical signals. 2. Conversely, in the case of climate model agreement-which does add some confidence of the direction of change-it may indicate a shared bias rather than an accurate prediction. That is, outlier projections cannot be disregarded either, as they may represent plausible future outcomes. 3. Closer coordination is needed with the climate science community, which dedicates significant effort to assessing the regionally specific suitability of different climate models to support local adaptation based on the fidelity of large-scale dynamics. If conducted carefully, expert elicitation of future likelihoods can help with these challenges (Dessai et al., 2018;Morgan, 2014;Refsgaard et al., 2006). 4. Relevant uncertainties can be qualitatively identified from the context of the planning problem. Table 2 shows an example, based on a handful of key studies that have examined the relative impacts of one or more of the uncertainties described above (Deser et al., 2012;Greve et al., 2018;Haddeland et al., 2014;Hawkins & Sutton, 2011). By contrast, only a few water resources planning studies have attempted to decompose the sources of uncertainty influencing decisions (e.g., Paton et al., 2013;Schlef et al., 2018). Note. Several of these classifications are subjective choices; in general, all of these uncertainties apply to any climate adaptation planning problem. Here we highlight those that likely dominate the total uncertainty in different planning contexts. The addition symbol indicates only that the uncertainty should be considered; it does not imply that the uncertainties or their effects are additive. The bold items were meant to show the type of uncertainty added in each case. (Some cases do not have any uncertainty added).

Uncertainty Classification to Inform Experimental Design
If the problem context yields physically based insights to help identify relevant uncertainties, it may also inform how they are treated in a dynamic planning study. This may point to appropriate optimization methods, indicator variables, and monitoring efforts to support planning, as well as sources of uncertainty that do not need to be considered. The dominant sources of uncertainty vary widely between studies, regions, and scales and are thus highly specific to the context of the planning problem. An important first step is therefore to identify the sources of uncertainty that most strongly influence the choice and timing of actions, possibly using some form of sensitivity analysis (Saltelli et al., 2004) and to devise methods to reduce these uncertainties, select actions that are robust to a wide range of outcomes, and/or prioritize responses to impacts of climate change that are currently better understood.
Uncertainties may be treated differently in the experimental design depending on their nature, level, and potential for learning (Döll & Romero-Lankao, 2017;Fletcher et al., 2017;Kwakkel et al., 2010). The nature of each uncertainty is either aleatory-irreducible uncertainty due to random variations in the variable of interest-or epistemic, arising from a lack of knowledge about the nature of the process of interest and how it should be modeled (Beven, 2016). The level of uncertainty refers to whether a variable can be well characterized by probability distributions or not (Döll & Romero-Lankao, 2017). Many uncertainties in projections of regional climate change can be considered deep, where an exact quantification is not possible (Spence & Brown, 2018). Table 3 shows an example classification, recognizing that this type of classification in practice is specific to individual case studies. Relative contributions to total uncertainty in climate change assessments vary widely across space and time Vetter et al., 2017), and the columns in Table 3 add another layer of subjectivity that prevents a generalizable classification. However, if this exercise can be performed as part of a planning study, it can provide a foundation for the experimental design.
Different signals of climate change might then be treated differently in the control framework. For instance, certain impacts of thermodynamic climate change have already been observed and are projected with high confidence to continue (e.g., reduced snowpack, rising sea level, and more intense storms), though with some uncertainty in their magnitude. Because emissions uncertainty has a high potential for learning over time, infrastructure expansion can be staged in response via endogenous learning, potentially using emissions observations as an indicator variable. Adaptation of water supply and flood control systems to dynamic climate change (and associated shifts in regional precipitation) is more difficult, since the direction and plausible magnitudes of change are often poorly understood. In addition, the large degree of internal, multidecadal climate variability poses a significant barrier to endogenous learning of emergent trends (Doss-Gollin et al., 2019). This emphasizes the need for a solution that is either robust or reversible, acknowledging the significant potential to have an incorrect uncertainty characterization in the optimization problem and/or in the postoptimization test set. Recognizing that some probabilistic assumptions are inevitable, they should be made explicit and justified in the problem formulation (Beven, Almeida, et al., 2018;. It may be more informative to determine the sensitivity of the optimized policies to the choice of uncertainty characterization, rather than only validating against more realizations of the same uncertainty characterization. This partially depends on the decision framework being used (Brekke et al., 2009;Refsgaard et al., 2013) but in general may reveal weaknesses or sensitivities of the decision-making framework to new or surprising information that would not be apparent otherwise.

Uncertainty in Endogenous Human-Environmental Model Structure
Unlike some forms of parametric or exogenous uncertainty, the uncertainty in endogenous model structure is not easily sampled or characterized. However, the sensitivity of optimized policies to alternative structural assumptions about how humans respond to climate change can be assessed, similar to their sensitivity to alternative distributions of climate forcing. For example, different structures and feedbacks can be tested as discrete hypotheses to determine if they are influential relative to the many other uncertainties discussed previously, in terms of one or more system performance metrics. Table 3 An Example Classification of Uncertainties According to Their Nature, Level, and Potential for Learning, Following Kwakkel et al. (2010), Döll and Romero-Lankao (2017), and Fletcher et al. (2017)

Water Resources Research
Capturing all of these multiscale dynamics in a simulation model remains difficult, but sensitivity testing can help prioritize their inclusion. Global-scale earth system models have made significant advances in this direction (Pokhrel et al., 2016;Wada et al., 2017), as have local-to-regional scale models in recent years (Konar et al., 2019). In practice, however, regional socioeconomic scenarios are often considered exogenous factors in planning studies, which may inadequately capture dynamic relationships between these and climate forcing (Verburg et al., 2016). Accounting for these feedbacks and dynamic preferences is key to any retrospective assessment of human behaviors or descriptive modeling of future system evolution (Di Baldassarre et al., 2017;Mason et al., 2018).
Reducing this uncertainty requires better scientific understanding of human-environment interactions, including the representative feedbacks described in section 3.2.3. While the consideration of human and ecological water needs has always been a part of the systems analysis field, focus has only recently shifted to the question of how to model endogenous system dynamics that are driven by the coevolution of hydroclimatic forcing and human behavior (Sivapalan & Blöschl, 2015;Thompson et al., 2013). This coupled modeling has been addressed in several ways: 1. Hydroeconomic models: water demand is a function of availability, assuming rationality (Draper et al., 2003;Kahil et al., 2018) 2. Descriptive models: infer behavioral rules from observational data or theory, originating in cognitive psychology and the social sciences (Camerer et al., 2004;Sanderson et al., 2017) or directly from observational data Turner et al., 2019) 3. Dynamical systems models: a set of differential equations, including socioecological systems (Anderies et al., 2004) and sociohydrology (Di Baldassarre et al., 2016;Sivapalan et al., 2014). 4. Agent-based models: rule-based individual actions, for example, in response to short-and long-term water scarcity conditions (Giuliani, Li, et al., 2016;Schlüter & Pahl-Wostl, 2007).
Opportunities exist to incorporate findings from recent studies that empirically measure human responses to climate change into simulation models using similar methods (e.g., Carleton & Hsiang, 2016). Moreover, the increasing availability of large observational data sets describing human activities (Sanderson et al., 2002) creates new possibilities for advancing data-driven behavioral modeling (Cominola et al., 2018(Cominola et al., , 2019. The extent to which dynamic simulation models can provide a reliable and unbiased representation of human behavior remains an important research question (Melsen et al., 2018).

Policy Validation and Robustness
One strategy to address the impacts of deep uncertainty in the control problem is testing optimized policies on scenarios or probability distributions other than those used in the optimization. The key question is not whether an adaptive policy can be optimized to a nonstationary climate scenario (it can) but whether the sequence, timing, and magnitude of actions in this optimized policy can generalize to other plausible realizations of climate and other uncertainties. The concept of testing optimized decisions has been explored extensively in the robust decision-making literature (e.g., Kasprzyk et al., 2013), as well as several of the dynamic planning studies in Table 1.
Here we distinguish two related goals of such testing (Figure 6a): (1) robustness to sampling uncertainty, represented by more realizations using the same uncertainty characterization-which we refer to as validation, in the machine learning sense-and (2) robustness to other uncertain variables or probability distributions not considered in the optimization. The first goal investigates whether the adaptation policy is overfit to the particular scenarios used in the optimization, especially given the strong dependence of control methods to the sampling of uncertainties over time. The second goal tests whether the adaptation policy is sensitive to key assumptions in the uncertainty characterization. The choice of which uncertainties to include in the optimization versus testing is highly subjective, but it is generally difficult to optimize to all possible sources of deep uncertainty, which would risk overdesign unless the uncertainties are included as indicator variables in the policy optimization.
As shown in Figure 6b, a few benchmarks in the validation step can help provide context for the optimization results and suggest improvements in the experimental design. Specifically, each validation scenario or set of scenarios can be evaluated with a few measures of system performance: (1) the optimized policy reevaluated in this scenario (validation); (2) a policy specifically optimized to this scenario, to establish an ideal 10.1029/2019WR025502

Water Resources Research
outcome (perfect foresight); and (3) a "no-action" scenario, to establish baseline system performance if no adaptations are implemented. All of these are compared to a target level of performance or possibly multiobjective targets, determined in consultation with stakeholders and decision-makers. The possible outcomes are then: 1. Meets requirement: the performance in validation exceeds the target performance. 2. Overfit: performance in optimization exceeds the target but validation does not. The optimization should include a larger sample size of scenarios, with attention paid to extreme events. 3. Action-constrained: even with perfect foresight, the target cannot be met. This suggests that the set of actions is too limited to adapt to future change. 4. Information-constrained: optimized performance does not meet the target, but perfect foresight does, suggesting that better indicator variables could improve the policy. 5. No adaptation needed: The target is met even if no action is taken, which might be the case if the current system is already robust to the range of projected future changes.
The outcomes shown in Figure 6b assume that the policy performs better in optimization and validation than no action, which is not guaranteed in practice. Additionally, comparing the actions taken by the optimized policies to those obtained from the perfect foresight optimization would show whether the actions chosen (sequence, timing, and magnitude) remain roughly the same, which could serve as a diagnostic step for policies performing poorly in validation. These experiments can be repeated on several types of scenarios (e.g., wet vs. dry) to investigate whether a similar series of optimal actions, or conditions to trigger them, might be reasonable even without exact knowledge of the future climate. Performing these comparisons in the decision space and objective space could also provide a basis to distinguish policies which show similar performance yet very different actions, that is, the case of equifinality in the optimization results.
Finally, in the robustness step a number of additional uncertainties can be considered, including model structure, variables, and distributions that were not modified in the validation step. The focus is not necessarily choosing the most robust policy, which likely comes at high cost. Instead, as with robust planning methods, the goal is to understand the sensitivity of the optimized policy to the key assumptions regarding deeply uncertain variables. In the dynamic planning problem this challenge is exacerbated by the need to represent deeply uncertain variables as time series, which contain nearly infinite possible sequences of events over a decades-long planning horizon and are therefore unlikely to be fully represented by an

10.1029/2019WR025502
Water Resources Research ensemble of any size. Still, this process can identify key assumptions for refinement, iteratively informing the optimization step if policies are found to be overly sensitive.

Computational Complexity
The choice of solution method is arguably not the most pressing issue in dynamic planning studies under climate change, because all methods face a number of other challenges discussed previously. However, several important research questions remain related to the computational complexity of using these methods to design and test optimal adaptation policies. The concepts presented in Figure 7 draw some inspiration from prior diagnostic studies of optimization algorithms (e.g., Reed et al., 2013;Zatarain Salazar et al., 2016), here with a specific focus on the dynamic planning problem: 1. Efficiency/effectiveness: For a given level of performance (possibly multiobjective), which method has the best runtime and vice versa? These experiments assume a fixed problem formulation and possibly a benchmark level of performance determined with perfect foresight. Heuristic methods will require multiple random trials. 2. Scalability with problem complexity: How does the runtime needed to converge (e.g., to a predefined acceptable level of performance) increase with the number of indicator variables? This type of analysis would reflect the curse of dimensionality in the dynamic programming family of methods and would confirm whether policy search methods can flexibly include more indicator variables at the cost of an approximate policy. Here the number of indicator variables is assumed to be a proxy for the complexity of the problem, since it will increase the size of the search space roughly exponentially depending on the policy structure. There may be opportunities to reduce the computational effort through aggregate proxy indicators that combine multiple sources of information while maintaining key dynamic signals (Zaniolo et al., 2018). 3. Overfitting to training scenarios: As the problem complexity increases, the resulting policy should improve in validation, to a point. After that, the more complex policy will not generalize well to other scenarios. This is not so much a comparison between methods as an important threshold to identify within each method. The point at which overfitting occurs also depends on the number (or length) of scenarios used in training, because more complex policies will require more training data to avoid overfitting. If policy interpretability is a concern, it is possible that the level of desired complexity will occur well before overfitting.
These experiments could be performed on a single case study but would be stronger if devised within a generalizable testing framework where the properties of the problem can be modified along with the properties of the solution methods. The results would then indicate which solution methods are most applicable to certain problem contexts and/or the extent to which the problem formulation would need to be modified in order to use each method.

Monitoring and Data Assimilation
Finally, the studies in Table 1 suggest significant potential to expand the set of indicator variables used to trigger adaptation, particularly using policy search approaches. Indicators must be chosen to ensure they are indicative of emerging trends and not merely noise (Haasnoot et al., 2018). We discuss three statistical approaches that might improve the use of information in dynamic adaptation to climate change: trend

10.1029/2019WR025502
Water Resources Research detection, Bayesian data assimilation, and formal approaches to indicator variable selection (Figure 8). While these approaches apply equally to hydroclimatic variables and endogenous variables, such as water demand and land use, most of the water resources literature focuses on the former.

Trend Detection
In the context of climate change, trend detection typically refers to the challenge of distinguishing nonstationarity from natural variability in an observation of interest (Hegerl & Zwiers, 2011). While a statistically significant trend is not a requirement for adaptation to occur, it is one of many possible indicator variables that could be used to trigger adaptation, including future values predicted by extrapolating a trend. Detecting trends in the impacts of thermodynamic climate change, such as snowpack decline and sea level rise (Ceres et al., 2017;Thorarinsdottir et al., 2017), is often more feasible than for dynamic climate changes and extreme events such as floods. In this case, a key question is how much information will need to be observed before a significant trend can be detected, which for precipitation may exceed relevant planning timescales of 50 years or more . The length of observations needed to detect a trend could inform the choice of moving window over which an indicator variable is aggregated. Several related studies have focused on the development of nonstationary hazard functions to characterize the frequency and severity of extreme events, particularly floods (Luke et al., 2017;Read & Vogel, 2015) estimates of which are updated through time based on observed precipitation trends. These methods are primarily concerned with failure to detect change (Type II error) (Rosner et al., 2014;Yu et al., 2015), which relates to false negatives for infrastructure adaptation.
Trend detection could support the development of indicator variables for dynamic planning in several ways. First, a trend could serve as a binary indicator variable for the policy, to trigger an action when statistical significance is detected. This would implicitly try to minimize false positives for infrastructure planning or to trigger action based on the expected costs of a false negative (Rosner et al., 2014). These methods could also be used to detect when the current or projected future scenario is trending outside the range of the scenario(s) against which the current policy was trained. Additionally, stochastic scenario generation enables controlled experiments to test adaptive policies based on trend detection methods. For instance, transient scenarios can be generated to combine physically based trends using parameterized representations of thermodynamic and dynamic climate change alongside spurious trends using high autocorrelation to test if a policy overadapts or underadapts to multidecadal internal climate variability and secular climate change. Such efforts provide a promising avenue to further advance adaptive control policies with an enhanced process-level understanding of regional climate variability and change.

Bayesian Inference
Observations of hydroclimatic and other variables that occur during the planning horizon can condition the characterization of stochastic forcing used in the control problem. For example, new observations might eliminate some emissions trajectories, refine estimates of climate sensitivity, or assimilate new GCM ensembles. An adaptive policy can be designed to incorporate these updates; for studies using formal probabilistic approaches, this could be done with Bayesian methods. As discussed previously, many of the uncertainties in long-term planning are not readily described by probability distributions. Bayesian methods still require probabilistic formulations just as scenario-based methods require scenario selection. However, the choice of prior distribution can be tailored to the information available, ranging from precise, data-driven priors to uninformative priors chosen to allow data collected over time to be the primary driver of the resulting Water Resources Research posterior. Further, the extent to which priors are truly uninformative can be tested via sensitivity on the priors (Gelman et al., 2013). In either case, prior estimates can be updated dynamically with observations, reducing uncertainty as more information becomes available and informing adaptive planning.
In the climate science field, Bayesian methods have been employed to reconcile simulation model output with observed data (e.g., Smith et al., 2009;Tebaldi & Knutti, 2007), building on a foundation of applications in water resources and environmental sciences (Bates et al., 2003;Hobbs, 1997;Hong et al., 2005). For optimal control problems, the formal probabilistic treatment of Bayesian methods lends itself to SDP approaches (Fletcher et al., 2019;Hui et al., 2018) and can also be accommodated within policy search methods by probabilistically weighting performance under different scenarios or by conditioning probabilistic indicator variables. In addition to forcing variables, such approaches can also be used to infer system parameter values and tipping points (Singh et al., 2018) to reduce endogenous uncertainty.
The idea of characterizing stochastic forcing based on dynamic future observations suggests that one could instead describe the nonstationary scenarios by means of stochastic models with state-dependent parameters (Priestley, 1988), as long as the parameters are changing slowly relative to the dynamics of the system under study (Young et al., 2001). Following this approach, the parameters of the stochastic forcing models would be defined as a function of other observable variables in the system that can be sampled over time (Young, 2000). The parameters can then be recursively estimated using the Kalman filter or associated algorithms (Kalman, 1960). Thus, as changes in the state are observed, changes in the parameters in the stochastic process change in response. The state space representation of the parameters of the stochastic forcing models makes this approach particularly suitable for the design of closed loop control policies (Taylor et al., 2000).

Indicator Variable Selection
Indicator variables should be a parsimonious subset of forcing and state variables that can effectively inform policy actions. To some extent, this choice can be initialized by analyst judgment. However, since the set of candidate indicator variables and their statistical transformations is technically infinite, comprising many redundant variables and their transformations, the process may benefit from formal techniques for Input Variable Selection (IVS) (Galelli et al., 2014;Guyon & Elisseeff, 2003). These methods provide flexibility to capture nonlinear interactions between input variables and the computational efficiency to handle potentially many candidate variables over long time series. Generally, IVS problems arise every time a variable of interest is modeled as a function of a subset of potential explanatory variables, or predictors, but there is uncertainty about which subset to use (George & Foster, 2000).
To the authors' knowledge, IVS methods to support the optimization of climate adaptation policies are still unexplored, but they can draw inspiration from studies of short-term reservoir operations that have dealt with the topic in more detail. For example, extending the framework proposed by Giuliani, Pianosi, et al. (2015), an IVS procedure for climate adaptation might involve the following steps: 1. Assume a single future scenario as truth and solve a deterministic optimization problem with perfect foresight, yielding an ideal reference solution. 2. Find the minimum subset of indicator variables that, when used to optimize a policy, best approximates the sequence of optimal adaptive decisions from the perfect foresight solution. 3. Once the best subset of indicator variables is identified, they can be used in the optimization of the adaptive policy. 4. Iterate multiple times using different reference scenarios to avoid overfitting, ideally identifying a common set of indicator variables selected across a wide range of scenarios.
The effectiveness of such an approach for a long-term climate adaptation problem remains an open question, along with several more general questions. For example, it is not clear whether there is an advantage to choosing a climate indicator or a second-order variable that is strongly correlated but may be easier to measure, such as reservoir storage. Indicator variables may need to be adjusted dynamically after an infrastructure adaptation is triggered by the policy, and therefore, the system behavior is changed. Also, nonstationary indicator variables may undergo different types of change, such as a step change rather than a gradual transient signal. The choice of indicators is strongly linked to the objective function; in multiobjective problems, a complex combination of information may be needed, a challenge that applies to both operations and planning (Quinn et al., 2019).

Conclusions
While optimal control methods cannot directly solve the climate adaptation problem any more than other public policy problems Rittel & Webber, 1973), they are still a valuable component in decision support. It remains the primary approach to frame a dynamic planning problem in which actions are taken in response to observed and projected states and fluxes and provides a useful way to define and classify recent studies in this area. The past decade of research suggests that dynamic planning has become a candidate successor to the stationary paradigm of water management (Milly et al., 2008) because of its ability to identify and adapt to nonstationary trends and also to navigate the numerous and interacting sources of uncertainty in long-term climate projections.
Going forward, we propose the following summary points to support the evolving science and practice of dynamic water resources planning under climate change: Water Resources Systems Sciences: 1. The purpose of optimal control for long-term planning is not necessarily to implement the policy directly (unlike short-term operations) but rather to provide decision support by identifying near-term plans that can best prepare the system for the long-term future. Policies will eventually be updated as new observations and projections become available. 2. Uncertainty in endogenous system dynamics may equal or exceed that contributed by climate change on long planning horizons. Dynamic planning would therefore benefit from an improved understanding of the nonlinear dynamics linking climate and hydrology with human behavior, including land use and water demand across multiple scales. As noted by Brown et al. (2015), the traditional prescriptive focus of the field cannot be separated from these descriptive questions of how water resources systems will evolve in the presence or absence of policy interventions. 3. No uncertainty characterization can be proven correct but can be justified according to the timescale, variable, and time horizon of the problem. An optimized adaptation policy implicitly reflects the probabilities of events that it was trained against, and how the objective function is aggregated, whether or not explicit probability distributions are defined. 4. Given that any future projection will not occur exactly, optimal control methods should employ sensitivity analyses to identify: (1) the sensitivity of the system to structural and parametric uncertainties throughout the modeling chain and (2) the sensitivity of an optimized policy to the approach used for uncertainty characterization.
Climate and Hydrologic Sciences: 1. The control problem requires dynamic sequences of hydroclimatic inputs which are physically plausible across timescales. GCM projections can inform these dynamic sequences, despite their known limitations in resolving precipitation processes. Stochastic weather and streamflow generators are rapidly improving and may be able to leverage physically based insights from the climate modeling field. 2. Ideally, ensemble projections made available to planners could include a broader set of uncertainties beyond GCM and emissions scenarios, such as perturbation of initial conditions, uncertainty in downscaling methods, and hydrologic models, in order to better validate the robustness of solutions from dynamic planning models. 3. A particularly valuable opportunity for collaboration is the identification of planning signposts from internal climate states and processes, rather than inferring these from streamflow alone. This can support the selection of indicator variables for policy search approaches, which remains underexplored.
Water Resources Agencies and Practitioners: 1. Dynamic planning methods are an important tool to design climate adaptation policies that adapt as uncertainties in both the climate and human system unfold over time. 2. Long-term planning studies cannot be expected to encompass all sources of uncertainty in climate forcing, human behavior, and natural variability. This underscores the need for careful problem formulation, framing, and interpretation of results. 3. Different impacts of climate change can be more accurately represented in scenarios or probability distributions than others. Some impacts are sufficiently well understood, as the direct consequence of rising temperatures, to begin planning adaptations in the near term.

10.1029/2019WR025502
Water Resources Research 4. When the uncertainty in climate impacts is difficult to quantify (e.g., extreme events), dynamic planning can still add value by asking: what future observations of precipitation and streamflow would necessitate action-either because the system is vulnerable or expected to become vulnerable-and what actions should be taken under those conditions? 5. Deep uncertainty regarding human responses to climate change can be studied according to how optimal planning decisions differ under alternative model assumptions defining these feedbacks.
In many ways, climate change only exacerbates uncertainties that have always been present in water resources planning, owing to the difficulty of enumerating all possible futures on decadal timescales. However, the increased uncertainty driven by climate change has pushed traditional planning methods beyond their limits-and also illuminated their limitations even in the absence of climate change. The challenge of dynamically mapping new observations and uncertain projections to actions will remain at the core of climate adaptation studies for the foreseeable future.

Data Availability Statement
No original data were created in this study.

Appendix A
This appendix describes the source of the hydrologic projections under climate change used in example Figure 4 and Figure 5.

Water Resources Research Acknowledgments
We thank Dr. Holger Maier and three anonymous reviewers for their thoughtful input on the structure and content of this review. This work was partially supported by the U.S. National Science Foundation grants CNH-1716130, CBET-1803563, and CBET-1803589. Any opinions, findings, and conclusions are those of the authors and do not necessarily reflect the views or policies of these funding agencies.