markov decision processes ( mdps ) are extensively used to solve sequential stochastic decision making problems in robotics @xcite and other disciplines @xcite . a solution to an mdp problem instance
provides a policy mapping states into actions with the property of optimizing ( e.g. , minimizing ) in expectation a given objective function . in many practical situations
a formulation based on expectation only is , however , not sufficient .
this is the case , in particular , when variability in the system s behavior can cause a highly undesirable outcome .
for example , in an autonomous navigation system , a robot attempting to minimize the expected length of the traveled path will likely travel close to obstacles , and a large deviation from the planned path may result in a collision causing a huge loss ( e.g. , damage to an expensive robot or failure of the whole mission altogether ) . metrics that study deviations from the expected value are often referred to as _
risk metrics_.    typical solution algorithms for mdp problems , like value iteration or policy iteration , aim exclusively at optimizing the expected cost .
the computed policies are therefore labeled as _
risk neutral_. the problem of quantifying risk associated with random variables has a rich history and is often related to the problem of managing financial assets @xcite .
in fact , many risk - related studies motivated by financial problems have recently found applications in domains such as robotics @xcite .
the term _ risk aversion _ refers to a preference for stochastic realizations with limited deviation from the expected value . in risk
averse optimal control one may prefer a policy with higher cost in expectation but lower deviations to one with lower cost but possibly larger deviations . particularly in the context of robotic planning , introducing risk aversion in mdps is crucial to guarantee mission safety . however introducing risk aversion in mdps creates a number of additional theoretical and computational hurdles .
for example , in risk averse mdps , optimal policies are not guaranteed to be markov stationary but are instead history dependent .
average value at risk ( avar  also known as conditional value at risk or cvar ) is a risk metric that has gained notable popularity in the area of risk averse control @xcite , @xcite . for a given random value and a predetermined confidence level , the avar is the _ tail average _ of the distribution exceeding a given confidence level ( see section [ sec : preliminaries ] for a formal definition ) .
risk averse policies considering the avar metric have been studied for the case of mdps with finite horizon and discounted infinite horizon cost criteria . in this paper
we instead consider how the avar metric can be applied when an undiscounted , total cost criterion is considered .
in fact , such cost criterion appears particularly useful and natural for robotic applications , whereby one is usually interested in optimizing the _ undiscounted , total cost _ accrued during a mission until a _ random _ , mission - dependent stopping time .
( as an aside , the total cost criterion is the typical cost model for stochastic shortest path problems , see , e.g. , @xcite . )
the contribution of this paper is three - fold :    * we identify conditions for the underlying mdp ensuring that the avar mdp problem is well defined when the total cost criterion is used .
* we define a surrogate mdp problem that can be efficiently solved and whose solution approximates the optimal policy for the original problem with arbitrary precision . * we validate our findings on a rapid robotic deployment task where the objective is to maximize the mission success rate under a given temporal deadline @xcite .    the rest of the paper is organized as follows .
we discuss related work in section [ sec : related ] and provide some background about risk metrics and mdps in section [ sec : preliminaries ] . in section [ sec : formulation ] we formulate the risk - averse , total cost mdp problem we wish to solve . in section [ sec : risk_aversion ] we propose and analyze an approximation strategy for the problem , and in section [ sec : avar ] we provide an algorithmic solution .
simulation results for a rapid deployment problem are given in section  [ sec : simulations ] , and conclusions and future work are discussed in section  [ sec : conclusions ] .
for a general introduction to mdps the reader is referred to textbooks such as @xcite or more recent collections such as @xcite . as pointed out in the introduction , risk aversion in mdps has been studied for over four decades , with earlier efforts focusing on exponential utility @xcite , mean - variance @xcite , and percentile risk criteria @xcite . with regard to mean - variance optimization in mdps
, it was recently shown that computing an optimal policy under a variance constraint is np - hard @xcite . recently
, average value at risk was introduced in @xcite in order to model the tail risk of a random outcome and to address some key limitations of the prevailing value - at - risk metric .
efficient methods to compute avar are discussed in @xcite .
leveraging the recent strides in avar risk modeling , there have been a number of efforts aimed at embedding the avar risk metric into risk - sensitive mdps . in @xcite
the authors address the problem of minimizing the avar of the discounted cost over a finite and an infinite horizon , and propose a dynamic programming approach based on state augmentation .
similar techniques can be found in @xcite , where the authors propose a dynamic programming algorithm for finite - horizon , avar - constrained mdps .
the algorithm is proven to asymptotically converge to an optimal risk - constrained policy .
however , the algorithm involves computing integrals over continuous variables ( algorithm 1 in @xcite ) and , in general , its implementation appears quite challenging .
a different approach is taken by @xcite where the authors consider a finite dimensional parameterization of the control policies , and show that an avar mdp can be optimized to a _ local _ optimum using stochastic gradient descent ( policy gradient ) .
however this approach imposes additional restrictions to the policy space and in general policy gradient algorithms only converge to a local optimum .
haskell and jain recently considered the problem of risk aversion in mdps using a framework based on occupancy measures @xcite ( closely connected to our recent works where constrained mdps are used to solve the multirobot rapid deployment problem @xcite ) .
while their findings are only valid for the case where an infinite horizon discounted cost criterion is considered , the solution we propose uses some of the ideas introduced in @xcite .
in this section we summarize some known concepts about risk metrics and mdps . the reader is referred to the aforementioned references for more details .      consider a probability space @xmath0 , and let @xmath1 be the space of all essentially bounded random variables on @xmath2 .
a _ risk function _ ( or risk metric ) @xmath3 is a function that maps an uncertain outcome @xmath4 onto the real line @xmath5 .
a risk function that is particularly popular in many financial applications is the _ value at risk_. for @xmath6 the _ value at risk _ of @xmath7 at level @xmath8 is defined as @xmath9 here var@xmath10 represents the percentile value of outcome @xmath11 at confidence level @xmath8 . despite its popularity , var@xmath12 has a number of limitations . in particular , var is not a _ coherent _ risk measure  @xcite and thus suffers from being unstable ( high fluctuations under perturbations ) when @xmath11 is not normally distributed .
more importantly it does not quantify the losses that might be incurred beyond its value in the @xmath8-tail of the distribution  @xcite .
an alternative measure that overcomes most shortcomings of var is the _ average value at risk _ ,
defined as @xmath13 where @xmath6 is the confidence level as before .
intuitively , @xmath14 is the expectation of @xmath11 in the conditional distribution of its upper @xmath8-tail . for this reason
, it can be interpreted as a metric of  how bad is bad . "
@xmath14 can be equivalently written as @xcite @xmath15\right\},\ ] ] where @xmath16 .
this paper relies extensively on eq . and aims at devising efficient methods to approximate the expectation in eq . when the random variable @xmath11 is the total cost of an mdp .
furthermore , it has been recently shown in @xcite that optimizing the cvar of total reward is equivalent to optimizing the worst - case ( robust ) expected total reward of a system whose model uncertainty is subjected to a trajectory budget .
this finding corroborates the fact that a cvar risk metric models both the variability of random costs , as well as the robustness to system transition errors .      for a finite set @xmath17 ,
let @xmath18 indicate the set of mass distributions with support on @xmath17 .
a finite , discrete - time markov decision process ( mdp ) is a tuple @xmath19 where    * @xmath20 , the state space , is a finite set comprising @xmath21 elements .
* @xmath22 , the control space , is a collection of @xmath21 finite sets @xmath23 .
set @xmath24 , @xmath25 , represents the actions that can be applied when in state @xmath26 .
the set of allowable state / action pairs is defined as @xmath27 * @xmath28 is the transition probability from state @xmath29 to state @xmath30 when action @xmath31 is applied . according to our definitions , @xmath32 .
* @xmath33 is a non - negative cost function .
specifically , @xmath34 is the cost incurred when executing action @xmath35 at state @xmath29 .
let @xmath36 and note that the maximum is attained as @xmath37 is a finite set .
define the set @xmath38 of admissible histories up to time @xmath39 by @xmath40 , for @xmath41 , and @xmath42 .
an element of @xmath43 has the form @xmath44 , and records all states traversed and actions taken up to time @xmath39 . in the most general case a policy is a function @xmath45 , i.e. , it decides which action to take in state @xmath46 considering the entire state - action history . note that according to this definition a policy is in general randomized .
let @xmath47 be the set of all policies , i.e. , including history - dependent , randomized policies .
it is well known that in the standard mdp setting where an expected cost is minimized there is no loss of optimality in restricting the optimization over deterministic , stationary markovian policies , i.e. , policies of the type @xmath48 .
however , in the risk - averse setting one needs to consider the more general class of history - dependent policies @xcite .
this is achieved through a state augmentation process described later .
following @xcite , we define the countable space @xmath49 , where @xmath50 , is the sample space and @xmath51 is the borel field on @xmath52 .
specific trajectories in the mdp are written as @xmath53 , and we denote by @xmath54 and @xmath55 the state and actions at time @xmath39 along trajectory @xmath56 . in general
the exact initial state @xmath57 is unknown .
rather it is described by an initial mass distribution @xmath58 over @xmath20 , i.e. , @xmath59 .
a policy @xmath60 and initial distribution @xmath58 induce a probability distribution over @xmath61 , that we will indicate as @xmath62 .
in this paper we focus on transient total cost mdps , defined as follows .
consider a partition of @xmath20 into sets @xmath63 and @xmath64 , i.e. , @xmath65 and @xmath66 .
transient _ mdp is an mdp where _ each _
policy @xmath60 satisfies the following two properties :    * @xmath67 < \infty$ ] for each @xmath68 , i.e. , the state will eventually enter set @xmath64 , and * @xmath69 for each @xmath70 , @xmath71 , @xmath35 , i.e. , once the state enters @xmath64 it can not leave it .    a transient , _ total cost _ mdp is a transient mdp where    * @xmath72 for each @xmath73 , i.e. , once the state enters @xmath64 no additional cost is incurred , * the cost associated with each trajectory @xmath56 is given by @xmath74    note that the cost @xmath75 is a random variable depending on both the policy @xmath60 and the initial distribution @xmath58 .
the name total cost stems from the fact that an ( undiscounted ) cost is incurred throughout the  lifetime " of the system ( i.e. , until the state hits the absorbing set ) .
transient , total cost mdps ( closely related to stochastic shortest path problems , e.g. , @xcite ) represent an alternative to the more commonly used discounted , infinite - horizon mdps or finite horizon mdps . as outlined in the introduction , for many robotic applications
the total cost , i.e. , @xmath75 , is the most appropriate cost function .
we justify this statement by noting that most robotic tasks have finite duration but such duration is usually not known in advance . in these circumstances
the finite horizon cost is inappropriate because one can not define the length of the finite horizon up front .
similarly , the discounted infinite horizon cost is also ill suited because the task does not continue forever and the cost will not be exponentially discounted over time .    without loss of generality , we assume that set @xmath64 consists of a single absorbing state @xmath76 equipped with a single action @xmath77 , i.e. , @xmath78 and @xmath79 with @xmath80 . in the following , with a slight abuse of notation ,
we denote by @xmath81 the set @xmath82 , i.e. , we exclude the absorbing state from the definition of @xmath81 .
moreover , we assume that for a transient , total cost mdp @xmath83 , i.e. , the probability of starting at the absorbing state is zero .
in fact , whenever @xmath84 the resulting state trajectory will deterministically remain in @xmath76 , and the corresponding cost is zero .
our problem formulation relies on the following two technical assumptions necessary to establish an a - priori upper bound on the total cost incurred by any trajectory obtained under any policy , and to define an approximate problem that can be efficiently solved .
the first assumption simply requires that all costs in the transient states are positive ( recall that we excluded @xmath76 when re - defining @xmath81 . )
as it will be shown later , this assumption ensures a non - zero discretization step when approximating the cumulative cost accrued by a system throughout the trajectory @xmath56 until it is absorbed in @xmath76 .
[ ass : nn ] all costs in @xmath85 except for state @xmath76 are positive and bounded , i.e. , @xmath86 .
when considering cost criteria like finite horizon or discounted infinite horizon with a finite state space , an a - priori upper bound on the accrued cost can be immediately established assuming that all costs are finite ( a fact crucially exploited in @xcite ) .
however , the situation is more complex when considering the total cost case , because without introducing further hypotheses on the structure of the mdp a malicious adversary could establish an history - dependent policy capable of invalidating any a - priori established bound on the cost , in the general case one could devise a history dependent policy ensuring that every trajectory generated by the policy is not absorbed in @xmath76 in less than @xmath87 steps , thus invalidating the bound . ] .
the second assumption then adds a  global reachability structure " to the mdp problem . to this end , in the following , it will be useful to consider the markov chain generated by the mdp when an input is selected for each state .
for an mdp @xmath85 , select @xmath88 .
the selected inputs and the transition probabilities in @xmath85 define a finite markov chain that we indicate as @xmath89 .
the state space of @xmath90 is equal to @xmath20 and for two states @xmath91 the transition probability @xmath92 is defined as @xmath93 where @xmath94 is the input selected in the definition of @xmath89 and @xmath95 is the transition probability of the associated mdp .
[ ass : irr ] let @xmath89 be the markov chain induced by the @xmath21 inputs @xmath94 . then the absorbing state @xmath76 , under markov chain @xmath89 , is reachable from any state @xmath68 , for all @xmath96
.    we recall that a state @xmath97 in a markov chain is said reachable from another state @xmath98 if there exists an integer @xmath99 such that the probability that the chain will be in state @xmath97 after @xmath100 transitions is positive @xcite . note that when assumption [ ass : irr ] holds , under every policy there is a path of non - zero probability connecting every state to @xmath76 .
therefore , it is impossible to devise a policy that prevents sure absorption for an arbitrary number of steps .
this holds for all policies , including history dependent policies ( see figure [ fig : assumption2 ] ) .    building upon the previous material
, we can now define the problem we aim to solve in this paper :    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ * risk - averse , total cost mdp *  given a transient total cost mdp satisfying assumptions [ ass : nn]-[ass : irr ] , and an initial distribution @xmath58 , determine a policy @xmath60 that minimizes @xmath101 , i.e. , find @xmath102 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _    note that , under the assumption of transient total cost mdp , one can easily verify that @xmath103<\infty$ ] . since , by equation , @xmath104 $ ] , one obtains @xmath105 for all @xmath56 , as well .
however , to derive an optimization algorithm for the computation of @xmath106 it is necessary to formulate an a - priori upper bound for the optimal cost in .
assumptions [ ass : nn ] and [ ass : irr ] are introduced to ensure that such bound exists and can be computed .
in this section we study an approximation strategy for the risk averse total cost mdp in equation .
similar to the method presented in @xcite , we aim at solving the problem by using the concept of _ occupation measures_. however , unlike for the cases studied in @xcite , in total cost mdps an explicit upper bound for the accrued cost is not available , which makes the solution strategy in @xcite not applicable .
our strategy is to find a surrogate to problem . by imposing an effective horizon
, we construct a total cost mdp with time - out and recast this problem into a bilinear programming problem .
furthermore we characterize the sub - optimality gap for such surrogate approximation .
we start with a technical result characterizing the convergence rate to the absorbing state .
consider a selection of inputs @xmath107 and the corresponding markov chain @xmath89 .
for each state @xmath68 , let @xmath108 denote the simple ( i.e. , without cycles ) path from @xmath29 to @xmath76 of lowest , strictly positive probability .
note that @xmath108 exists due to assumption [ ass : irr ] .
let @xmath109 be the probability of the path , i.e. , the product of the probabilities of all the transitions along the path .
since there are @xmath21 nodes and by definition the path is simple , @xmath108 includes at most @xmath110 transitions between @xmath21 nodes .
let @xmath111    note that the minimum is achieved as the minimization is over a finite set , and that @xmath112 is strictly positive due to assumption [ ass : irr ] .
the constant @xmath112 lower bounds the probability that , under _ any _ policy @xmath113 , the absorbing state is reached in no more than @xmath21 steps , from _ any _ state @xmath68 .
we are now in a position to characterize the convergence rate to the absorbing state .
[ lem : tech1 ]    for any policy @xmath113 and initial distribution @xmath58 , @xmath114\leq(1 - \gamma)^k,\,\ , \forall k\in \mathbb{n}.\ ] ]    the claim is proven by induction on @xmath100 .
base case : we prove that @xmath115 \leq 1- \gamma.\ ] ] indeed , @xmath116
= \sum_{x\in x_t } \pr_{\beta}^{\pi}[x_n\neq x_m|x_0 = x]\pr_{\beta}^{\pi}[x_0 = x]$ ] .
because of assumption [ ass : nn ] , for any policy @xmath60 , @xmath117 \geq \gamma$ ] , and the base case follows
.    for the inductive step , assume that @xmath118<(1 - \gamma)^k$ ] , for some @xmath119 . then , @xmath120 = \pr_{\beta}^{\pi}[x_{(k+1)n } \neq x_m |x_{kn}\neq x_m]\pr_{\beta}^{\pi}[x_{kn}\neq x_m]$ ] . by definition of @xmath112 ,
@xmath121 \leq ( 1-\gamma)^{k+1},\ ] ] and the claim follows .      our solution strategy is to solve a _ surrogate _ problem , whereby after a deterministic number @xmath122 of steps , the state moves to the absorbing state @xmath76 surely . in other words , @xmath123 acts as a  timeout " for the mdp problem .
the surrogate problem is simpler to solve , and we will show in the following that its solution can approximate the solution of the original problem with arbitrary precision
. denote by @xmath124}(\omega)$ ] the total cost for such surrogate problem .
additionally , for the original problem , let @xmath125 denote the _ absorbing time _ , i.e. , the time at which the state reaches @xmath76 . if @xmath126 , then the two processes coincide and then @xmath124}(\omega)=c(\omega)$ ] .
otherwise , for each trajectory @xmath56 such that @xmath127 , the random process is stopped after @xmath123 steps , and the state goes , deterministically , to @xmath76 at stage @xmath128 .
in such a case one has @xmath124}(\omega)\leq c(\omega)$ ] .
we want to characterize the relation between @xmath129}(\omega))$ ] ( i.e. , the risk for the surrogate problem ) and @xmath101 ( i.e. , the risk for the original problem ) . to this end , let @xmath130 be the total cost for the original problem up to time @xmath123 , i.e. , @xmath131 the following lemma shows the equivalence between @xmath124}(\omega)$ ] and @xmath130 .
[ lem : tech2 ] for any policy @xmath132 and any trajectory @xmath56 , @xmath124}(\omega ) = c_{d}(\omega)$ ] .    given a policy @xmath60 , for any trajectory @xmath56
, the cost cumulated up to time @xmath123 is the same for both the original and the surrogate problem .
after time @xmath123 , both @xmath133 and @xmath124}(\omega)$ ] do not cumulate any additional cost , then the claim follows .
the following theorem represents the main result of this section    [ them : sub ] the surrogate problem approximates the original problem according to @xmath134}(\omega ) ) \leq   \min_{\pi } \textrm{avar}_{\tau}(c(\omega))\\   & \qquad   \qquad \leq   \min_{\pi } \textrm{avar}_{\tau}(c^{[d]}(\omega ) ) +    \frac{n\overline{k}}{1 - \tau}\ ,   \frac{(1- \gamma)^{\lfloor ( d+1)/n\rfloor}}{\gamma}.   \end{split}\ ] ]    the left inequality is proven by noticing that @xmath135}(\omega))$ ] , where the equality follows from lemma [ lem : tech2 ] .
we now prove the right inequality . for any @xmath136 and policy @xmath60
, one has @xmath137=\mathbb{e}[(c(\omega)-s)^+\mid t^*(\omega)\leq d   ] \mathbb p(t^*(\omega)\leq d ) \nonumber\\   & + \mathbb{e}[(c(\omega)-s)^+\mid t^*(\omega ) > d ] \mathbb p(t^*(\omega ) > d ) .\label{eq : decompose }   \end{aligned}\ ] ]    let @xmath138 be the tail cumulated cost , and , as before , @xmath139 . since the function @xmath140 is sub - additive , i.e. , @xmath141 and the expectation operator preserves monotonicity , one obtains the inequality @xmath142 \\ & = \mathbb{e}[(c_d(\omega)+ c_l(\omega)-s)^+\mid t^*(\omega ) > d ] \\ & \leq\mathbb{e}[(c_d(\omega)-s)^+ \mid t^*(\omega ) > d ] + \mathbb{e}[c_l(\omega )   \mid t^*(\omega ) > d ] . \end{split}\ ] ] furthermore , for each trajectory in the event set @xmath143 , one has @xmath144=
\mathbb{e}\left[\left(c_d(\omega)-s\right)^+\mid t^*(\omega)\leq d   \right].\ ] ] collecting the results so far , one has the following inequalities : @xmath145\\ & \leq \mathbb{e}[(c_d(\omega)-s)^+\mid t^*(\omega)\leq d   ] \mathbb p(t^*(\omega)\leq d )   \\ & \qquad + \mathbb{e}[(c_d(\omega)-s)^+ \mid t^*(\omega ) > d ] \mathbb p(t^*(\omega ) > d ) + \\ & \qquad +   \mathbb{e}\left[c_l(\omega)\mid t^*(\omega ) > d \right]\mathbb
p(t^*(\omega ) >
d)\\ & = \mathbb{e}[(c_d(\omega)-s)^+ ] + \mathbb{e}\left[c_l(\omega)\mid t^*(\omega ) > d \right]\mathbb
p(t^*(\omega ) > d)\\ & \leq \mathbb{e}[(c_d(\omega)-s)^+ ] + \mathbb{e}\left[c_l(\omega)\right ] .
\end{split}\ ] ]    equation implies @xmath146\right\}\\ & \leq \min_{\pi } \
, \min_{s\in \mathbb{r}}\left\{s+\frac{1}{1-\tau } \left(\mathbb{e}[(c_d(\omega)-s)^+ ] + \mathbb{e}\left[c_l(\omega)\right]\right)\right\}\\ & \leq \min_{\pi } \ , \min_{s\in \mathbb{r}}\left\{s+\frac{1}{1-\tau } \left(\mathbb{e}[(c_d(\omega)-s)^+ ] \right)\right\ } \\ & \qquad + \max_{\pi } \ , \frac{1}{1-\tau}\mathbb{e}\left[c_l(\omega)\right]\\ & = \min_{\pi } \ , \min_{s\in \mathbb{r}}\left\{s+\frac{1}{1-\tau } \left(\mathbb{e}[(c^{[d]}(\omega)-s)^+ ] \right)\right\ } \\ & \qquad + \max_{\pi } \ , \frac{1}{1-\tau}\mathbb{e}\left[c_l(\omega)\right]\\ & =   \min_{\pi }   \textrm { avar}_{\tau}(c^{[d]}(\omega ) ) + \max_{\pi } \ , \frac{1}{1-\tau}\mathbb{e}\left[c_l(\omega)\right ] , \end{split}\ ] ] where the second to last equality follows from lemma [ lem : tech2 ] .
we are left with the task of upper bounding @xmath147 $ ] . to this purpose ,
one can write @xmath148\leq \overline{k } \sum_{t = d+1}^{\infty } \ ,
\pr_{\beta}^{\pi}(x_t \neq x_m).\ ] ]    note that the result in lemma [ lem : tech1 ] implies @xmath149 \leq\!\ !
\!\sum_{k=\lfloor ( d+1)/n\rfloor}^{\infty}\!\!\!\!\sum_{t = kn}^{(k+1)n-1}\pr_{\beta}^{\pi}[x_{t}\neq x_m]\\ & \leq \sum_{k=\lfloor ( d+1)/n\rfloor}^{\infty } n(1-\gamma)^k    = n \frac{(1- \gamma)^{\lfloor ( d+1)/n\rfloor}}{\gamma}. \end{split}\ ] ] the claim then follows immediately , as the above upper bound is policy - independent .    note that according to theorem [ them : sub ] , as @xmath150 , the optimal cost of the surrogate problem recovers the optimal cost of the original problem , i.e. , the surrogate problem provides a consistent approximation to the original problem , with a sub optimality factor that is computable from problem data .
lemma [ lem : tech2 ] and theorem [ them : sub ] ensure that @xmath124}(\omega)$ ] can approximate @xmath75 with arbitrary precision for a sufficiently large value of @xmath123 . in the next section
we then show how to solve the minimization problem : @xmath151}(\omega)).\ ] ]
leveraging the surrogate problem from the previous section , we can now adapt the results proposed in @xcite to solve problem .
an essential step to solve this optimization problem is to compute @xmath152}(\omega)-s)^+]$ ] , which entails deriving the probability distribution for the possible costs generated by the random variable @xmath124}(\omega)$ ] .
this problem can be solved by suitably augmenting the state space as described in the following , and then using occupancy measures . in the space of occupancy measures
, an optimal policy is determined through the solution of a bilinear program , as explained below . for a given policy @xmath60 and initial distribution @xmath58
, we define the occupancy measure for @xmath153 as @xmath154.\ ] ] note that @xmath155 is non negative but is in general not a probability itself . in the following
we will use occupancy measures to determine the probability distribution of the total costs @xmath124}(\omega)$ ] and then to compute the needed expectation . according to the definition ,
occupancy measures depend on the policy @xmath60 and the initial distribution @xmath58 . given an absorbing mdp @xmath19
, we define a new _ state - augmented _ absorbing mdp with additional state components that track the cumulated total cost and current stage .
although the original mdp @xmath85 is finite and absorbing , the set of costs @xmath124}(\omega)$ ] generated by all possible policies can be very large , and this can subsequently lead to a linear program with an unmanageable number of decision variables . to counter this problem , we introduce a discretized approximation for @xmath124}(\pi,\beta)$ ] whose error can be arbitrarily bounded . to this end
, we set @xmath156 , where @xmath157 is a parameter describing the desired number of discretized values for the cumulated cost .
due to assumption [ ass : nn ] , @xmath158 is strictly positive . the effective number of different values is @xmath159 this value may be higher than @xmath160 due to our definition of @xmath158 . we then define a new mdp @xmath161 as follows .
its state space is @xmath162 , where @xmath163 and @xmath164 .
elements in the augmented states will be indicated as @xmath165 . as clarified in the following , the two additional components store the cumulated running cost ( @xmath30 ) and current stage ( @xmath166 ) .
recall that in the surrogate problem , after @xmath123 steps , the state is guaranteed to have entered the absorbing set , i.e. , it is guaranteed that @xmath167 .
thus the value of the @xmath166 component is in @xmath164 . on the other hand
the input sets are defined as @xmath168 .
@xmath169 and @xmath170 induce a new set @xmath171 .
the new cost function @xmath172 is @xmath173 .
the transition probability function is modified as follows : @xmath174 as evident from the definition of the new transition function , the new variables included in the state stores the discretized .
] running cost and the stage .
consistently with our definition of the surrogate problem , the revised transition function includes a timeout that imposes a transition to the absorbing state @xmath76 after @xmath123 steps , and from that point onwards the accrued cost does not change .
note also that the additional state components @xmath30 and @xmath166 are deterministic functions of the previous state and control input @xmath175 . extending the formerly introduced notation , for a given trajectory @xmath56 of @xmath176 , we write @xmath177 for the second component of the state at time @xmath39 and @xmath178 for the third component .
finally , for a given initial distribution @xmath58 on @xmath20 , we define the following new initial distribution @xmath179 on @xmath169 , @xmath180    note that the properties of @xmath85 carry over to @xmath176 . in particular , if assumptions [ ass : nn]-[ass : irr ] hold for @xmath85 then they hold for @xmath176 too , and , if @xmath85 is absorbing , then @xmath176 is also absorbing .
thus we indicate with @xmath181 its transient set of states . for a given realization @xmath56 , consider now @xmath182}(\omega ) = \sum_{i=0}^tc(x_i , u_i)$ ] , i.e. , the true cumulative cost of the surrogate mdp problem without discretization .
the following theorem establishes that even though the approximation error introduced by discretizing the running cost grows linearly with @xmath39 , it is possible to bound it with arbitrary precision .
[ th : approx ] for each @xmath183 and each @xmath184 , there exists a discretization step @xmath158 such that @xmath185}(\omega)| < \varepsilon$ ] .    _
proof_. pick @xmath186 .
let @xmath187}(\omega)-\zeta y_t(\omega)$ ] be the approximation error at time @xmath39 .
note that by definition @xmath188 and @xmath189}(\omega)-\zeta y_0(\omega)=0 $ ] . from the definition of the transition probability function , @xmath190
, it follows that @xmath191 , which implies @xmath192 .
since for @xmath193 we have @xmath194 , the claim follows .
@xmath195 + a key step towards the solution of problem in is therefore to derive the statistical description of the discretized total cost @xmath196 that is used to approximate @xmath124}(\omega)$ ] .
this objective can be achieved by exploiting the occupancy measures for the state - augmented mdp @xmath197 . for @xmath198 , the occupancy measure on @xmath197 induced by a policy @xmath60 and an initial distribution @xmath58
is given as : @xmath199.\nonumber \ ] ] the occupancy measure , @xmath200 , is a vector in @xmath201 , i.e. , it is a vector with @xmath202 non negative components .
the set of legitimate occupancy vectors is constrained by the initial distribution @xmath58 and defined by the policy @xmath60 .
it is well known @xcite that these constraints can be expressed as follows :    @xmath203=\beta(x , y , z)~\forall ( x , y , z)\in { x}'^t \nonumber\end{aligned}\ ] ]    where @xmath204 if and only if @xmath205 .
for @xmath206 we introduce random variables @xmath207 with the property that @xmath208 $ ] .
this is easily achieved using occupancy measures , i.e. , @xmath209 where @xmath210 is the indicator function equal to 1 when its argument is true , and 0 otherwise .
note that by definition @xmath207 is equal to @xmath211 $ ] , and by theorem [ th : approx ] @xmath196 approximates @xmath124}(\omega)$ ] with arbitrary precision . combining the above definitions we then get to the following problem whose solution approximates the solution to : @xmath212 } s+\frac{1}{1-\tau } \sum_{y \in \mathbb{n}_n}(y - s)^+\theta(y )   \\ & \textrm{s.t . }
\nonumber \\ & \sum_{(x',y',z')\in { x}'^t } \sum_{u\in a(x',y',z')}\rho(x',y',z',u)[\delta_{(x , y , z)}(x',y',z')- \nonumber \\ & p'((x',y',z')|(x , y , z),u)]=\beta(x , y , z)~\forall ( x , y , z)\in { x}'^t \nonumber\\   & ~ ~ ~ \theta(k ) = \!\!\!\!\ !   \sum_{(x , y , z , u)\in \mathcal{k}'}\!\!\!\!\!i(y = k~\wedge z = d ) \rho(x , y , z , u),\ , 0 \leq k \leq n.\nonumber\end{aligned}\ ] ] when comparing this last optimization problem with , the reader will note that the variable @xmath213 is constrained in the interval @xmath214 $ ] .
indeed , the objective function is continuous with respect to @xmath213 , and it is straightforward to verify that the partial derivative of the objective function with respect to @xmath213 is negative for @xmath215 and positive for @xmath216 . the objective function given in eq .
is concave with respect to @xmath217 and is defined over a convex feasibility set @xcite . to the best of our knowledge ,
there exist no efficient methods to determine the global minimum for this class of problems .
hence , the problem is approximately solved fixing different values of @xmath213 within the range @xmath214 $ ] , and then solving the corresponding linear problem over the optimization variables @xmath200 and @xmath218 . comparing the problem in eq . with the one in eq .
one might initially think that the objective function in eq . does not depend on the policy @xmath60 .
however , the dependency on @xmath60 is carried over by the occupancy measure @xmath200 , as evident from eq . .
moreover , it is well known from the theory of constrained mdps @xcite that there is a one to one correspondence between policies and occupancy measures , i.e. , every policy defines a unique occupancy measure and every occupancy measure induces a policy .
to illustrate the performance of the proposed algorithm , we adopt the rapid deployment scenario considered in @xcite .
a graph is used to abstract and model the connectivity of a given map of an environment ( see , e.g. , @xcite ) .
one robot is positioned at a start vertex and is tasked to reach the goal vertex within a given temporal deadline while providing some guarantee about its probability of successfully completing the task . when moving from vertex to vertex
, the robot can choose from a set of actions , each trading off speed with probability of success .
in particular , actions with rapid transitions between two vertices have higher probability of failure ; and conversely when the robot moves slowly between two vertices it has a higher probability of success . in this scenario ,
_ failure _ means that the robot does not move ( e.g. , fails to pass through an opening ) , so elapsed time increases without making progress towards the goal . with a given temporal deadline @xmath219 and success probability @xmath220 ,
the robot is tasked to reach the target vertex  safely "
( such that the true mission success probability is at least @xmath220 ) , while satisfying the temporal constraint . from a design perspective
it is of interest to know if there exists a policy @xmath60 achieving this objective , and to compute it . if the policy does not exist , it is of interest to know how to modify the parameters in order to make the task feasible .    in our previous work we solved this problem by modeling it using constrained markov decision processes ( cmdp ) . in the cmdp approach ,
one maximizes the probability of success while imposing a constraint on the temporal deadline .
however , this method only returns risk - neutral policies , i.e. , the resultant policies only guarantee that the temporal deadline is met in expectation , and there is no explicit control on the tail probability of the constraint . as a radical departure from the original problem formulation , the avar minimization method proposed in eq .
searches for a policy that is feasible with respect to the temporal deadline constraint with finite expectation , @xmath221 $ ] for @xmath222 $ ] . therefore , if the solution to the avar minimization problem is bounded above by the temporal deadline , then the corresponding minimizer is also a feasible policy to the original problem . ] and systematically controls the worst - case variability of total travel time .
note that a policy with low success probability will have large tail probability in total travel time even if the expected temporal deadline is met .
therefore the optimal policies from avar minimization will have _ high success probability_. this motivates the application of avar minimization to rapid robotic deployment .
first , note that , in the devised setting , the robot will eventually reach the final goal with positive probability .
however , due to possible failures one can not put an a - priori bound on the random total travel time .
therefore , the total cost criterion is indeed a natural choice for this task .
moreover , assumptions [ ass : nn ] and [ ass : irr ] are easily justified because the immediate cost function ( i.e. , time to move ) is always positive and the global reachability property follows from the graph structure .    to illustrate the performance of risk - averse deployment ,
two different policies are compared . here
both policies are computed using unconstrained stochastic control methodologies for which the immediate cost is the travel time between two vertices , and the actions correspond to all possible node transitions on the graph .
the first is the classic risk - neutral policy obtained with value iteration .
the second is a risk averse policy obtained with the algorithm presented in this paper using @xmath223 . for each policy
, 1000 executions are run , and the distribution of total travel time is reported .
figures [ fig : riskneutral ] and [ fig : riskaverse ] show the distributions for the two cases .
the risk - neutral policy obtains a lower expected cost , but has a longer tail , as evidenced by the 61 instances with a cost larger than or equal to 15 ( notice that @xmath224 is the desired time of completion in this example ) .
moreover , as evidenced by the shape of the histogram , costs are more spread out . the risk averse policy , on the other hand , results in less variability as desired .
less than 30 instances have a cost larger or equal than 15 , a reduction of the weight of the tail by more than one half .        .
]    importantly , when computing a risk - neutral policy using classic methods like policy iteration or value iteration , one is merely provided with a policy that minimizes the expected cost ( in our case time to completion ) , and no additional information is readily available . with our approach instead , one not only obtains a policy minimizing the avar criterion , but a statistical description of the costs is also obtained as a byproduct .
that is to say , for each discretized completion time @xmath100 , the probability @xmath225}=k]$ ] is computed as well , thus unveiling the relationship between the time to complete the deployment task and its probability .
this is shown in figure [ fig : probcomparison ] for different values of @xmath8 .
hence , if the computed policy does not meet the desired performance , the designer has information on how to tune @xmath219 and @xmath220 .    .
in this paper we have considered how risk aversion in mdps can be introduced jointly with the avar risk metric under the total cost criterion .
our results advance the state of the art because avar has only been previously considered in mdps with finite horizon or discounted infinite horizon cost criteria .
such extension is _ important _ as the total cost criterion appears as a natural model for robotic applications , and is _ non - straightforward _ as current algorithms , e.g. , from @xcite and @xcite , only work with bounded cumulated costs ( which is not the case for total cost formulations ) .
under two mild assumptions , an approximation algorithm with provable sub - optimality gap was provided .
furthermore , a rapid deployment scenario was used to demonstrate that risk - aversion gives more informative policies when compared to traditional risk - neutral formulations .
while our findings focus on risk averse mdps with an avar risk metric , our approach can be easily extended along multiple dimensions .
in particular , by exploiting the results presented in @xcite , it is possible to use our approximation for a broader range of risk metrics , i.e. , metrics that are uniformly continuous and law invariant .
moreover , since the algorithm we considered is based on occupancy measures , it can be easily extended to the cmdp case
. this will be the focus of future work .