the seminal paper  @xcite introduced an important class of sequential optimization problems , otherwise known as multi - armed bandits .
these models have since been used extensively in such fields as statistics , operations research , engineering , computer science and economics .
the traditional multi - armed bandit problem can be described as follows .
consider @xmath0 statistical populations ( arms ) , where at each point in time it is possible to sample from ( pull ) only one of them and receive a random reward dictated by the properties of the sampled population .
the objective is to devise a sampling policy that maximizes expected cumulative rewards over a finite time horizon .
the difference between the performance of a given sampling policy and that of an oracle , that repeatedly samples from the population with the highest mean reward , is called the _ regret_.
thus , one can re - phrase the objective as minimizing the regret .
when the populations being sampled are homogeneous , that is , when the sequential rewards are independent and identically distributed ( i.i.d . ) in each arm , the family of upper - confidence - bound ( ucb ) policies , introduced in  @xcite , incur a regret of order @xmath1 , where @xmath2 is the length of the time horizon , and no other `` good '' policy can ( asymptotically ) achieve a smaller regret ; see also  @xcite .
the elegance of the theory and sharp results developed in  @xcite hinge to a large extent on the assumption of homogenous populations and hence identically distributed rewards .
this , however , is clearly too restrictive for many applications of interest .
often , the decision maker observes further information and based on that , a more _ customized _ allocation can be made . in such settings
, rewards may still be assumed to be independent , but no longer identically distributed in each arm . a particular way to encode
this is to allow for an exogenous variable ( a covariate ) that affects the rewards generated by each arm at each point in time when this arm is pulled .
such a formulation was first introduced in @xcite under parametric assumptions and in a somewhat restricted setting ; see  @xcite and @xcite for very different recent approaches to the study of such bandit problems , as well as references therein for further links to antecedent literature . the first work to venture outside the realm of parametric modeling assumptions appeared in @xcite . in particular , the mean response in each arm , conditionally on the covariate value , was assumed to follow a general functional form ; hence one can view their setting as a _ nonparametric _ bandit problem .
they propose a variant of the @xmath3-greedy policy ( see , e.g. ,  @xcite ) and show that the average regret tends to zero as the time horizon @xmath2 grows to infinity .
however , it is unclear whether this policy satisfies a more refined notion of optimality , insofar as the magnitude of the regret is concerned , as is the case for ucb - type policies in traditional bandit problems .
such questions were partially addressed in  @xcite where near - optimal bounds on the regret are proved in the case of a two - armed bandit problem under only two assumptions on the underlying functional form that governs the arms responses .
the first is a mild smoothness condition , and the second is a so - called _ margin condition _ that involves a _ margin parameter _ which encodes the `` separation '' between the functions that describe the arms responses .    the purpose of the present paper is to extend the setup of @xcite to the @xmath4-armed bandit problem with covariates when @xmath4 may be large .
this involves a customized definition of the margin assumption .
moreover , the bounds proved in  @xcite suffered two deficiencies .
first , they hold only for a limited range of values of the margin parameter and second , the upper bounds and the lower bounds mismatch by a logarithmic factor . improving upon these results requires radically new ideas . to that end , we introduce three policies :    successive elimination ( se ) is dedicated to the static bandit case .
it is the cornerstone of the other policies that deal with covariates . during a first phase
, this policy explores the different arms , builds estimates and eliminates sequentially suboptimal arms ; when only one arm remains , it is pulled until the horizon is reached .
a variant of se was originally introduced in @xcite .
however , it was not tuned to minimize the regret as other measures of performance were investigated in this paper .
we prove new regret bounds for this policy that improve upon the canonical papers  @xcite and  @xcite .
binned successive elimination ( bse ) follows a simple principle to solve the problem with covariates .
it consists of grouping similar covariates into bins and then looks only at the average reward over each bin .
these bins are viewed as indexing `` local '' bandit problems , solved by the aforementioned se policy .
we prove optimal regret bounds , polynomial in the horizon but only for a restricted class of difficult problems .
for the remaining class of easy problems , the bse policy is suboptimal .
adaptively binned successive elimination ( abse ) overcomes a severe limitation of the naive bse .
indeed , if the problem is globally easy ( this is characterized by the margin condition ) , the bse policy employs a fixed and too fine discretization of the covariate space . instead , the abse policy partitions the space of covariates in a fashion that adapts to the local difficulty of the problem : cells are smaller when different arms are hard to distinguish and bigger when one arm dominates the other .
this adaptive partitioning allows us to prove optimal regrets bounds for the whole class of problems .
the optimal polynomial regret bounds that we prove are much larger than the logarithmic bounds proved in the static case .
nevertheless , it is important to keep in mind that they are valid for a much more flexible model that incorporates covariates . in the particular case where @xmath5 and the problem is _ difficult _ , these bounds improve upon the results of  @xcite by removing a logarithmic factor that is idiosyncratic to the _ exploration vs. exploitation _ dilemma encountered in bandit problems .
moreover , it follows immediately from the previous minimax lower bounds of  @xcite and @xcite , that these bounds are optimal in a minimax sense and thus can not be further improved .
it reveals an interesting and somewhat surprising phenomenon : the price to pay for the partial information in the bandit problem is dominated by the price to pay for nonparametric estimation .
indeed the bound on the regret that we obtain in the bandit setup for @xmath5 is of the same order as the best attainable bound in the _ full information _ case , where at each round , the operator receives the reward from only one arm but observes the rewards of both arms .
an important example of the full information case is sequential binary classification .
our policies for the problem with covariates fall into the family of `` plug - in '' policies as opposed `` minimum contrast '' policies ; a detailed account of the differences and similarities between these two setups in the full information case can be found in  @xcite .
minimum contrast type policies have already received some attention in the bandit literature with side information , aka _
contextual bandits _ , in the papers
@xcite and also @xcite .
a related problem online convex optimization with side information was studied in @xcite , where the authors use a discretization technique similar to the one employed in this paper .
it is worth noting that the cumulative regret in these papers is defined in a weaker form compared to the traditional bandit literature , since the cumulative reward of a proposed policy is compared to that of the best policy in a certain restricted class of policies .
therefore , bounds on the regret depend , among other things , on the complexity of said class of policies .
plug - in type policies have received attention in the context of the continuum armed bandit problem , where as the name suggests there are uncountably many arms .
notable entries in that stream of work are  @xcite and  @xcite , who impose a smoothness condition both on the space of arms and the space of covariates , obtaining optimal regret bounds up to logarithmic terms .
in this section , it will be convenient for notational purposes , to consider a multi - armed bandit problem with @xmath6 arms .
we revisit the successive elimination ( se ) policy introduced in  @xcite in the traditional setup of multi - armed bandit problems .
as opposed to the more popular ucb policy ( see , e.g. , @xcite ) , it allows us in the next section , to construct an adaptive partition that is crucial to attain optimal rates on the regret for the dynamic case with covariates . in this section , we prove refined regret bounds for the se policy that exhibit a better dependence on the expected rewards of the arms compared to the bounds for ucb that were derived in  @xcite .
such an improvement was recently attempted in  @xcite and also in  @xcite for modified ucb policies and we compare these results to ours below .
let us recall the traditional setup for the static multi - armed bandit problem ; see , for example ,  @xcite .
let @xmath7 be a given set of @xmath8 arms .
successive pulls of arm @xmath9 yield rewards @xmath10 that are i.i.d . random variables in
@xmath11 $ ] with expectation given by @xmath12 = f^{(i)}\in[0,1]$ ] .
assume without loss of generality that @xmath13 so that @xmath6 is one of the best arms .
for simplicity , we further assume that the best arm is _ unique _ since for the se policy , having multiple optimal arms only improves the regret bound . in the analysis
, it is convenient to denote this optimal arm by @xmath14 and to define the _ gaps _ traditionally denoted by @xmath15 , by @xmath16 .    a _ policy _
@xmath17 is a sequence of random variables @xmath18 indicating which arm to pull at each time @xmath19 , and such that @xmath20 depends only on observations strictly anterior to @xmath21 .
the performance of a policy @xmath22 is measured by its ( _ cumulative _ ) _ regret _ at time @xmath2 defined by @xmath23 note that for a data - driven policy @xmath24 , this quantity is random and , in the rest of the paper , we provide upper bounds on @xmath25 .
such bounds are referred to as _ regret bounds_.    we begin with a high - level description of the se policy denoted by @xmath24 .
it operates in rounds that are different from the decision times @xmath26 . at the beginning of each round @xmath27 ,
a subset of the arms has been eliminated and only a subset @xmath28 remains . during round @xmath29 , each arm in @xmath28 is pulled exactly once ( exploration ) . at the end of the round , for each remaining arm in @xmath28 , we decide whether to eliminate it using a simple statistical hypothesis test : if we conclude that its mean is significantly smaller than the mean of any remaining arm , then we eliminate this arm and we keep it otherwise ( elimination ) .
we repeat this procedure until @xmath2 pulls have been made .
the number of rounds is random but obviously smaller than  @xmath2 .    the se policy , which is parameterized by two quantities @xmath30 and @xmath31 and described in policy  [ algse ] , outputs an infinite sequence of arms @xmath32 without a prescribed horizon .
of course , it can be truncated at any horizon @xmath2 .
this description emphasizes the fact that the policy can be implemented without perfect knowledge of the horizon @xmath2 and in particular , when the horizon is a random variable with expected value @xmath2 ; nevertheless , in the static case , it is manifest from our result that , when the horizon is known to be @xmath2 , choosing @xmath33 is always the best choice when possible and that other choices may lead to suboptimal results .
set of arms @xmath34 ; parameters @xmath35 ; horizon @xmath2 .
@xmath37 , @xmath38 , @xmath39 , @xmath40^k$ ]    @xmath41 @xmath42 @xmath43 ( observe @xmath44 ) exploration @xmath45 , @xmath46 @xmath47 $ ] @xmath48 .
elimination @xmath49 .
note that after the exploration phase of each round @xmath50 each remaining arm @xmath51 has been pulled exactly @xmath29 times , generating rewards @xmath52 .
denote by @xmath53 the average reward collected from arm @xmath54 at round @xmath29 that is defined by @xmath55 , where here and throughout this paper , we use the convention @xmath56 . in the rest of the paper
, @xmath57 denotes the natural logarithm and @xmath58 . for any positive integer @xmath59 ,
define also @xmath60 which is essentially a high probability upper bound on the magnitude of deviations of @xmath61 from its mean @xmath62 .
the se policy for a @xmath4-armed bandit problem can be implemented according to the pseudo - code of policy  [ algse ] .
note that , to ease the presentation of sections  [ secbse ] and  [ secabse ] , the se policy also returns at each time @xmath21 , the number of rounds @xmath63 completed at time @xmath21 and a subset @xmath64 of arms that are active at time @xmath21 , where @xmath65 denotes the power set of @xmath66 .
the following theorem gives a first upper bound on the expected regret of the se policy .
[ thstatic ] consider a @xmath67-armed bandit problem where horizon is a random variable @xmath68 of expectation @xmath2 that is independent of the random rewards .
when implemented with parameters @xmath69 , the se policy @xmath24 exhibits an expected regret bounded , for any @xmath70 , as @xmath71\leq392 \gamma^2 \biggl(1 + \frac{n}{t } \biggr)\frac { k}{\delta}{\overline{\log}}\biggl(\frac{t\delta^2}{18\gamma^2 } \biggr ) + n \delta^-,\ ] ] where @xmath72 is the largest @xmath73 such that @xmath74 if it exists , otherwise @xmath75 .
assume without loss of generality that @xmath76 , for @xmath77 since arms @xmath78 such that @xmath79 do not contribute to the regret .
define @xmath80 . moreover , for any @xmath81 in the set @xmath28 of arms that remain active at the beginning of round @xmath29 , define @xmath82 .
recall that , at round @xmath29 , if arms @xmath83 , then ( i ) the optimal arm @xmath84 eliminates arm @xmath81 if @xmath85 , and ( ii ) arm @xmath81 eliminates arm @xmath84 if @xmath86 .
since @xmath87 estimates @xmath88 , the event in ( i ) happens approximately , when @xmath89 , so we introduce the deterministic , but unknown , quantity @xmath90 ( and its approximation @xmath91 ) defined as the solution of @xmath92 note that @xmath93 as well as the bound @xmath94    we are going to decompose the regret accumulated by a suboptimal arm @xmath81 into three quantities :    * the regret accumulated by pulling this arm at most until round @xmath95 : this regret is smaller than @xmath96 ; * the regret accumulated by eliminating the optimal arm @xmath84 between round @xmath97 and @xmath95 ; * the regret induced if arm @xmath81 is still present at round @xmath95 ( and in particular , if it has not been eliminated by the optimal arm @xmath84 ) .
we prove that the second and third events happen with small probability , because of the choice of @xmath95 .
formally , define the following _ good _ events : @xmath98 moreover , define @xmath99 and observe that @xmath100 . for any @xmath101 ,
the contribution to the regret incurred after time @xmath95 on @xmath102 is at most @xmath103 since each pull of arm @xmath104 contributes to the regret by @xmath105 .
we decompose the underlying sample space denoted by @xmath106 into the disjoint union @xmath107 where @xmath108 is chosen later .
it implies the following decomposition of the expected regret : @xmath109 define by @xmath110 the complement of an event @xmath111 .
note that the first term on the right - hand side of the above inequality can be decomposed as follows : @xmath112\\[-8pt ] & & { } + n\sum _ { i=1}^{k_0}\delta_i { \mathbb p}\bigl ( { \mathcal{b}}_i^c\cap{\mathcal{a}}_i\cap{\mathcal{b}}_{i-1 } \bigr),\nonumber\end{aligned}\ ] ] where the right - hand side was obtained using the decomposition @xmath113 and the fact that @xmath114 .    from hoeffding s inequality
, we have that for every @xmath115 , @xmath116\\[-8pt ] & \leq&\exp\biggl(-\frac{\tau(\delta_i-\gamma\varepsilon _
\tau ) ^2}{2 } \biggr).\nonumber\end{aligned}\ ] ] on the event @xmath117 , arm @xmath84 has not eliminated arm @xmath81 at @xmath95 .
therefore @xmath118 .
together with the above display with , it yields @xmath119 where we used the fact that @xmath120 .
it remains to bound the first term in the right - hand side of ( [ eqdecompci ] ) . on the event @xmath121 ,
the optimal arm @xmath84 has not been eliminated before round @xmath122 , but every suboptimal arm @xmath123 has .
so the probability that there exists an arm @xmath124 that eliminates @xmath84 between @xmath122 and @xmath95 can be bounded as @xmath125,\end{aligned}\ ] ] where @xmath126 .
using lemma  [ lempeeling ] , we get @xmath127 .
this bound implies that @xmath128 \\ & & \qquad\le\sum_{j=1}^k \sum_{i=1}^{j\wedge k_0 - 1 } \phi_j ( \tau_i ) ( \delta_i -\delta_{i+1 } ) + \sum _ { j=1}^k \phi_{j\wedge k_0}(\tau _ { j\wedge k_0})\delta_{j\wedge k_0 } \\ & & \qquad\leq\frac{4}{t } \sum_{j=1}^k \sum_{i=1}^{j\wedge k_0 - 1 } \tau_i ( \delta_i -\delta_{i+1 } ) + \frac{4}{t}\sum _ { j=1}^k \tau_{j\wedge k_0}\delta_{j\wedge k_0}.\end{aligned}\ ] ] using ( [ eqboundtau ] ) and @xmath129 , the first sum can be bounded as @xmath130.\end{aligned}\ ] ] the previous two displays together with ( [ eqboundtau ] ) yield @xmath131 putting together ( [ eqregret1 ] ) , ( [ eqdecompci ] ) , ( [ eqboundbi ] ) and the above display yield that the expected regret @xmath132of the se policy is bounded above by @xmath133\\[-8pt ] & & \qquad{}+304\frac{\gamma^2n}{t } \frac{k - k_0}{\delta_{k_0}}{\overline{\log}}\biggl(\frac{n\delta^2_{k_0}}{18\gamma^2 } \biggr)+n\delta_{k_0 + 1}.\nonumber\end{aligned}\ ] ] fix @xmath134 and let @xmath135 be such that @xmath136 .
an easy study of the variations of the function @xmath137 reveals that @xmath138 for any @xmath139 . using this bound equation ( [ eqregretgen ] ) with @xmath140 and @xmath141 completes the proof .
the following corollary is obtained from a slight variations on the proof of theorem  [ thstatic ] .
it allows us to better compare our results to the extant literature .
[ corstatic ] under the setup of theorem  [ thstatic ] , the se policy @xmath24 run with parameter @xmath33 and @xmath142 satisfies for any @xmath143 , @xmath144 in particular , @xmath145    note that ( [ eqcorstatic1 ] ) follows from ( [ eqregretgen ] ) . to prove ( [ eqthstatic ] ) , take @xmath146 in ( [ eqcorstatic1 ] ) and @xmath147 in theorem  [ thstatic ] , respectively .
this corollary is actually closer to the result of  @xcite .
the additional second term in our bound comes from the fact that we had to take into account the probability that an optimal arm @xmath84 can be eliminated by any arm , not just by some _ suboptimal arm _
with index lower than @xmath135 ; see  @xcite , page 8 .
it is unclear why it is enough to look at the elimination by those arms , since if @xmath84 is eliminated  no matter the arm that eliminated it  the hoeffding bound ( [ eqhoeff ] ) no longer holds .    the right - hand side of ( [ eqthstatic ] ) is the minimum of two terms .
the first term is distribution - dependent and shows that the se policy adapts to the unknown distribution of the rewards .
it is very much in the spirit of the original bound of  @xcite and of the more recent finite sample result of  @xcite .
our bound for the se policy is smaller than the aforementioned bounds for the ucb policy by a logarithmic factor .
reference  @xcite did not provide the first bounds on the expected regret .
indeed ,  @xcite and  @xcite had previously derived what is often called _ gap - free _ bound as they hold uniformly over the @xmath88 s .
the second term in our bound is such a gap - free bound .
it is of secondary interest in this paper and arise as a byproduct of refined distribution dependent bound .
nevertheless , it allows us to recover near optimal bounds of the same order as  @xcite .
they depart from optimal rates by a factor @xmath148 as proved in  @xcite .
actually , the result of  @xcite is much stronger than our gap - free bound since it holds for any sequence of bounded rewards , not necessarily drawn independently .
none of the distribution - dependent bounds in corollary [ corstatic ] or the one provided in  @xcite is stronger than the other .
the superiority of one over the other depends on the set @xmath149 : in some cases ( e.g. , if all suboptimal arms have the same expectation ) the latter is the best while in other cases ( if the @xmath88 are spread ) our bounds are better .
this section is dedicated to a detailed description of the nonparametric bandit with covariates .
a _ @xmath4-armed bandit machine with covariates _ ( with @xmath4 an integer greater than 2 ) is characterized by a sequence @xmath150 of independent random vectors , where @xmath151 , is a sequence of i.i.d .
covariates in @xmath152^d$ ] with probability distribution @xmath153 , and @xmath154 denotes the random reward yielded by arm @xmath81 at time @xmath21 . throughout the paper , we assume that @xmath153 has a density , with respect to the lebesgue measure , bounded above and below by some @xmath155 and @xmath156 , respectively .
we denote by @xmath157 the expectation with respect to @xmath153 .
we assume that , for each @xmath158 , rewards @xmath159 , are random variables in @xmath11 $ ] with conditional expectation given by @xmath160 = f^{(i)}(x_t),\qquad i=1,\ldots , k , t=1,2 , \ldots,\ ] ] where @xmath161 , are unknown functions such that @xmath162 , for any @xmath163 .
a natural example is where @xmath154 takes values in @xmath164 so that the conditional distribution of @xmath154 given @xmath165 is bernoulli with parameter @xmath166 .
the _ game _ takes place sequentially on this machine , pulling one of the arms at each time @xmath26 .
@xmath17 is a sequence of random functions @xmath167 indicating to the operator which arm to pull at each time  @xmath21 , and such that @xmath20 depends only on observations strictly anterior to @xmath21 .
the _ oracle policy _
@xmath168 , refers to the strategy that would be run by an omniscient operator with complete knowledge of the functions @xmath169 .
given side information  @xmath165 , the oracle policy @xmath170 prescribes to pull any arm with the largest expected reward , that is , @xmath171 with ties broken arbitrarily .
note that the function @xmath172 is equal to the pointwise maximum of the functions @xmath173 , defined by @xmath174 the oracle rule is used to benchmark any proposed policy @xmath22 and to measure the performance of the latter via its ( _ cumulative _ ) _ regret _ at time @xmath2 defined by @xmath175    without further assumptions on the machine , the game can be arbitrarily difficult and , as a result , expected regret can be arbitrarily close to @xmath2 . in the following subsection , we describe natural regularity conditions under which it is possible to achieve sublinear growth rate of the expected regret , and characterize policies that perform in a near - optimal manner .      as usual in nonparametric estimation we first impose some regularity on the functions @xmath176 . here and in what follows we use @xmath177 to denote the euclidean norm on @xmath178 . smoothness condition .
we say that the machine satisfies the smoothness condition with parameters @xmath179 if @xmath180 is @xmath179-hlder , that is , if @xmath181 for some @xmath182 $ ] and @xmath183 .
now denote the second pointwise maximum of the functions @xmath173 , by @xmath184 ; formally for every @xmath185 such that @xmath186 it is defined by @xmath187 and by @xmath188 otherwise .
notice that a direct consequence of the smoothness condition is that the function @xmath189 is @xmath179-hlder ; however , @xmath190  might not even be continuous .
the behavior of the function @xmath191 critically controls the complexity of the problem and the hlder regularity gives a local upper bound on this quantity .
the second condition gives a lower bound on this function though in a weaker global sense .
it is closely related to the margin condition employed in classification  @xcite , which drives the terminology employed here .
it was originally imported to the bandit setup by  @xcite .
margin condition .
we say that the machine satisfies the margin condition with parameter @xmath192 if there exists @xmath193 , @xmath194 such that @xmath195\le
c_0 \delta^\alpha\qquad \forall\delta\in[0 , \delta_0].\ ] ] if the marginal @xmath153 has a density bounded above and below , the margin condition contains only information about the behavior of the function @xmath196 and not the marginal @xmath153 itself .
this is in contrast with  @xcite where the margin assumption is used precisely to control the behavior of the marginal @xmath153 while that of the reward functions is fixed .
a large value of the parameter @xmath197 means that the function @xmath196 either takes value 0 or is bounded away from 0 , except over a set of small @xmath153-probability .
conversely , for values of @xmath197 close to 0 , the margin condition is essentially void , and the two functions can be arbitrary close , making it difficult to distinguish them .
this reflects in the bounds on the expected regret derived in the subsequent section .
intuitively , the smoothness condition and the margin condition work in opposite directions .
indeed , the former ensures that the function @xmath196 does not `` depart from zero '' too fast whereas the latter warrants the opposite .
the following proposition quantifies the extent of this conflict .
[ propalpha - beta ] under the smoothness condition with parameters@xmath179 , and the margin condition with parameter @xmath197 , the following hold :    * if @xmath198 , then a given arm is either always or never optimal ; in the latter case , it is bounded away from @xmath199 and one can take @xmath200 ; * if @xmath201 , then there exist machines with nontrivial oracle policies .
this proposition is a straightforward consequences of , respectively , the first two points of proposition 3.4 in  @xcite .    for completeness , we provide an example with @xmath202 , @xmath152 $ ] , @xmath203 and @xmath204 .
notice that @xmath205 is @xmath206-hlder if and only if @xmath207 .
any oracle policy is nontrivial , and , for example , one can define @xmath208 if @xmath209 and @xmath210 if @xmath211 .
moreover , it can be easily shown that the machine satisfies the margin condition with parameter @xmath197 and with @xmath212 .
we denote by @xmath213 the class of @xmath4-armed bandit problems with covariates in @xmath152^d$ ] with a machine satisfying the margin condition with parameter @xmath192 , the smoothness condition with parameters @xmath214 and such that @xmath153 has a density , with respect to the lebesgue measure , bounded above and below by some @xmath155 and @xmath156 , respectively .
to design a policy that solves the bandit problem with covariates described above , one has to inevitably find an estimate of the functions @xmath176 , at the current point @xmath165 .
there exists a wide variety of nonparametric regression estimators ranging from local polynomials to wavelet estimators .
both of the policies introduced below are based on estimators of @xmath176 , that are @xmath153 almost surely piecewise constant over a particular collection of subsets , called _
bins _ of the covariate space @xmath215 .
we define a partition of @xmath215 in a measure theoretic sense as a collection of measurable sets , hereafter called _ bins _ , @xmath216 such that @xmath217 , @xmath218 and @xmath219 , @xmath220 , up to sets of null @xmath153 probability . for any @xmath221 and any bin @xmath222 ,
define @xmath223 to define and analyze our policies , it is convenient to reindex the random vectors @xmath224 as follows . given a bin @xmath222 ,
let @xmath225 denote the @xmath226th time at which the sequence @xmath227 is in @xmath222 and observe that it is a stopping time .
it is a standard exercise to show that , for any bin @xmath222 and any arm @xmath81 , the random variables @xmath228 , @xmath229 are i.i.d .
with expectation given by @xmath230 $ ] . as a result , the random variables @xmath231 obtained by successive pulls of arm @xmath81 when @xmath232 form an i.i.d .
sequence in @xmath11 $ ] with expectation given by @xmath230 $ ] .
therefore , if we restrict our attention to observations in a given bin @xmath222 , we are in the same setup as the static bandit problem studied in the previous section .
this observation leads to the notion of _ policy on @xmath222_. more precisely , fix a subset @xmath233 , an integer @xmath234 and recall that @xmath235 is the set of chronological times @xmath21 posterior to @xmath236 at which @xmath232 .
fix @xmath237 and consider the static bandit problem with arms @xmath238 defined in section  [ secstatic ] where successive pulls of arm @xmath239 , at times posterior to @xmath236 , yield rewards @xmath240 that are i.i.d .
in @xmath11 $ ] with mean @xmath230 $ ] . the se policy with parameters @xmath241 on this static problem
is called _ se policy on @xmath222 initialized at time @xmath236 with initial set of arms @xmath238 and parameters @xmath35_.
we first outline a naive policy to operate the bandit machine described in section  [ secdesc ] .
it consists of fixing a partition of @xmath215 and for each set @xmath222 in this partition , to run the se policy on @xmath222 initialized at time @xmath242 with initial set of arms @xmath66 and parameters @xmath241 to be defined below .    the _ binned successive elimination _ ( bse ) policy @xmath243 relies on a specific partition of @xmath215 .
let @xmath244 be the regular partition of @xmath245^d$ ] : the collection of hypercubes defined for @xmath246 by @xmath247 in this paper , all sets are defined up to sets of null lebesgue measure . as mentioned in section  [ subbin ] , the problem can be decomposed into @xmath248 independent static bandit problems , one for each @xmath249 .
denote by @xmath250 the se policy on bin @xmath222 with initial set of arms @xmath66 and parameters @xmath251 .
for any @xmath185 , let @xmath252 denote the bin such that @xmath253 .
moreover , for any time @xmath254 , define @xmath255 to be the number of times before @xmath21 when the covariate fell in bin @xmath222 .
the bse policy @xmath256 is a sequence of functions @xmath257 defined by @xmath258 , where @xmath259 .
it can be implemented according to the pseudo - code of policy  [ algbse ] .
set of arms @xmath34 .
parameters @xmath260 .
@xmath262 initialize a se policy @xmath250 with parameters @xmath251 .
@xmath266 ( observe @xmath267 ) .    the following theorem gives an upper bound on the expected regret of the bse policy in the case where the problem is difficult , that is , when the margin parameter @xmath197 satisfies @xmath268 .
[ thmain ] fix @xmath182 $ ] , @xmath183 and @xmath269 and consider a problem in @xmath270 .
then the bse policy @xmath243 with @xmath271 has an expected regret at time @xmath2 bounded as follows : @xmath272 where @xmath273 is a positive constant that does not depend on @xmath4 .
the case @xmath5 was studied in  @xcite using a similar policy called ucbogram . unlike in  @xcite where suboptimal bounds for the ucb policy are used ,
we use here the sharper regret bounds of theorem  [ thstatic ] and the se policy as a running horse for our policy , thus leading to a better bound than  @xcite .
optimality for the two - armed case is discussed after theorem  [ thmainadap ] .
proof of theorem  [ thmain ] we assume that @xmath274 where the indexing will be made clearer later in the proof . moreover , to keep track of positive constants , we number them @xmath275 . for any real valued function @xmath276 on @xmath215 and any measurable @xmath277 , we use the notation @xmath278 .
moreover , for any @xmath221 , we use the notation @xmath279 .
define @xmath280 , and let @xmath281 be the largest integer such that @xmath282 , where @xmath283 is the constant appearing in the margin condition . if @xmath284 , we have @xmath285 so that the result of the theorem holds when @xmath286 is chosen large enough , depending on the constant @xmath287 . in the rest of the proof , we assume that @xmath288 so that @xmath289 .
recall that the bse policy @xmath256 is a collection of functions @xmath290 that are constant on each @xmath291 .
therefore , the regret of @xmath256 can be decomposed as @xmath292 , where @xmath293 conditioning on the event @xmath294 , it follows from ( [ eqaverage ] ) that @xmath295={\mathbb e}\biggl[\sum _ { s=1}^{n_{j}(n ) } \bigl(\bar f_j^\star -\bar f{}^{(\hat{\pi}_{b_j , s})}_j \bigr ) \biggr],\ ] ] where @xmath296 is defined in ( [ eqdefnbj ] ) ; it satisfies , by assumption , @xmath297\leq\bar{c}nm^{-d}$ ] .
let @xmath298 be the regret associated to a static bandit problem with arm @xmath81 yielding reward @xmath299 and where @xmath300 is the largest average reward .
it follows from the smoothness condition that @xmath301 so that @xmath302    consider _ well - behaved _ bins on which the expected reward functions are well separated .
these are bins @xmath291 with indices in @xmath303 defined by @xmath304 a bin @xmath222 that is not well behaved is called _ strongly ill behaved _ if there is some @xmath305 such that @xmath306 , for all @xmath9 , and _ weakly ill behaved _ otherwise .
strongly and weakly ill behaved bins have indices in @xmath307 and @xmath308 respectively .
note that for any @xmath9 , the function @xmath309 is @xmath310-hlder .
thus for any @xmath311 and any @xmath312 , we have @xmath313 for all @xmath314 so that the inclusion @xmath315 indeed holds .
_ first part _ : _ strongly ill behaved bins in @xmath316 .
_ recall that for any @xmath317 , any arm @xmath318 , and any @xmath319 , @xmath320 .
therefore , @xmath321\\[-8pt ] & \leq & c_1^{1+\alpha } n m^{-\beta(1+\alpha)},\nonumber\end{aligned}\ ] ] where we used the fact that the set @xmath322 does not contribute to the regret .
_ second part _ : _ weakly ill behaved bins in @xmath323 . _ the numbers of weakly ill behaved bins can be bounded using @xmath324 on such a bin ; indeed , the margin condition implies that @xmath325 it yields @xmath326 .
moreover , we bound the expected regret on weakly ill behaved bins using theorem  [ thstatic ] with specific values @xmath327 together with ( [ eqwsfr ] ) , it yields @xmath328.\ ] ]    _ third part _ : _ well - behaved bins in @xmath303 .
_ this part is decomposed into two steps . in the first step ,
we bound the expected regret in a given bin @xmath329 ; in the second step we use the margin condition to control the sum of all these expected regrets .
fix @xmath330 and recall that there exists @xmath331 such that @xmath332 . define @xmath333 and @xmath334 .
we call @xmath335 the set of ( almost ) _ optimal _ arms over @xmath291 and @xmath336 the set of _ suboptimal _ arms over @xmath291 . note that @xmath337 for any @xmath330 .
the smoothness condition implies that for any @xmath338 , @xmath339 therefore , @xmath340 on @xmath291 .
moreover , for any arm @xmath341 that is not the best arm at some @xmath342 , then necessarily @xmath343 .
so for any @xmath314 and any @xmath341 , it holds that either @xmath344 or @xmath345 .
it yields @xmath346 thus , for any optimal arm @xmath341 , the reward functions averaged over @xmath291 satisfy @xmath347 , where @xmath348 together with ( [ eqwsfr ] ) , it yields @xmath349 . for any suboptimal
arms @xmath350 , ( [ eqinaught ] ) implies that @xmath351 .
assume now without loss of generality that the average gaps @xmath352 are ordered in such a way that @xmath353 .
define @xmath354 and observe that if @xmath355 is such that @xmath356 , then @xmath341 .
therefore , it follows from ( [ eqistar ] ) that @xmath357 for such @xmath81 .
applying theorem  [ thstatic ] with @xmath358 as above and @xmath142 , we find that there exists a constant @xmath359 such that , for any @xmath360 , @xmath361 hence , @xmath362    step 2 .
we now use the margin condition to provide lower bounds on @xmath358 for each @xmath330 . assume without loss of generality that the indexing of the bins is such that @xmath363 and that the gaps are ordered @xmath364 . for any @xmath330 , from the definition of @xmath358 , there exists a suboptimal arm @xmath350 such that @xmath365 .
but since the function @xmath366 satisfies the smoothness condition with parameters @xmath367 , we find that if @xmath368 for some @xmath369 , then @xmath370 together with the fact that @xmath371 over @xmath291 for any @xmath330 ( see step 1 above ) , it yields @xmath372\ge\sum _
{ k=1}^{j_1 } p_k { \mathbh1}(0<{\underaccent{\bar } } { \delta}_k \le{\underaccent{\bar}}{\delta}_{j})\ge\frac { { \underaccent{\bar}}{c } j}{m^d},\ ] ] where we used the fact that @xmath373 .
define @xmath374 to be the largest integer such that @xmath375 .
since for any @xmath330 , we have @xmath376 , the margin condition yields for any @xmath377 that @xmath372\le c_\delta(c_1{\underaccent{\bar}}{\delta}_j)^\alpha,\ ] ] where we have used the fact that @xmath378 , for any @xmath379 . the previous two inequalities , together with the fact that @xmath380 for any @xmath330 , yield @xmath381 therefore , using the fact that @xmath382 for @xmath383 , we get from ( [ eqregj ] ) that @xmath384\\[-8pt ] & & \qquad\le c_5 \biggl[\sum _ { j=1}^{j_2 } k \frac{{\overline{\log}}({n}\gamma _ j^2/{m^d } ) } { \gamma_j } + \sum _ { j = j_2 + 1}^{j_1}k\log(n)+ \sum _ { j \in { \mathcal{j } } } nm^{-d-\beta}q_j \biggr].\nonumber\end{aligned}\ ] ]    _ fourth part _ : _ putting things together .
_ combining ( [ eqstronglyill ] ) , ( [ eqweaklyill ] ) and ( [ eqwellbehaved ] ) , we obtain the following bound : @xmath385.\nonumber\end{aligned}\ ] ]    we now bound from above the first sum in ( [ eqreg2 ] ) by decomposing it into two terms . from the definition of @xmath386 , there exists an integer @xmath387 satisfying @xmath388 and such that @xmath389 for @xmath390 and @xmath391 for @xmath392 .
it holds @xmath393 and @xmath394^{{2}/{\alpha } } \biggr ) \nonumber\\[-8pt]\\[-8pt ] & \le & c_{10 } m^d \int_{m^{-\alpha\beta}}^{1 } { \overline{\log}}\biggl(\frac{n}{m^d}x^{{2}/{\alpha } } \biggr)x^{-1/\alpha } \,{\mathrm{d}}x .
\nonumber\end{aligned}\ ] ] since @xmath395 , this integral is bounded by @xmath396 .
the second sum in ( [ eqreg2 ] ) can be bounded as @xmath397\\[-8pt ] & \leq&\frac{m^d}{{\underaccent{\bar}}{c}}{\mathbb p}\bigl\{0 < f^{\star}(x)-f^{\sharp } ( x)\leq c_1m^{-\beta } \bigr \ } \leq\frac{c_1^\alpha}{{\underaccent{\bar}}{c } } m^ { d-\beta\alpha}.\nonumber\end{aligned}\ ] ] putting together ( [ eqreg2])([eqsumqj ] ) , we obtain @xmath398,\end{aligned}\ ] ] and the result follows by choosing @xmath399 as prescribed .
we should point out that the version of the bse described above specifies the number of bins @xmath399 as a function of the horizon @xmath2 , while in practice one may not have foreknowledge of this value .
this limitation can be easily circumvented by using the so - called _ doubling argument _ ( see , e.g. , page 17 in  @xcite ) which consists of `` reseting '' the game at times @xmath400 .
the reader will note that when @xmath401 there is a potentially superfluous @xmath1 factor appearing in the upper bound using the same proof .
more generally , for any @xmath402 , it is possible to minimize the expression in ( [ eqreg2 ] ) with respect to @xmath399 , but the optimal value of @xmath399 would then depend on the value of @xmath197 .
this sheds some light on a significant limitation of the bse which surfaces in this parameter regime : for @xmath2 large enough , it requires the operator to pull each arm at least once in each bin and therefore to incur an expected regret of at least order @xmath248 .
in other words , the bse splits the space @xmath215 in `` too many '' bins when @xmath402 .
intuitively this can be understood as follows .
when @xmath402 , the gap function @xmath403 is bounded away from zero on a large subset of @xmath215 . hence there is no need to carefully estimate it since the optimal arm is the same across the region . as a result
, one could use larger bins in such regions reducing the overall number of bins and therefore removing the extra logarithmic term alluded to above .
we need the following definitions .
assume that @xmath404 and let @xmath405 be the smallest integer such that @xmath406 for any bin @xmath407 , let @xmath408 be the smallest integer such that @xmath409 where @xmath410 is defined in ( [ eqdefu ] ) and @xmath411 .
this definition implies that @xmath412 for @xmath413 , because @xmath414 is decreasing for @xmath415 .
the abse policy operates akin to the bse except that instead of fixing a partition  @xmath416 , it relies on an adaptive partition that is refined over time .
this partition is better understood using the notion of rooted tree .
let @xmath417 be a tree with root @xmath215 and maximum depth @xmath405 .
a node @xmath222 of @xmath417 with depth @xmath418 is a set from the regular partition @xmath419 .
the children of node @xmath420 are given by @xmath421 , defined to be the collection of @xmath422 bins in @xmath423 that forms a partition of @xmath222 .
note that the set @xmath424 of leaves of each subtree @xmath425 of @xmath417 forms a partition of @xmath215 .
the abse policy constructs a sequence of partitions @xmath426 that are leaves of subtrees of @xmath417 . at a given time @xmath26
, we refer to the elements of the current partition @xmath427 as _ live _ bins .
the sequence of partitions is nested in the sense that if @xmath428 , then either @xmath429 or @xmath430 .
the sequence @xmath426 is constructed as follows . in the initialization step ,
set @xmath431 , @xmath432 , and the initial set of arms @xmath433 .
let @xmath434 be a time such that @xmath435 , and let @xmath436 be the collection of sets @xmath222 such that @xmath437 .
we say that the bins @xmath438 are _ born _ at time @xmath21 . for each set @xmath439 , assume that we are given a set of active arms @xmath440 .
note that @xmath441 is such a time with @xmath442 and active arms @xmath443 .
for each born bin @xmath438 , we run a se policy @xmath250 initialized at time @xmath21 with initial set of arms @xmath444 and parameters @xmath445 , @xmath446 .
such a policy is defined in section [ subbin ] .
let @xmath447 denote the time at which @xmath250 has reached @xmath408 rounds and let @xmath448 denote the number of times covariate @xmath165 fell in bin @xmath222 while @xmath222 was a live @xmath222 . at time @xmath449 , we replace the node @xmath222 by its children @xmath421 in the current partition .
namely , @xmath450 .
moreover , to each bin @xmath451 , we assign the set @xmath452 of arms that were left active by policy @xmath250 on its parent @xmath222 at the end of the @xmath408 rounds .
this procedure is repeated until the horizon @xmath2 is reached .
the intuition behind this policy is the following .
the parameters of the se policy @xmath250 run at the birth of bin @xmath222 are chosen exactly such that arms @xmath81 with average gap @xmath453 are eliminated by the end of @xmath408 rounds with high probability .
the smoothness condition ensures that these eliminated arms satisfy @xmath454 for all @xmath305 so that such arms are uniformly suboptimal on bin @xmath222 . among the kept arms ,
none is uniformly better than another , so bin @xmath222 is burst and the process is repeated on the children of @xmath222 where other arms may be uniformly suboptimal .
the formal definition of the abse is given in policy  [ algabse ] ; it satisfies the following theorem .
set of arms @xmath455 .
parameters @xmath456 .
@xmath458 , @xmath459 , @xmath460 .
initialize a se policy @xmath461 with parameters @xmath462 and arms @xmath463 . @xmath464 . @xmath465 . @xmath265 .
@xmath466 ( observe @xmath467 ) . @xmath468
@xmath469 @xmath470initialize se policy @xmath471 with @xmath472 and arms @xmath473 .
@xmath474.@xmath475 @xmath476    [ thmainadap ] fix @xmath182 $ ] , @xmath183 , @xmath192 , assume that @xmath404 and consider a problem in @xmath270 .
if @xmath477 , then the abse policy @xmath478 has an expected regret at time @xmath2 bounded by @xmath479 where @xmath273 does not depend on @xmath4 . if @xmath200 , then @xmath480 .
note that the bounds given in theorem  [ thmainadap ] are optimal in a minimax sense when @xmath5 .
indeed , the lower bounds of @xcite and  @xcite imply that the bound on expected regret can not be improved as a function of @xmath2 except for a constant multiplicative term .
the lower bound proved in  @xcite implies that any policy that received information from _ both _ arms at each round has a regret bound at least as large as the one from theorem  [ thmainadap ] , up to a multiplicative constant . as a result
, there is no price to pay for being in a partial information setup and one could say that the problem of nonparametric estimation dominates the problem associated to making decisions sequentially .
note also that when @xmath200 , proposition [ propalpha - beta ] implies that there exists a unique optimal arm over @xmath215 and that all other arms have reward bounded away from that of the optimal arm . as a result , given this information
, one could operate as if the problem was static by simply discarding the covariates .
theorem  [ thmainadap ] implies that in this case , one recovers the traditional regret bound of the static case without the knowledge that @xmath200 .
proof of theorem  [ thmainadap ] we first consider the case where @xmath481 , which implies that @xmath482 ; see proposition  [ propalpha - beta ] .
we keep track of positive constants by numbering them @xmath483 , yet they might differ from previous sections . on each newly created bin @xmath222 , a new se policy is initialized , and we denote by @xmath484 , the rewards obtained by successive pulls of a remaining arm @xmath81 .
their average after @xmath29 rounds / pulls is denoted by @xmath485 for any integer @xmath226 , define @xmath486 , where @xmath410 is defined in ( [ eqdefu ] ) .    for any @xmath487 , define the unique _ parent _ of @xmath222 by @xmath488 and @xmath489 .
moreover , let @xmath490 and for any @xmath491 define recursively @xmath492 .
then the set of _ ancestors _ of any @xmath493 is denoted by @xmath494 and defined by @xmath495 denote by @xmath496 the regret incurred by the abse policy @xmath478 when covariate @xmath165 fell in a _ live _ bin @xmath497 , where we recall that @xmath427 denotes the current partition at time @xmath21 .
it is defined by @xmath498 { \mathbh1}(x_t \in b){\mathbh1}(b\in{\mathcal{l}}_t).\ ] ] we also define @xmath499 to be the set of bins that were born at some time @xmath500 .
we denote by @xmath501 the regret incurred when covariate @xmath165 fell in such a bin .
it is defined by @xmath502 { \mathbh1}(x_t \in b){\mathbh1}(b \in{\mathcal{b}}_t).\ ] ] observe that if we define @xmath503 , we have @xmath504 since @xmath505 and @xmath506 for all @xmath21 .
note that for any @xmath493 , @xmath507 denote by @xmath508 the set of arms left active by the se policy @xmath250 on @xmath222 at the end of @xmath408 rounds .
moreover , define the following reference sets of arms : @xmath509 @xmath510 define the event @xmath511 on which the remaining arms have a gap of the correct order and observe that ( [ eqdecompborn ] ) implies that @xmath512 let @xmath513 denote the set of leaves of @xmath514 , that is the set of bins @xmath222 such that @xmath515 . in what follows ,
we adapt the convention that @xmath516 .
we are going to treat regret incurred on live nonterminal nodes and live leaves separately and differently . as a result ,
the quantity we are interested in is decomposed as @xmath517 where @xmath518 is the regret accumulated on live nonterminal nodes , and @xmath519 is regret accumulated on live leaves .
our proof relies on the following events : @xmath520 .    _
first part _ : _ control of the regret on the nonterminal nodes . _ fix . on @xmath521 ,
we have @xmath522 so that any active arm @xmath523 satisfies @xmath524 . moreover , regret is only incurred at points where @xmath525 , so defining @xmath526 and conditioning on events @xmath527 yields @xmath528\le{\mathbb e}\bigl[\tilde n_b(n ) \bigr]c_1|b|^\beta q_b \le c_1
k \ell_b |b|^\beta q_b,\ ] ] where @xmath529 and @xmath530 is defined in ( [ eqtildesb ] ) .
we can always assume that @xmath2 is greater than @xmath531 , defined by @xmath532 and let @xmath533 be the smallest integer such that @xmath534 .
indeed , if @xmath535 , the result is true with a constant large enough .
applying the same argument as in ( [ eqsumqj ] ) yields the existence of @xmath536 such that , for any @xmath537 , @xmath538 indeed , for @xmath539 one can define @xmath540 , and the same equation holds with @xmath541 if @xmath542 . summing over all depths @xmath543 , we obtain @xmath544\nonumber\\[-10pt]\\[-10pt ] & & \qquad\leq c_1c_2c_\ell k \sum_{k=0}^{k_0 - 1 } 2^{k(d+\beta-\alpha\beta)}\log \bigl(n2^{-k(2\beta + d ) } \bigr).\nonumber\end{aligned}\ ] ]    on the other hand , for every bin @xmath545 , one also has @xmath546\le c_1 n |b|^\beta q_bp_x(b ) { \mathbb p}\bigl({\mathcal{g}}_b\cap{\mathcal{a}}_b^c\bigr).\ ] ]    it remains to control the probability of @xmath547 ; we define @xmath548 . on @xmath521 ,
the event @xmath549 can occur in two ways :    by eliminating an arm @xmath550 at the end of the at most @xmath408 rounds played on bin @xmath222 .
these arms satisfy @xmath551 ; this event is denoted by @xmath552 .    by not eliminating an arm @xmath553 within the at most @xmath408 rounds played on bin @xmath222 .
these arms satisfy @xmath554 ; this event is denoted by @xmath555 .
we use the following decomposition : @xmath556    we first control the probability of making error ( i ) . note that for any @xmath557 and any arms @xmath558 , it holds @xmath559 therefore , if an arm @xmath550 is eliminated , that is , if there exists @xmath560 such that @xmath561 for some @xmath557 , then either @xmath562 or @xmath563 does not belong to its respective confidence interval @xmath564 $ ] or @xmath565 $ ] for some @xmath557 .
therefore , since @xmath566 , @xmath567 where in the second inequality , we used lemma  [ lempeeling ] .
next , we treat error ( ii ) . for any @xmath553
, there exists @xmath568 such that @xmath569 .
let @xmath570 be any arm such that @xmath571 ; the smoothness condition implies that @xmath572\\[-9pt ] & \ge&\bar f^{(i)}_b + 6c_0|b|^\beta\geq\bar f^{(i)}_b + \tfrac{3}{2 } \varepsilon_{b,\ell _ b}.\nonumber\end{aligned}\ ] ] on the event @xmath573 , no arm in @xmath574 , and in particular any of the arms @xmath575 , has been eliminated until round @xmath408 .
therefore , the event @xmath576 occurs if there exists @xmath577 such that @xmath578 . in view of ( [ eqgapg2 ] ) and ( [ eqlbeps ] )
, it implies that there exists @xmath523 such that @xmath579 hence , the probability of error ( ii ) can be bounded by @xmath580\\[-10pt ] & \leq&2k \frac{\ell _ b}{n|b|^d},\nonumber\end{aligned}\ ] ] where the second inequality follows from ( [ eqhoeffding ] ) .
putting together ( [ eqdecompg ] ) , ( [ eqboundg1 ] ) , ( [ eqboundg2 ] ) and ( [ eqboundlb ] ) , we get @xmath581 together with ( [ eqrn1 ] ) , it yields for @xmath582 that @xmath583\le c_3 k    @xmath584 is such that @xmath585 , then any bin @xmath222 such that @xmath586 satisfies @xmath587\le c_4 k\log n$ ] .
if @xmath584 is such that @xmath588 , then the above display together with the margin condition yield @xmath589 \le c_5 k 2^{k(\beta+d-\alpha\beta)}\log\bigl(n2^{-k(2\beta+d)}\bigr).\ ] ] summing over all depths @xmath418 and using ( [ eqhatrn1 ] ) , we obtain @xmath590\le c_6 k \sum _ { k=0}^{k_0 - 1}2^{k(\beta+d-\alpha\beta)}\log\bigl(n2^{-k(2\beta+d ) } \bigr).\vadjust{\goodbreak}\ ] ] we now compute an upper bound on the right - hand side of the above inequality .
fix @xmath591 and define @xmath592 observe that @xmath593^{-({d+2\beta } ) /({d+\beta-\beta\alpha } ) } \bigr),\ ] ] where @xmath594 . therefore , ( [ eqregnodes ] ) can be rewritten as @xmath595\nonumber\\ & & \qquad\le c_6k \biggl[\sum _ { k=1}^{k_0 - 1 } ( s_{k}-s_{k-1})\log\bigl(n[c_{7 } s_{k}+1]^{- ( { d+2\beta})/({d+\beta-\beta\alpha } ) } \bigr ) + \log n \biggr ] \nonumber \\ & & \qquad\le c_6k \biggl[\int_{0}^{s_{k_0 - 1 } } \log\bigl(n[c_{7 } x+1]^{- ( { d+2\beta})/({d+\beta-\beta\alpha } ) } \bigr ) \,{\mathrm{d}}x + \log n \biggr ] \\ & & \qquad\le c_8k \bigl[2^{k_0(d+\beta-\beta\alpha ) } \log\bigl ( n2^{-k_0(d+2\beta ) } \bigr ) + \log n \bigr ] \nonumber \\ & & \qquad\le c_9 n \biggl(\frac{n}{k\log(k ) } \biggr)^{-{\beta ( 1+\alpha)}/({d+2\beta})},\nonumber\end{aligned}\ ] ] where we used ( [ eqdefk0 ] ) in the last inequality and the fact that @xmath596 is dominated by @xmath597 since @xmath482 .    _
second part _ : _ control of the regret on the leaves .
_ recall that the set of leaves @xmath513 is composed of bins @xmath222 such that @xmath515 . proceeding in the same way as in  ( [ eqrn1 ] )
, we find that for any @xmath598 , it holds @xmath599\le c_1 n |b|^\beta p_x\bigl(0<f^\star - f^\sharp \le c_1 |b|^\beta , x \in b\bigr).\ ] ] since @xmath600 , then @xmath601 and using the margin assumption , we find @xmath602 & \le & c_1 n 2^{-k_0\beta(1+\alpha ) } \nonumber\\[-8pt]\\[-8pt ] & \le & c_1 n \biggl ( \frac{n}{k\log ( k ) } \biggr)^{-{\beta(1+\alpha)}/({d+2\beta})},\nonumber\end{aligned}\ ] ] where we used ( [ eqdefk0 ] ) in the second inequality .
the theorem follows by summing ( [ eqregnodes2 ] ) and ( [ eqregleaves ] ) . if @xmath603 , then the same proof holds except that @xmath596 dominates @xmath604 in  ( [ eqregnodes2 ] ) .
the following lemma is central to our proof of theorem [ thstatic ] .
we recall that a process @xmath605 is a martingale difference sequence if @xmath606=0 $ ] . moreover , if @xmath607 and if we denote the sequence of averages by @xmath608 , then hoeffding
azuma s inequality yields that , for every integer @xmath609 , @xmath610 the following lemma is a generalization of this result :      define @xmath612 .
recall first the hoeffding
azuma maximal concentration inequality . for every @xmath613 and every integer @xmath254
, @xmath614 using a peeling argument , one obtains @xmath615 hence the result .