in many applications where high frequency data are observed , we wish to predict the next values of this time series through an online prediction learning algorithm able to process a large amount of data .
the classical stationarity assumption on the distribution of the observations has to be weakened to take into account some smooth evolution of the environment . from a statistical modelling point of view , this is described by some time varying parameters . in order to sequentially track them from high - frequency data , the algorithms must require few operations and a low storage capacity to update the parameters estimation and the prediction after each new observation .
the most common online methods are least mean squares ( lms ) , normalized least mean squares ( nlms ) , regularized least squares ( rls ) or kalman .
all of them rely on the choice of a gradient step , a forgetting factor , or more generally on a tuning parameter corresponding to some a priori knowledge on how smoothly the local statistical distribution of the data evolves along the time . to adapt automatically to this smoothness , usually unknown in practice
, we propose to use an exponentially weighted aggregation of several such predictors , with various tuning parameters .
we emphasize that to meet the online constraint , we can not use methods that require a large amount of computations ( such as cross validation ) .
the exponential weighting technique in aggregation have been developed in parallel in the machine learning community [ see the seminal paper @xcite ] , in the statistical community [ see @xcite , @xcite ( @xcite ) , @xcite , or more recently @xcite ] and in the game theory community for individual sequences prediction [ see @xcite and @xcite for recent surveys ] .
in contrast to the classical statistical setting , in the individual sequence setting the observations are not assumed to be generated by an underlying stochastic process .
the link between both settings has been analyzed in @xcite for the regression model with fixed and random designs .
exponential weighting has also been investigated in the case of weakly dependent stationary data in @xcite .
more recently , an approach inspired from individual sequences prediction has been studied in @xcite for bounded arma processes under some specific conditions on the ( constant ) arma coefficients .    in this contribution , we consider two possible aggregation schemes based on exponential weights which can be computed recursively .
we provide oracle inequalities applying to the aggregated predictor under the following main assumptions that ( 1 ) the observations are sub - linearly with respect to a sequence of random variables with possibly time varying linear coefficients and ( 2 ) the predictors to be aggregated are lipschitz functions of the past
. an important feature of our observation model is that it embeds the well - known class of _ locally stationary _ processes .
we refer to @xcite and the references therein for a recent general view about statistical inference for locally stationary processes . as an application , we focus on a particular locally stationary model , that of the time varying autoregressive ( tvar ) process .
the minimax rate of certain recursive estimators of the tvar coefficients is studied in @xcite .
to our knowledge , there is not a well - established method on the automatic choice of the gradient step when the smoothness index is unknown .
here , we are interested in the prediction problem which is closely related to the estimation problem .
we show that the proposed aggregation methods provide a solution to this question , in the sense that they give rise to recursive adaptive minimax predictors .
the paper is organized as follows . in section  [ sec : online - aggr - pred ] , we provide oracle inequalities for the aggregated predictors under general conditions applying to nonstationary sub - linear processes .
tvar processes are introduced in section  [ section : time - varying_autoregressive_model ] in a nonparametric setting based on hlder smoothness assumptions on the tvar coefficients .
a  lower bound of the prediction risk is given in this setting and this result is used to show that the proposed aggregation methods achieve the minimax adaptive rate . section  [ subsection : oracle_inequalities ] contains the proofs of the oracle inequalities .
the proof of the lower bound of the minimax prediction risk is presented in section  [ section : proof_lower_bound ] .
numerical experiments illustrating these results are then described in section  [ sec : numer - exper ] .
one and one supplementary material [ @xcite ] complete this paper . and
[ @xcite ( @xcite ) , section  a ] contain some postponed proofs and useful lemmas , [ giraud , roueff and sanchez - perez ( @xcite ) , section  b ] explains how to build nonadaptive minimax predictors which can be used in the aggregation step and [ giraud , roueff and sanchez - perez ( @xcite ) , section  c ] provides additional results with improved aggregation rates .
in this section , we consider a time series @xmath1 admitting the following _ nonstationary _ sub - linear property with respect to the nonnegative process @xmath2 .
[ hyp : gen ]    the process @xmath3 satisfies @xmath4 where @xmath5 are nonnegative coefficients such that @xmath6    additional assumptions will be required on @xmath2 to deduce useful properties for @xmath1 .
note , for instance , that the condition on @xmath7 in  ( [ eq : unif - coeff - bound ] ) guarantees that , if @xmath8 has a uniformly bounded @xmath9-norm , the convergence of the infinite sum in  ( [ eq : gen - series - representation ] ) holds almost surely and in the @xmath9-sense , with both convergences defining the same limit .
it follows that @xmath1 also has uniformly bounded @xmath9 moments .
let us give some particular contexts where the representation  ( m-1 ) can be used .
[ exple : tvlinear ] standard weakly stationary processes such as arma processes [ see @xcite ] admit a wold decomposition of the form @xmath10 where @xmath11 is a weak white noise with , says , unit variance .
this model , sometimes referred to as an ma(@xmath12 ) representation , is often extended to a two - sided sum representation @xmath13 and additional assumptions on the existence of higher moments for @xmath11 or on the independence of the @xmath14 s are often used for statistical inference or prediction ; see @xcite , chapters  7 and  8 . because the sequence @xmath15 may vary with @xmath16 in  ( m-1 ) , we may extend this standard stationary setting and also consider linear processes with time varying coefficients . in this case
, we have @xmath17 where @xmath18 is a sequence of centered independent random variables with unit variance and @xmath19 is supposed to satisfy  ( [ eq : unif - coeff - bound ] ) with @xmath20 , so that   holds with @xmath21 . for this general class of processes , statistical inference is not easily carried out : each new observation @xmath22 comes with a new unknown sequence @xmath23 .
however , additional assumptions on this set of sequences allow to derive and study appropriate statistical inference procedures . a sensible approach in this direction is to consider a _ locally stationary _ model as introduced in @xcite . in this framework ,
the set of sequences @xmath24 is controlled as @xmath25 by artificially ( but meaningfully ) introducing a dependence in @xmath26 , hence is written as @xmath27 , and by approximating it with a set of sequences rescaled on the time interval @xmath28 $ ] , @xmath29 , @xmath30 $ ] , @xmath31 , for example , in the following way : @xmath32 then various interesting statistical inference problems based on @xmath33 can be tackled by assuming some smoothness on the mapping @xmath34 and , possibly , additional assumptions on the structure of the sequence@xmath35 for each @xmath30 $ ] [ see @xcite and the references therein ] .
[ exple : tvar ] a particular instance of example  [ exple : tvlinear ] is the _ time varying autoregressive _ ( tvar ) process , which is assumed to satisfy the recursive equation @xmath36 where @xmath37 is a white noise process ; see @xcite .
it turns out that , in the framework introduced by @xcite , under suitable assumptions , such processes admit a time varying linear representation of the form  ( [ eq : tvlin - rep ] ) ; see @xcite . in section  [ section : time - varying_autoregressive_model ] , we focus on such a class of processes and use the aggregation of predictors to derive adaptive minimax predictors under specific smoothness assumptions on the time varying coefficients .
[ example : non - linear ] it can also be interesting to consider nonlinear extensions of example  [ exple : tvar ] .
a simple example is obtained by setting @xmath38 where @xmath37 is an i.i.d .
sequence and @xmath39 is a time varying sub - linear sequence of functions satisfying , for all @xmath16 that @xmath40 for some @xmath41 .
since @xmath39 is no longer linear but sub - linear , such a model does not enjoy an exact linear representation of the form  ( [ eq : tvlin - rep ] ) .
nevertheless , since we have @xmath42 and iterating this equation backwards yields assumption  ( m-1 ) with @xmath21 and @xmath43 . in the stationary case , where @xmath44 does not depend on @xmath16 ,
a well - known nonlinear extension is the threshold autoregressive model where @xmath45 is piecewise linear ; see @xcite .
our goal in this section is to derive oracle bounds for the aggregation of predictors that hold for the general model  ( m-1 ) with one of the two following additional assumptions on @xmath2 .    the nonnegative process @xmath46 satisfies @xmath47<\infty .
\ ] ]    the nonnegative process @xmath46 is a sequence of independent random variables fulfilling @xmath48 < \infty . \ ] ]    assumptions   and   appear to be quite mild .
as mentioned in example  [ exple : tvlinear ] , basic assumptions in stationary time series usually include moments of sufficiently high order for the innovations and their independence , or rely on the gaussian assumption , which is contained in  .
we also note that , in the context of locally stationary time series , our assumptions on the innovations are weaker than those used in the recent works @xcite , @xcite ( @xcite ) .
precise comparisons between our assumptions and usual ones in the aggregation literature will be given after corollary  [ corollary : oracle - bounds ] .
let @xmath49 be a real valued sequence .
we say that @xmath50 is a predictor of @xmath51 if it is a measurable function of @xmath52 . throughout this paper ,
the quality of a sequence of predictors @xmath53 is evaluated for some @xmath54 using the @xmath55 loss averaged over the time period @xmath56 @xmath57 now , given a collection of @xmath58 sequences of predictors @xmath59 , we wish to sequentially derive a new predictor which predicts almost as accurately as or more accurately than the best of them .    in the present paper and for our purposes , aggregating the predictors amounts to compute a convex combination of them at each time @xmath16 .
this corresponds to choosing at each time @xmath16 an element @xmath60 of the simplex @xmath61 and compute @xmath62}=\sum_{i=1}^n \alpha_{i , t } \widehat{x}_t^{(i ) } . \ ] ] we consider two strategies of aggregation , which are studied in the context of bounded sequences in @xcite .
more recent contributions and extensions can be found in @xcite .
see also @xcite for a pedagogical .
these strategies are sequential and online , meaning that :    to compute the aggregation weights @xmath60 at time @xmath16 , only the values of @xmath63 and @xmath64 up to time @xmath65 are used ,    the computation can be done recursively by updating a set of quantities , the number of which does not depend on @xmath16 .
these two properties are met in the algorithm  [ algo : strat12 ] detailed below .
we consider in the remaining of the paper a convex aggregation of predictors @xmath66}= \sum   _ { i=1}^{n}\widehat{\alpha}_{i , t}\widehat { x}_{t}^{(i)},\qquad1\leq t\leq t , \ ] ] with some specific weights @xmath67 defined as follows .      the first strategy is to define for all @xmath68 and @xmath69 , the weights @xmath70 by @xmath71 with the convention that a sum over no element is zero , so @xmath72 for all @xmath73 .
the parameter @xmath74 , usually called the _ learning rate _ , will be specified later .
the second strategy is to define for all @xmath68 and @xmath69 , the weights @xmath70 by @xmath75 with again the convention that a sum over no element is zero .
both strategies yield the same algorithm up to the line where @xmath76 is computed .
for sake of brevity , we write only one algorithm ( see algorithm  [ algo : strat12 ] ) and use a switch / case statement to distinguish between the two strategies . note , however , that the choice of the strategy ( @xmath77 or @xmath78 ) holds for the whole sequence of predictions .
we establish oracle bounds on the average prediction error of the aggregated predictors .
these bounds ensure that the error is equal to that associated with the best convex combination of the predictors or with the best predictor ( depending on the aggregation strategy ) , up to two remaining terms .
one remaining term depends on the number @xmath58 of predictors to aggregate and the other one on the _ variability _ of the original process .
the learning rate @xmath79 can then be chosen to achieve the best trade - off between these two terms .
the second remaining term indirectly depends on the variability of the predictors .
we control below this variability in terms of the variability of the original process by using the following lipschitz property .
[ def : unif - lip - pred ] let @xmath80 be a sequence of nonnegative numbers . a  predictor @xmath50 of @xmath81 from @xmath52 is said to be @xmath82-lipschitz if @xmath83    we more specifically consider a sequence @xmath82 satisfying the following assumption .
the sequence @xmath80 satisfies @xmath84    this condition is trivially satisfied by constant linear predictors depending only on a finite number of previous observations , that is , @xmath85 . in @xcite
[ ( @xcite ) , section  b.1 ] , we extend this case in the context of the tvar process where the coefficients @xmath86 are replaced by estimates of the time varying autoregressive coefficients .
more generally , assumption   appears to be quite natural in the general context where @xmath87=f_t((x_{t - s})_{s\geq1})$ ] , where @xmath88 is a lipschitz function from @xmath89 to @xmath90 , with lipschitz coefficients satisfying a condition similar to  ( [ eq : hyp : lip ] ) ; see , for instance , @xcite in the case of stationary time series .
we now state two upper - bounds on the mean quadratic prediction error of the aggregated predictors defined in the previous section , when the process @xmath91 fulfills the sub - linear property  ( m-1 ) .
[ thm : oracle - bounds ] assume that assumption   holds .
let @xmath92 , @xmath93 be a collection of sequences of @xmath82-lipschitz predictors with @xmath82 satisfying  .    assume that the noise @xmath94 fulfills   with @xmath95 and let @xmath96 denote the aggregated predictor obtained using the weights  ( [ definition : alphas_gradient_quadratic_loss ] ) with any @xmath74 .
then we have @xmath97 & \leq&\inf_{\nu\in\mathcal{s}_n } { 1\over t}\sum _ { t=1}^t{\mathbb{e}}\bigl [ \bigl(\widehat x_t^{[\nu]}-x_{t } \bigr)^2 \bigr ] \nonumber\\[-9pt]\\[-9pt]\nonumber & & { } + { \log n\over t\eta}+2\eta ( 1+l_*)^{4 } a_{*}^4 \m_{4 } .\end{aligned}\ ] ]    assume that the noise @xmath94 satisfies   with a given @xmath98 and let @xmath96 denote the aggregated predictor obtained using the weights ( [ definition : alphas_quadratic_loss ] ) with any @xmath74 .
then we have @xmath99 & \leq & \min_{1\leq i\leq n } { 1\over t}\sum _ { t=1}^t{\mathbb{e}}\bigl [ \bigl(\widehat x_{t}^{(i)}-x_{t } \bigr)^2 \bigr ] \nonumber\\[-10pt]\\[-10pt]\nonumber & & { } + \frac{\log n}{t\eta } + ( 2\eta)^{p/2 - 1}a_{*}^{p}(1+l_*)^{p}m_{p } .\end{aligned}\ ] ]    assume that the noise @xmath94 fulfills   for some positive @xmath100 and let @xmath96 denote the aggregated predictor obtained using the weights  ( [ definition : alphas_quadratic_loss ] ) with @xmath74 .
then , for any @xmath101\qquad\mbox{with } a^ * : = \sup _ { j\in{\mathbb{z}}}\sup_{t\in{\mathbb{z } } } a_t(j )   \leq    a _ { * } , \ ] ]
we have @xmath102\nonumber \\ & & \qquad \leq\min_{1\leq i\leq
n } { 1\over t}\sum
_ { t=1}^t{\mathbb{e}}\bigl [ \bigl(\widehat x_{t}^{(i)}-x_{t } \bigr)^2 \bigr ] \\ & & \quad\qquad { } + \frac{\log n}{t\eta } + \frac{2}{{\mathrm{e}}}\lambda^{-2 } \bigl(2+\lambda(2 \eta)^{-1/2 } \bigr )   { \mathrm{e}}^{- \lambda(2\eta)^{-1/2 } }   \bigl(\mphi(\zeta ) \bigr)^{\lambda a_*(1+l_*)/\zeta } .\nonumber\end{aligned}\ ] ]    the proof can be found in section  [ sec : proof - theor - strat12 ] .
[ rem : oracle - bounds - explicites ] the bounds  ( [ eq : strat1-final ] ) ,  ( [ eq : strat2_p - final ] ) and ( [ eq : strat2_exp - final ] ) are explicit in the sense that all the constants appearing in them are directly derived from those appearing in assumptions  ( m-1 ) , , and  .
the following corollary is obtained by choosing @xmath79 [ and @xmath103 in the case  ( iii ) ] adequately in the three cases of theorem  [ thm : oracle - bounds ] .
[ corollary : oracle - bounds ] assume that assumption   holds .
let @xmath92 , @xmath93 be a collection of sequences of @xmath82-lipschitz predictors with @xmath82 satisfying  .
assume that the noise @xmath94 fulfills   with @xmath95 and let @xmath96 denote the aggregated predictor obtained using the weights  ( [ definition : alphas_gradient_quadratic_loss ] ) with @xmath104 this gives @xmath105 \leq\inf_{\nu\in\mathcal{s}_n } { 1\over t}\sum _ { t=1}^t{\mathbb{e}}\bigl [ \bigl(\widehat x_t^{[\nu]}-x_{t } \bigr)^2 \bigr ] + c_1   \biggl(\frac{\log n}{t } \biggr)^{1/2 } , \ ] ] with @xmath106 .
assume that the noise @xmath94 satisfies   with a given @xmath107 and let @xmath96 denote the aggregated predictor obtained using the weights  ( [ definition : alphas_quadratic_loss ] ) with @xmath108 we then have @xmath109 \nonumber\\[-8pt]\\[-8pt]\nonumber & & \qquad \leq\min_{1\leq i\leq n } { 1\over t}\sum
_ { t=1}^t{\mathbb{e}}\bigl [ \bigl(\widehat x_{t}^{(i)}-x_{t } \bigr)^2 \bigr ] + c_2   \biggl(\frac{\log n}{t } \biggr)^{1 - 2/p } , \end{aligned}\ ] ] with @xmath110 .
assume that the noise @xmath94 fulfills   for some positive @xmath100 and let @xmath96 denote the aggregated predictor obtained using the weights  ( [ definition : alphas_quadratic_loss ] ) with @xmath111 then we have @xmath112\nonumber \\ & & \qquad \leq\min_{1\leq i\leq
n } { 1\over t}\sum
_ { t=1}^t{\mathbb{e}}\bigl [ \bigl(\widehat x_{t}^{(i)}-x_{t } \bigr)^2 \bigr]+ \frac{2a_{*}^2(l_*+1)^2}{\zeta^2 }   \frac{\log n}{t } \\ & & \quad\qquad { }   \times \biggl\ { \biggl(\log\biggl ( \frac{t}{\log n } \biggr ) \biggr)^{2 } + \frac{\phi(\zeta)}{\mathrm{e}}\biggl(2 + \log\biggl(\frac{t}{\log n } \biggr ) \biggr ) \biggr\ } .\nonumber\end{aligned}\ ] ] [ note that when @xmath113 , the term between curly brackets is equivalent to @xmath114 . ]    cases  ( i ) and  ( ii ) in corollary  [ corollary : oracle - bounds ] follow directly from theorem  [ thm : oracle - bounds ] .
case  ( iii ) is more delicate since it requires optimizing @xmath103 as well as @xmath115 in the second line of  ( [ eq : strat2_exp - final ] ) .
the details are postponed to section  [ sec : proof - corollary - strat12 ] .
we observe that the bound in  ( [ eq : strat2_exp - final - opt - eta ] ) improves that in ( [ eq : strat2_p - final - eta - opt ] ) for any @xmath98 . for @xmath116 , the remaining term @xmath117 in  ( [ eq : strat2_p - final - eta - opt ] )
is smaller than the remaining term @xmath118 in  ( [ eq : strat1-final - eta - opt ] ) .
similarly , the remaining term @xmath119 in  ( [ eq : strat2_exp - final - opt - eta ] ) is smaller than @xmath118 in  ( [ eq : strat1-final - eta - opt ] ) .
yet , we emphasize that the oracle inequalities  ( [ eq : strat2_p - final - eta - opt ] ) and  ( [ eq : strat2_exp - final - opt - eta ] ) compare the prediction risk of @xmath120 to the prediction risk of the _ best predictor _ @xmath121 , while the oracle inequality  ( [ eq : strat1-final - eta - opt ] ) compare the prediction risk of @xmath120 to the prediction risk of the _ best convex combination of the predictors _ @xmath122 , so they can not be directly compared .
[ improve - with - net ] as explained in @xcite [ ( @xcite ) , section  c ] , under the hypotheses of cases  ( ii ) and  ( iii ) and for certain values of @xmath26  and  @xmath58 , using a more involved aggregation step , we can get a new predictor satisfying an oracle inequality better than that in  ( [ eq : strat1-final - eta - opt ] ) .
for example , under the hypotheses of case  ( iii ) , for @xmath123 , the remaining term @xmath118 in  ( [ eq : strat1-final - eta - opt ] ) can be replaced by @xmath124 which is smaller ; see @xcite [ ( @xcite ) , inequality  ( c.7 ) , page  8 ] . yet , this aggregation has a prohibitive computational cost and seems difficult to implement in practice .
[ rem : eta - choice - vs - conditions ] in cases  ( ii ) and  ( iii ) , which correspond to the weights  ( [ definition : alphas_quadratic_loss ] ) , the choice of the optimal @xmath79 depends on the assumptions on the noise , namely or . under a moment condition of order @xmath125 ,
the optimal @xmath79 is of order @xmath126 and under an exponential condition , it is of order @xmath127 .
it is known from @xcite [ ( @xcite ) , proposition  2.2.1 ] and @xcite [ ( @xcite ) , theorem  5 ] that @xmath79 can be chosen as a constant ( provided that it is small enough ) under a bounded noise condition , or under an exponential moment condition on the noise for predictors at a bounded distance from the conditional mean .
hence , coarsely speaking , the heavier the tail of the noise , the smallest @xmath79 should be chosen . observing that @xmath79 allows us to tune the influence of the empirical risk on the weights from no influence at all ( @xmath128 yielding uniform weights ) to the selection of the empirical risk minimizer ( @xmath129 )
, the specific choices of @xmath79 can be interpreted as follows : the heavier the tail of the noise , the less we can trust the empirical risk .      in the literature , prediction risk bounds of the form  ( [ eq : strat1-final - eta - opt ] ) [ case  ( i ) of corollary  [ corollary : oracle - bounds ] ] are sometimes called _
convex regret bounds _ , and prediction risk bounds of the form  ( [ eq : strat2_p - final - eta - opt ] ) and  ( [ eq : strat2_exp - final - opt - eta ] ) [ cases  ( ii ) and  ( iii ) of corollary  [ corollary : oracle - bounds ] ] are sometimes called _
best predictor regret bounds_.    @xcite exhibits convex regret bounds in a setting close to ours , namely for an online aggregation of predictors for a sequence of possibly dependent random variables . under our moment condition   with @xmath95 , @xcite [ ( @xcite ) , theorem  2 ] provides an upper bound similar to  ( [ eq : strat1-final - eta - opt ] ) but with our remaining term @xmath130 replaced by @xmath131 . under the exponential condition  , @xcite [ ( @xcite ) ,
theorem  1 ] provides an upper bound similar to  ( [ eq : strat1-final - eta - opt ] ) but with a remaining term @xmath132 , which is still larger than our remaining term under moment conditions .
best predictor regret bounds can be found in @xcite for some sequences of possibly dependent random variables .
the predictors are assumed to remain at a bounded distance to the conditional means and the scaled innovation noise is assumed to have either a known distribution ( satisfying a certain technical condition ) or an exponential moment .
the regret bounds are presented in a slightly different fashion from ours but it is easy to see that a similar result as our bound  ( [ eq : strat2_exp - final - opt - eta ] ) is obtained in this setting .
however , we do not require bounded prediction errors and our conditions on the noise are milder .    the i.i.d .
setting has received much more attention and , even if the setting is quite different , it is interesting to briefly compare our results to previous works in this case .
let us start with the convex regret bound in case  ( i ) of corollary  [ corollary : oracle - bounds ] .
most of the existing results [ see , e.g. , @xcite , @xcite , @xcite or @xcite for recent extensions to @xmath133 aggregation ] assume the predictors to be bounded and various conditions on the noise are considered ( very often the noise is assumed to be gaussian ) . in such settings ,
the best possible remaining term typically takes the form @xmath118 when @xmath58 is much larger than @xmath134 and of the form @xmath135 if @xmath58 is smaller than @xmath134 ; see @xcite [ ( @xcite ) , theorem  3.1 ] , @xcite [ ( @xcite ) , theorem  6 ] and @xcite [ ( @xcite ) , theorem  2 ] . hence , our bound  ( [ eq : strat1-final - eta - opt ] ) is similar only in the case where @xmath58 is much larger than @xmath134 .
however , as explained in remark  [ improve - with - net ] and [ giraud , roueff and sanchez - perez ( @xcite ) , section  c ] , when @xmath26 is larger than @xmath136 and under the moment condition  , we can get via a more involved aggregation procedure , a convex regret bound with a remaining term of the same order @xmath135 up to a @xmath137 factor [ see @xcite , inequality ( c.7 ) , page  8 ] .
let us now compare our bound  ( [ eq : strat2_p - final - eta - opt ] ) in case  ( ii ) to optimal bounds in the i.i.d .
setting under moment conditions on the noise .
corollary  7.2 and theorem  8.6 in  @xcite shows that the optimal aggregation rate is @xmath138 in the i.i.d . setting with bounded predictors and moment conditions of order @xmath125 on the noise .
our remaining term @xmath117 in  ( [ eq : strat2_p - final - eta - opt ] ) is slightly larger , yet an inspection of the proof of @xcite [ ( @xcite ) , corollary  7.2 ] shows that the aggregation rate would also be @xmath117 in this corollary , if the predictors were assumed to have a moment condition of order @xmath125 instead of being uniformly bounded ( we are not aware of any lower bound in this setting matching this rate ) . finally , when the data and the predictors are bounded , the best aggregation rate is known to be @xmath139 in the i.i.d .
setting ; see , for example , @xcite , theorem  8.4 .
our bound  ( [ eq : strat2_exp - final - opt - eta ] ) in case  ( iii ) achieves the same rate up to a @xmath140 factor .
we introduce some preliminary notation before defining the model . in the remainder of this article , vectors are denoted using boldface symbols and @xmath141 denotes the euclidean norm of @xmath142 , @xmath143 .    for @xmath144 $ ] and an interval @xmath145 ,
the @xmath146-hlder semi - norm of a function @xmath147 is defined by @xmath148 this semi - norm is extended to any @xmath149 as follows .
let @xmath150 and @xmath151 $ ] be such that @xmath152 .
if @xmath153 is @xmath154 times differentiable on @xmath155 , we define @xmath156 and @xmath157 otherwise .
we consider the case @xmath158 $ ] . for @xmath159 and @xmath149 , the @xmath160-hlder ball
is denoted by @xmath161\rightarrow{\mathbb{r}}^{d } , \mbox { such that } { \vert}\mathbf{f}{\vert}_{\beta } \leq r \bigr\ }   . \ ] ]      the idea of using a rescaled time with the sample size @xmath26 for the tvar parameters goes back to  @xcite .
since then , it has always been a central example of locally stationary linear processes . in this
setting , the time varying autoregressive coefficients and variance which generate the observations @xmath162 for @xmath163 are represented by functions from @xmath28 $ ] to @xmath164 and from @xmath28 $ ] to @xmath165 , respectively .
the definition sets of these functions are extended to @xmath166 $ ] in the following definition .
[ definition : tvar ] let @xmath167 .
let @xmath168 and @xmath169 be functions defined on @xmath166 $ ] and @xmath11 be a sequence of i.i.d .
random variables with zero mean and unit variance . for any @xmath54
, we say that @xmath170 is a tvar process with time varying parameters @xmath171 sampled at frequency @xmath172 and normalized innovations @xmath18 if the two following assertions hold :    the process @xmath91 fulfills the time varying autoregressive equation @xmath173    the sequence @xmath170 is bounded in probability , @xmath174    this definition extends the usual definition of tvar processes , where the time varying parameters @xmath168 and @xmath175 are assumed to be constant on @xmath176 ; see , for example , @xcite [ ( @xcite ) , page  144 ] .
the tvar model is generally used for the sample @xmath177 .
the definition of the process for negative times @xmath16 can be seen as a way to define initial conditions for @xmath178 , which are then sufficient to compute @xmath177 by iterating  ( [ equation : definition_tvar ] ) .
however , in the context of prediction , it can be useful to consider predictors @xmath179 which may rely on historical data @xmath180 arbitrarily far away in the past , that is , with @xmath181 tending to @xmath182 . to cope with this situation , our definition of the tvar process @xmath183 holds for all time indices @xmath184 and we use the following definition for predictors .
[ def : predictor ] for all @xmath163 , we say that @xmath179 is a predictor of @xmath162 if it is @xmath185-measurable , where @xmath186 is the @xmath169-field generated by @xmath187 . for any @xmath54 , we denote by @xmath188 the set of sequences @xmath189 of predictors for @xmath177 , that is , the set of all processes @xmath189 adapted to the filtration @xmath190 .    in this general framework
, the time @xmath191 corresponds to the beginning of the aggregation procedure .
such a framework applies in two practical situations . in the first one , we start collecting data @xmath162 at @xmath192 and
compute several predictors @xmath193 , @xmath194 from them .
thus , the resulting aggregated predictor only depends on @xmath195 .
a somewhat different situation is when historical data is available beforehand the aggregation step , so that a given predictor @xmath193 is allowed to depend also on data @xmath180 with @xmath196 , while the aggregation step only starts at @xmath192 , and thus depends on the data @xmath197 only through the predictors .
it is important to note that , in contrast to the usual stationary situation , having observed the process @xmath180 for infinitely many @xmath181 s in the past ( for all @xmath198 ) is not so decisive for deriving a predictor of @xmath162 , since observations far away in the past may have a completely different statistical behavior .
the next proposition proves that under standard stability conditions on the time varying parameters @xmath168 and @xmath175 , condition  ( ii ) in definition  [ definition : tvar ] ensures the existence and uniqueness of the solution of equation  ( [ equation : definition_tvar ] ) for @xmath199 ( and thus for all @xmath200 ) .
we define the time varying autoregressive polynomial by @xmath201 let us denote , for any @xmath202 , @xmath203\rightarrow { \mathbb{r}}^{d},{\bolds\theta}(z;u)\neq0 , \forall{\vert}z{\vert } < \delta^{-1 } , u\in[0,1 ] \bigr\ } .\ ] ]    define , for @xmath149 , @xmath159 , @xmath204 , @xmath205 $ ] and @xmath206 , the class of parameters @xmath207\to{\mathbb{r}}^d\times[\rho\sigma_+ , \sigma_+]:\bolds{\theta}\in\lambda_d(\beta , r)\cap s_d(\delta ) \bigr\ } .\end{aligned}\ ] ]    the definition of the class @xmath208 is very similar to that of @xcite .
the domain of definition in their case is @xmath28 $ ] whereas it is @xmath166 $ ] in ours .
we have the following stability result .
[ proposition : stationary_tvar ] assume that the time varying ar coefficients @xmath209 are uniformly continuous on @xmath166 $ ] and the time varying variance @xmath210 is bounded on @xmath166 $ ] .
assume moreover that there exists @xmath204 such that @xmath211 .
then there exists @xmath212 such that , for all @xmath213 , there exists a unique process @xmath170 which satisfies   and   in definition  [ definition : tvar ] .
this solution admits the linear representation @xmath214 where the coefficients @xmath215 satisfy that for any @xmath216 , @xmath217 moreover , if @xmath218 for some positive constants @xmath146 , @xmath219 and @xmath220 , then the constants @xmath221 and @xmath222 can be chosen only depending on @xmath223 , @xmath224 , @xmath146 and @xmath219 .
a proof of proposition  [ proposition : stationary_tvar ] is provided in .
this kind of result is classical under various smoothness assumptions on the parameters and initial conditions for @xmath225 , @xmath226 .
for instance , in @xcite , bounded variations and a constant @xmath227 for negative times are used for the smoothness assumption on @xmath227 and for defining the initial conditions .
the linear representation  ( [ eq : inf - series - representation ] ) of tvar processes was first obtained in the seminal papers @xcite .
we note that an important consequence of proposition  [ proposition : stationary_tvar ] is that for any @xmath213 , the process @xmath170 satisfies assumption  ( m-1 ) with @xmath21 and @xmath228 for @xmath229 .
moreover , the constant @xmath7 in  ( [ eq : unif - coeff - bound ] ) is bounded independently of @xmath26 , and we have , for all @xmath218 , @xmath230 where @xmath231 and @xmath232 can be chosen only depending on @xmath224 , @xmath146 and @xmath219 .      based on proposition  [ proposition : stationary_tvar ] , given an i.i.d .
sequence @xmath37 and constants @xmath204 , @xmath205 $ ] , @xmath206 , @xmath149 and @xmath159 , we consider the following assumption .    the sequence @xmath170 is a tvar process with time varying standard deviation @xmath169 , time varying ar coefficients @xmath168 and innovations @xmath11 , and @xmath233 .
let @xmath234 denote a generic random variable with the same distribution as the @xmath235 s .
under assumption  ( m-2 ) , the distribution of @xmath236 only depends on that of @xmath237 and on the functions @xmath227 and @xmath169 . for a given distribution @xmath238 on @xmath90 for @xmath234 ,
we denote by @xmath239 the probability distribution of the whole sequence @xmath170 and by @xmath240 its corresponding expectation .
the next two assumptions on the innovations are useful to prove upper bounds of the prediction error .
[ hyp : innov - moment ]    the innovations @xmath11 satisfy @xmath241<\infty$ ] .    the innovations @xmath242
satisfy @xmath243<\infty$ ] .
the following one will be used to obtain a lower bound .
[ hyp : innov - lowerbound ]    the innovations @xmath11 admit a density @xmath244 such that @xmath245    assumption  ( i-3 ) is standard for proving lower bounds in nonparametric regression estimation , see @xcite , chapter  2 .
it is satisfied by gaussian density with @xmath246 .
the setting of definition  [ definition : tvar ] and of assumptions derived thereafter is essentially nonparametric , since for given initial distribution  @xmath238 , the distribution of the observations @xmath247 are determined by the unknown parameter function @xmath248 .
the doubly indexed @xmath162 refers to the fact that this distribution can not be seen as a distribution on @xmath249 marginalized on @xmath250 as the usual time series setting but rather as a sequence of distributions on @xmath250 indexed by @xmath26 .
it corresponds to the usual nonparametric approach for studying statistical inference based on this model . in this contribution , we focus on the prediction problem , which is to answer the question : for given smoothness conditions on @xmath248 , what is the mean prediction error for predicting @xmath162 from its past ?
the standard nonparametric approach is to answer this question in a minimax sense by determining , for a given sequence of predictors @xmath251 , the maximal risk @xmath252\\[-8pt]\nonumber & & \qquad = \sup_{({\bolds\theta},\sigma ) } { \frac{1}{t } } \sum_{t=1}^{t } \biggl({\mathbb{e}}^{\psi } _ { ( { \bolds\theta},\sigma ) } \bigl [ ( \widehat{x}_{t , t}-x_{t , t } ) ^{2 } \bigr ] - \sigma^{2 } \biggl ( { \frac{t}{t } } \biggr ) \biggr ) , \end{aligned}\ ] ] where :    @xmath253 is assumed to belong to @xmath188 as in definition  [ def : predictor ] ,    the sup is taken over @xmath254 within a smoothness class of functions ,    the expectation @xmath255 is that associated to assumption  ( m-2 ) .
the reason for subtracting the average @xmath256 over all @xmath163 in this prediction risk is that it corresponds to the best prediction risk , would the parameters @xmath248 be exactly known .
we observe that dividing @xmath162 by the class parameter @xmath220 amounts to take @xmath257 .
in addition , we have @xmath258 so the prediction problem in the class @xmath259 can be reduced to the prediction problem in the class @xmath260 .
accordingly , we define the reduced minimax risk by @xmath261    in section  [ section : lower_bound ] , we provide a lower bound of the minimax rate in the case where the smoothness class is of the form @xmath259 .
then , in section  [ subsection : minimax ] , relying on the aggregation oracle bounds of section  [ subsection : general : bound ] , we derive an upper bound with the same rate as the lower bound using the same smoothness class of the parameters . moreover , we exhibit an online predictor which does not require any knowledge about the smoothness class and which is thus minimax adaptive . in other words
, it is able to adapt to the unknown smoothness of the parameters from the data . to our knowledge ,
such theoretical results are new for locally stationary models .
a lower bound on the minimax rate for the estimation error of @xmath227 is given by @xcite [ ( @xcite ) , theorem  4 ] .
clearly , a  predictor @xmath262 can be defined from an estimator @xmath263 , and the resulting prediction rate can be controlled using the estimation rate ( see @xcite [ ( @xcite ) , section  b.1 ] for the details ) .
the next theorem provides a lower bound of the minimax rate of the risk of _ any _ predictor of the process @xmath177 . combining this result with [ @xcite ( @xcite ) , lemma  9 ] ,
we show that a predictor obtained by [ giraud , roueff and sanchez - perez ( @xcite ) , equation  ( b.1 ) ] from a minimax rate estimator of @xmath227 automatically achieves the minimax prediction rate .
[ theorem : lower_bound_quadratic_risk ] let @xmath204 , @xmath149 , @xmath159 and @xmath205 $ ] .
suppose that assumption   holds and assume on the distribution @xmath238 of the innovations .
then we have @xmath264 where @xmath265 is defined in  ( [ eq : reduced - minimax - risk ] ) .
the proof is postponed to section  [ section : proof_lower_bound ] .      in @xcite , an adaptive estimator of the autoregressive function of a gaussian tvar process of order 1
is studied .
it relies on the lepski s procedure [ see @xcite ] , which seems difficult to implement in an online context .
our minimax adaptive predictor is based on the aggregation of sufficiently many predictors , assuming that at least one of them converges at the minimax rate .
the oracle bounds found in section  [ subsection : general : bound ] imply that the aggregated predictor is minimax rate adaptive under appropriate assumptions .
seminal works using the aggregation to adapt to the minimax convergence rate are @xcite ( nonparametric regression ) and @xcite ( density estimation ) ; see also @xcite for a more general presentation .    in the tvar model  ( m-2 ) , it is natural to consider @xmath82-lipschitz predictors @xmath266 of @xmath177 with a sequence @xmath82 supported on @xmath267
. then @xmath268 in  ( [ eq : hyp : lip ] ) corresponds to the maximal @xmath0-norm of the tvar parameters . since for the process itself to be stable , this norm has to be bounded independently of  @xmath26 , condition   is a quite natural assumption for the tvar model ; see @xcite [ ( @xcite ) , section  b.1 ] for the details .
a practical advantage of the proposed procedures is that , given a set of predictors that behaves well under specific smoothness assumptions , we obtain an aggregated predictor which performs almost as well as or better than the best of these predictors , hence which behaves well without any prior knowledge on the smoothness of the unknown parameter .
such an adaptive property can be formally demonstrated by exhibiting an adaptive minimax rate for the aggregated predictor which coincides with the lower bound given in theorem  [ theorem : lower_bound_quadratic_risk ] .
the first ingredient that we need is the following .
[ def : minimax - rate - expert ] let @xmath238 be a distribution on  @xmath90 and @xmath149 .
we say that @xmath269 is a @xmath270-minimax - rate sequence of predictors if , for all @xmath54 , @xmath271 and , for all @xmath204 , @xmath159 , @xmath272 $ ] and @xmath206 , @xmath273 where @xmath274 is defined by  ( [ eq : max - risk ] ) .
the term _ minimax - rate _ in this definition refers to the fact that the maximal rate in  ( [ equation : definition_minimax ] ) is equal to the minimax lower bound  ( [ eq : lower - bound - minimax ] ) for the class @xmath259 .
we explain in @xcite [ ( @xcite ) , section  b ] how to build such predictors which are moreover @xmath82-lipschitz for some @xmath82 only depending on @xmath275 . to adapt to an unknown smoothness
, we rely on a collection of @xmath276-minimax - rate predictors with @xmath146 within @xmath277 , where @xmath278 is the ( possibly infinite ) maximal smoothness index .
[ def : minimax - rate - expert - loc - bounded ] let @xmath238 be a distribution on @xmath90 and @xmath279 $ ] .
we say that @xmath280 is a locally bounded set of @xmath238-minimax - rate predictors if for each @xmath146 , @xmath281 is a @xmath276-minimax - rate predictor and if moreover , for all @xmath204 , @xmath159 , @xmath272 $ ] , @xmath282 and for each closed interval @xmath283 , @xmath284 where @xmath274 is defined by  ( [ eq : max - risk ] ) .
the following lemma shows that , given a locally bounded set of minimax - rate predictors , we can always pick a finite subset of at most @xmath285 predictors among which the best one achieves the minimax rate of any unknown smoothness index .
[ lemma : minimax_experts ] let @xmath238 be a distribution on @xmath90 .
let @xmath286 $ ] and @xmath287 be a corresponding locally bounded set of @xmath238-minimax - rate predictors .
set , for any @xmath288 , @xmath289 suppose moreover , in the case where @xmath290 , that @xmath291 , and , in the case where @xmath292 , that @xmath293 .
then we have , for all @xmath294 , @xmath204 , @xmath159 , @xmath295 and @xmath206 , @xmath296    the proof of this lemma is postponed to @xcite [ ( @xcite ) , section  a.8 ] .
lemma  [ lemma : minimax_experts ] says that to obtain a minimax - rate predictor which adapts to an unknown smoothness index @xmath146 , it is sufficient to select it judiciously among @xmath297 or @xmath140 well chosen nonadaptive minimax - rate predictors .    as a consequence of theorem  [ thm : oracle - bounds ] and
lemma  [ lemma : minimax_experts ] , we obtain an adaptive predictor by aggregating them ( instead of selecting one of them ) , as stated in the following result .    [
theorem : upper_bound_aggregated_forecaster_adaptative ] let @xmath238 be a distribution on @xmath90 .
let @xmath286 $ ] and @xmath298 , @xmath299 be a locally bounded set of @xmath238-minimax - rate and @xmath82-lipschitz predictors with @xmath82 satisfying  .
define @xmath300 as the predictor aggregated from @xmath301 with @xmath58 defined by @xmath302 @xmath303 defined by  ( [ eq : betai - def ] ) , and with weights defined according to one of the following setting depending on the assumption on @xmath238 and @xmath278 :    if @xmath238 satisfies   with @xmath304 and @xmath305 , use the weights  ( [ definition : alphas_gradient_quadratic_loss ] ) with @xmath306 .
if @xmath238 satisfies   with @xmath98 and @xmath307 , use the weights  ( [ definition : alphas_quadratic_loss ] ) with @xmath308 .
if @xmath238 satisfies  , use the weights  ( [ definition : alphas_quadratic_loss ] ) with @xmath309 .
then we have , for any @xmath310 , @xmath204 , @xmath159 , @xmath272 $ ] and @xmath206 , @xmath311    the proof of this theorem is postponed to giraud , roueff and sanchez - perez [ ( @xcite ) , section  a.9 ] .
[ remark : theorem_upper_bound_aggregated_forecaster_adaptative ] the limitation to @xmath305 in  ( i ) under assumption  ( i-1 ) for @xmath238 follows from the factor @xmath118 obtained in the oracle inequality  ( [ eq : strat1-final ] ) of theorem  [ thm : oracle - bounds ] after optimizing in @xmath79 [ see  ( [ eq : strat1-final - eta - opt ] ) ] .
if @xmath116 this restriction is weakened to @xmath307 in  ( ii ) taking into account the factor @xmath117 obtained in the oracle inequality  ( [ eq : strat2_p - final ] ) of theorem  [ thm : oracle - bounds ] after optimizing in @xmath79 [ see  ( [ eq : strat2_p - final - eta - opt ] ) ] . in the last case ,
the limitation of @xmath312 drops when applying the oracle inequality  ( [ eq : strat2_exp - final ] ) of the same theorem .
however , a stronger condition on @xmath238 is then required .
[ rem : beta0-eta - minim - adapt - forec ] it may happen that the locally bounded set of @xmath238-minimax - rate predictors is limited to some @xmath313 [ see the example of the nlms predictors in @xcite , section  b.2 ] . in this case
, the result roughly needs @xmath297 predictors and the computation of the aggregated one requires less operations than if @xmath312 were infinite . for these reasons , we do not consider in general that @xmath314 . on the one hand ,
a finite @xmath278 yields a restriction on the set of ( unknown ) smoothness indices @xmath146 for which the aggregated predictors are minimax rate adaptive . on the other hand , if @xmath292 , theorem  [ theorem : upper_bound_aggregated_forecaster_adaptative ] then requires the stronger assumption  ( i-2 ) on the process .
[ rem : sigma - eta - minim - adapt - forec ] the constant @xmath315 present in the definitions of @xmath79 in the three cases  ( i ) , ( ii ) and ( iii ) corresponds to the homogenization of the remaining terms appearing in theorem  [ thm : oracle - bounds ] [ the second lines of  ( [ eq : strat1-final ] ) , ( [ eq : strat2_p - final ] ) and ( [ eq : strat2_exp - final ] ) ] . indeed with the proposed choices and in the three cases , the constant @xmath316 factors out in front of the remaining terms [ see the last three displayed equations in @xcite , section  a.9 ] .
however , the @xmath317 in the definitions of @xmath79 does not impact the convergence rate in the sense that theorem  [ theorem : upper_bound_aggregated_forecaster_adaptative ] is still valid using any other constant ( @xmath77 , e.g. ) in these definitions .
we start with a lemma which gathers useful adaptations of well - known inequalities applying to the aggregation of deterministic predicting sequences .
[ lemma : deterministic_upper_bound_aggregated_forecaster ] let @xmath318 be a real valued sequence and @xmath319 be a collection of predicting sequences . define @xmath320 as the sequence of aggregated predictors obtained from this collection with the weights ( [ definition : alphas_gradient_quadratic_loss ] ) . then , for any @xmath74 , we have @xmath321}-x_{t } \bigr)^2 + \frac{\log n}{t\eta } + \frac{2\eta}{t}\sum _ { t=1}^{t } y_{t}^{4 } , \end{aligned}\ ] ] where @xmath322 .
define now @xmath320 as the sequence of aggregated predictors obtained with the weights  ( [ definition : alphas_quadratic_loss ] ) .
then , for any @xmath323 , we have @xmath324\\[-8pt]\nonumber & & \qquad \leq\min _ { i=1,\dots , n}\frac{1}{t}\sum_{t=1}^t \bigl(\widehat{x}_{t}^{(i)}-x_{t }
\bigr)^2 + \frac{\log n}{t\eta } + \frac{1}{t}\sum _ { t=1}^{t } \biggl(y_{t}^{2}- \frac{1}{2\eta } \biggr)_{+ } , \end{aligned}\ ] ] where @xmath322 .    with the weights defined by  ( [ definition : alphas_gradient_quadratic_loss ] ) , by slightly adapting [ @xcite ( @xcite ) , theorem 1.7 ]
, we have that @xmath325}-x_{t } \bigr)^2 & \leq & \frac{\log n}{t\eta } + \frac{\eta}{8 t }   s_{t}^ { * } , \end{aligned}\ ] ] where @xmath326 and @xmath327 .
the bound ( [ eq : strat1-deterministic ] ) follows by using that @xmath328 is in the simplex @xmath329 defined in  ( [ eq : def - simplexn ] ) .
we now prove  ( [ eq : strat2-deterministic - allcases ] ) .
we adapt the proof of @xcite [ ( @xcite ) , proposition 2.2.1 . ] to unbounded sequences by replacing the convexity argument by the following lemma .
[ lemma : bound_integral_concave_exponential ] let @xmath330 and @xmath331 a probability distribution supported on @xmath332 $ ]
. then we have @xmath333    the proof of lemma  [ lemma : bound_integral_concave_exponential ] is postponed to section  [ sec : proof - lemma - technical - exp - concave ] in .
now , let @xmath74 and @xmath334 .
using lemma  [ lemma : bound_integral_concave_exponential ] with the probability distribution @xmath331 defined by @xmath335 and @xmath336 , we get that @xmath337 taking the log , multiplying by @xmath338 and re - ordering the terms , we obtain that @xmath339 taking the average over @xmath69 and developing the expression of @xmath67 , we obtain @xmath340\\[-8pt]\nonumber & & { } + \frac{1}{t}\sum_{t=1}^{t } \biggl(y_{t}^{2}-\frac{1}{2\eta } \biggr)_{+}.\end{aligned}\ ] ] using that @xmath341 , we get the bound  ( [ eq : strat2-deterministic - allcases ] ) .      we prove the cases  ( i ) , ( ii ) and  ( iii ) successively .
we denote @xmath342 .    applying  ( [ eq : strat1-deterministic ] ) in lemma  [ lemma : deterministic_upper_bound_aggregated_forecaster ] with @xmath343\leq\inf{\mathbb{e}}[\cdots]$ ] , we obtain @xmath344 & \leq&\inf_{\bolds{\nu}\in\mathcal{s}_n}\frac{1}{t}\sum _
{ t=1}^t{\mathbb{e}}\bigl [ \bigl(\widehat{x}_{t}^{[\bolds\nu]}-x_{t } \bigr)^2 \bigr ] \nonumber\\[-8pt]\\[-8pt]\nonumber & & { } + \frac{\log n}{t\eta } + \frac{2\eta}{t}\sum _ { t=1}^{t } { \mathbb{e}}\bigl[y_{t}^{4 } \bigr ]   .\end{aligned}\ ] ] using that the predictors are @xmath82-lipschitz and the process @xmath1 satisfies  ( m-1 ) , we have , for all @xmath345 , @xmath346 where @xmath347 applying the minkowski inequality together with  ( [ inequality : x - z ] ) , ( [ eq : unif - coeff - bound ] ) and  ( [ eq : hyp : lip ] ) , we obtain , for all @xmath163 , @xmath348 \leq{\mathbb{e}}\biggl [ \biggl(\sum _ { j\in{\mathbb{z } } } b_t(j ) z_{t - j } \biggr)^{4 } \biggr ] \leq a_{*}^4(1+l_*)^{4 }   \sup_{t\in { \mathbb{z}}}{\mathbb{e}}\bigl[z_t^4 \bigr ] .\end{aligned}\ ] ] since the process @xmath94 fulfills   with @xmath95 , plugging this bound in  ( [ inequality : expectation_aggregated_forecaster_moment ] ) we obtain  ( [ eq : strat1-final ] ) .
we use  ( [ eq : strat2-deterministic - allcases ] ) in lemma  [ lemma : deterministic_upper_bound_aggregated_forecaster ] and the inequality @xmath349 which holds for @xmath350 and @xmath351 .
we get , taking the expectation , @xmath352 & \leq&\min_{i=1,\dots , n}\frac{1}{t}\sum _ { t=1}^{t}{\mathbb{e}}\bigl [ \bigl(\widehat{x}_{t , t}^{(i)}-x_{t , t } \bigr)^2 \bigr ] + \frac{\log n}{t\eta } \nonumber\\[-8pt]\\[-8pt]\nonumber & & { } + ( 2\eta)^{p/2 - 1}\max_{t=1,\ldots , t}{\mathbb{e}}\bigl[y_{t}^p \bigr ]   .\end{aligned}\ ] ] applying the minkowski inequality , ( [ inequality : x - z ] ) and assumption  , @xmath353 \leq\biggl(\sum _ { j\in z}b_{t}(j ) \bigl({\mathbb{e}}\bigl[z_{t - j}^{p } \bigr ] \bigr)^{1/p } \biggr)^{p } \leq a_{*}^{p}(1+l_*)^{p }   \sup_{t\in z}{\mathbb{e}}\bigl[z_t^{p } \bigr ] . \ ] ] using this bound which is independent of @xmath16 , with   and  ( [ inequality : expectation_aggregated_forecaster_moment_1 ] ) , the inequality  ( [ eq : strat2_p - final ] ) follows .    to obtain  ( [ eq : strat2_exp - final ] )
, we again use  ( [ eq : strat2-deterministic - allcases ] ) in lemma  [ lemma : deterministic_upper_bound_aggregated_forecaster ] but now with an exponential bound for @xmath354 .
we note that , or all @xmath355 , @xmath356 it follows that , for all @xmath357 and @xmath355 , @xmath358 applying this bound with @xmath359 and @xmath360 we get @xmath361 plugging this into  ( [ eq : strat2-deterministic - allcases ] ) and taking the expectation , we obtain that @xmath362\nonumber \\ & & \qquad \leq\min_{i=1,\dots , n}\frac{1}{t}\sum _ { t=1}^{t}{\mathbb{e}}\bigl [ \bigl(\widehat{x}_{t , t}^{(i)}-x_{t , t } \bigr)^2 \bigr ] + \frac{\log n}{t\eta } \\ & & \quad\qquad { } + 2\lambda^{-2 } \bigl(2+\lambda(2\eta)^{-1/2 } \bigr ) { \mathrm{e}}^{-1- \lambda(2\eta)^{-1/2 } }   \max_{t=1,\ldots , t}{\mathbb{e}}\bigl[{\mathrm{e}}^{\lambda y_t } \bigr ]   .\nonumber\end{aligned}\ ] ] we now use assumption  . since @xmath363 for all @xmath364 and @xmath365 jensen s inequality and ( [ inequality : x - z ] ) gives that , for any @xmath366 , @xmath367&\leq&{\mathbb{e}}\bigl[{\mathrm{e}}^{\lambda ( { \vert}{x}_{t}{\vert}+\max_{1\leq i\leq n}{\vert}\widehat { x}_{t}^{(i)}{\vert } ) } \bigr ] \\ & \leq&\prod_{j\in{\mathbb{z}}}{\mathbb{e}}\bigl[{\mathrm{e}}^{\lambda b_t(j ) z_{t - j } } \bigr ] \\ & \leq&\prod_{j\in{\mathbb{z } } } \bigl(\phi(\zeta ) \bigr)^{\lambda b_t(j)/\zeta } \leq\bigl(\phi(\zeta ) \bigr)^{\lambda a_{*}(1+l_{*})/\zeta}.\end{aligned}\ ] ]    the combination of this bound with  ( [ inequality : expectation_aggregated_forecaster_exponential_moment ] ) gives  ( [ eq : strat2_exp - final ] ) .
the proof of theorem  [ thm : oracle - bounds ] is complete .      minimizing the sum of the two terms appearing in the second line of  ( [ eq : strat2_exp - final ] )
is a bit more involved , since it depends both on @xmath79 and @xmath103 . under condition  ( [ eq :
def - sup - unif - coeff - bound ] ) , the quantity @xmath368 remains between two positive constants while , for any @xmath74 , @xmath369 is decreasing as @xmath103 increases . to simplify @xmath370 into @xmath371
, we simply take @xmath372 which satisfies  ( [ eq : def - sup - unif - coeff - bound ] ) . now that @xmath103 is set
, it remains to choose a value of @xmath79 which ( almost ) minimizes @xmath373 the @xmath79 defined as in  ( [ eq : strat2_exp - choice - eta ] ) is chosen so that @xmath374 , and we get  ( [ eq : strat2_exp - final - opt - eta ] ) .
we now provide a proof of theorem  [ theorem : lower_bound_quadratic_risk ] .
we consider an autoregressive equation of order one @xmath375 where @xmath11 is i.i.d . with density @xmath244 as in  ( i-3 ) . in this case , if @xmath376 , the representation  ( [ eq : inf - series - representation ] ) of the stationary solution reads , for all @xmath377 as @xmath378 with the convention @xmath379 .
the class of models so defined with @xmath380 corresponds to assumption  ( m-2 ) with @xmath248 in @xmath260 such that only the first component of @xmath227 is nonzero and @xmath169 is constant and equal to one .
we write henceforth in this proof @xmath381 for the law of the process @xmath382 and @xmath383 for the corresponding expectation .
let @xmath384 be any predictor of @xmath385 in the sense of definition  [ def : predictor ] .
define @xmath386 by @xmath387 for any vectors @xmath388 , we define @xmath389 by  ( [ eq : lb - tvar - eq ] ) , since @xmath162 and @xmath390 are @xmath391-measurable , they are independent of @xmath392 and we have @xmath393 - 1={\mathbb{e}}_{\theta } \bigl[d_{x}^2 \bigl(\widehat\theta,\mathrm{v}_{t}\{\theta\}\bigr ) \bigr ] , \ ] ] where , for any @xmath394\to{\mathbb{r}}$ ] , @xmath395 denotes the @xmath26-sample of @xmath396 on the regular grid @xmath397 , @xmath398 hence , to prove the lower bound of theorem  [ theorem : lower_bound_quadratic_risk ] , it is sufficient to show that there exist @xmath399 , @xmath400 and @xmath401 both depending only on @xmath402 , @xmath146 , @xmath219 and the density @xmath244 , such that for any @xmath403 adapted to @xmath404 and @xmath405 , we have @xmath406\geq c   t^{-2\beta/ ( 2\beta+1 ) } .\ ] ] we now face the more standard problem of providing a lower bound for the minimax rate of an estimation error , since @xmath407 is an estimator of @xmath408 .
the path for deriving such a lower bound is explained in [ @xcite ( @xcite ) , chapter  2 ] .
however , we have to deal with a loss function @xmath409 which depends on the observed process  @xmath91 .
not only the loss function is random , but it is also not independent of the estimator  @xmath410 .
the proof of the lower bound  ( [ eq : objectif : minimax ] ) thus requires nontrivial adaptations .
it relies on some intermediate lemmas .
[ lemma : birge ] we write @xmath411 for the kullback
leibler divergence between @xmath331 and @xmath412 . for any functions @xmath413 from @xmath28
$ ] to @xmath90 such that @xmath414 and any @xmath415 we have @xmath416 \\ & & \qquad \geq{r^2\over4 } \biggl({1\over2{\mathrm{e}}+1}-\max _ { j=0,\ldots , m}{\mathbb{p}}_{\theta _ { j } } \bigl(\min_{i : i\neq j}d_{x , t } ( \theta_{i},\theta_{j})\leq r \bigr ) \biggr ) , \ ] ] where we denote , for any two functions @xmath417 from @xmath166 $ ] to @xmath90 , @xmath418    the proof is postponed to section  [ section : proof_lemma_birge ] in .
we next construct certain functions @xmath419 fulfilling  ( [ eq : kullback - condition ] ) and well spread in terms of the pseudo - distance @xmath420 .
consider the infinitely differentiable kernel @xmath421 defined by @xmath422 given any @xmath423 , vershamov
gilbert s lemma [ @xcite ( @xcite ) , lemma 2.9 ] ensures the existence of @xmath424 points @xmath425 in the hypercube @xmath426 such that @xmath427\\[-8pt ] \eqntext{\mbox{for all } j\neq i.}\end{aligned}\ ] ] we then define @xmath413 by setting , for all @xmath428 , @xmath429 where @xmath430 since @xmath431 out of @xmath432 , we observe that @xmath433 and @xmath434,\ ] ] where @xmath435 denotes the fractional part of @xmath436 .
thus , we have @xmath437 } \bigl{\vert}\theta_{j}(x)\bigr{\vert}\leq{r_0{\mathrm{e}}^{-1}\over m^{\beta}}\leq \delta<1   .\ ] ] we first check that the definition of @xmath438 ensures that the @xmath439 s are in the expected set of parameters .
[ lemma : admissible - part1 ] for all @xmath440 , we have @xmath441 .
the proof can be found in section  [ section : proof_lemma_admissible - part1 ] of .
next , we provide a bound to check the required condition  ( [ eq : kullback - condition ] ) on the chosen  @xmath442 s .
[ lemma : admissible - part2 ] for all @xmath443 , we have @xmath444 where @xmath445 is the constant appearing in
.    we prove it in section  [ section : proof_lemma_admissible - part2 ] of .    finally , we need a control on the distances @xmath446 .
[ lemma : min_distance ] for any @xmath447 , there exists a constant @xmath448 depending only on @xmath449 and the density @xmath244 of @xmath234 such that for all @xmath450 , @xmath451 and @xmath440 , @xmath452    the proof is postponed to section  [ section : proof_lemma_min_distance ] of .
we can now complete the proof of theorem  [ theorem : lower_bound_quadratic_risk ] .
proof of theorem  [ theorem : lower_bound_quadratic_risk ] recall that @xmath453 in  ( [ eq : def - thetaj ] ) are some parameters only depending on @xmath146 and @xmath224 and a certain integer @xmath423 and that , whatever the value of @xmath454 , lemma  [ lemma : admissible - part1 ] insures that @xmath453 belongs to @xmath455 .
hence , it is now sufficient to show that  ( [ eq : objectif : minimax ] ) holds for a correct choice of @xmath454 , relying on lemmas  [ lemma : birge ] ,  [ lemma : admissible - part2 ] and  [ lemma : min_distance ] .
let us set @xmath456 where @xmath457 is a constant to be chosen .
then @xmath458 and , by lemma  [ lemma : admissible - part2 ] , we can choose @xmath457 only depending on @xmath146 , @xmath219 , @xmath445 and @xmath224 so that condition  ( [ eq : kullback - condition ] ) of lemma  [ lemma : birge ] is met .
we thus get that , for any @xmath415 , @xmath459 \\ & & \qquad { } \geq{r^2\over4 } \biggl({1\over2{\mathrm{e}}+1}-\max _ { j=0,\ldots , m}{\mathbb{p}}_{\theta _ { j } } \bigl(\min_{i : i\neq j}d_{x , t } ( \theta_{i},\theta_{j})\leq r \bigr ) \biggr ) . \ ] ] applying lemma  [ lemma : min_distance ] with @xmath460 and the previous bound with @xmath461 , we get , as soon as @xmath451 , @xmath462\geq{r^2\over4 } \biggl({1\over4{\mathrm{e}}+2}- \frac{2 r_0{\mathrm{e}}^{-1}}{a ( 1-\delta)m^\beta } \biggr )   . \ ] ] the proof is concluded by observing that , as a consequence of  ( [ eq : m ] ) , we can choose a constant @xmath221 only depending on @xmath146 , @xmath219 , @xmath445 and @xmath224 such that @xmath213 implies that @xmath451 and that the term between parentheses is bounded by @xmath463 from below .
in this section , we test the proposed aggregation methods on data simulated according to a tvar process with @xmath464 .
the choice of a smooth parameter function @xmath465 within @xmath466 for some @xmath467 is done by first picking randomly some smoothly time varying partial autocorrelation functions up to the order @xmath275 that are bounded between @xmath468 and @xmath77 and then by relying on the levinson
durbin algorithm .
we show the three components of the obtained @xmath469 on @xmath470 $ ] in the top parts of figure  [ figure : thetas_and_tvar ] .
realizations of the tvar process are then obtained from an innovation sequence @xmath37 of i.i.d
. centered gaussian process with unit variance as in definition  [ definition : tvar ] by sampling @xmath227 at a given rate @xmath54 .
figure  [ figure : thetas_and_tvar ] displays one realization of such a tvar process for @xmath471 .
, @xmath472 and @xmath473 on the interval @xmath28 $ ] .
the last plot displays @xmath471 samples of the corresponding tvar process with gaussian innovations . ]    the nlms algorithm [ see @xcite , algorithm  1 ] studied in @xcite provides an online estimator of @xmath227 depending on a gradient step size @xmath474 . for any @xmath144
$ ] , choosing @xmath475 yields a @xmath260-minimax - rate online @xmath82-lipschitz predictor as explained in @xcite , section  b.1 . hence , proceeding as in lemma  [ lemma : minimax_experts ] to define @xmath58 and @xmath303 , @xmath68 , with @xmath476 , we obtain a finite set of nlms predictors corresponding to gradient step sizes @xmath477 .
this set of predictors is aggregated in two possible ways according to the online algorithm  [ algo : strat12 ] with the specifications on @xmath79 and @xmath58 given in theorem  [ theorem : upper_bound_aggregated_forecaster_adaptative ] .
the overall running time of @xmath26 iterates of the algorithm leading to the aggregated predictors from the data @xmath33 is then @xmath478 .
since the algorithm is recursive , the corresponding required storage capacity is @xmath479 .
we evaluate the obtained nlms predictors and their aggregated predictors by running @xmath480 simulations based on equally distributed realizations of the above gaussian tvar process in the case @xmath471 which yields @xmath481 . in figure
[ figure : beta_0=5_t=1024 ] , we compare the averaged downward shifted empirical losses defined for any predictor @xmath266 by @xmath482 this empirical averaged loss mimics the risk considered in  ( [ eq : max - risk ] ) .     of the nlms predictors @xmath483 .
the ones on the right of the same line are those associated to the aggregated predictors using the weights ( [ definition : alphas_gradient_quadratic_loss ] ) and ( [ definition : alphas_quadratic_loss ] ) . ]
we observe that the best nlms predictor is the third one while the aggregated predictor of strategy  1 enjoys a smaller loss and that of strategy  2 a slightly larger one .
this is in accordance with theorem [ thm : oracle - bounds](i ) and  ( iii ) where it is shown that the aggregated predictor of the first strategy may outperform the best predictor as it nearly achieves the loss of the best possible convex combination of the original predictors while the aggregated predictor of the second strategy nearly achieves the loss of the best original predictor .
[ section : useful_bounds_tvar_postponed_proofs ]
the following lemma provides a uniform bound on the norm of a product of matrices sampled from a continuous function defined on an interval @xmath155 and valued in a set of @xmath484 matrices with bounded spectral radius and norm .
[ lem : product ] let @xmath167 and @xmath155 an interval of @xmath90 .
let @xmath448 be a function defined on @xmath155 taking values in the set of @xmath485 matrices with eigenvalues moduli at most equal to @xmath224 .
let @xmath486 be any matrix norm .
denote by @xmath487 the corresponding uniform norm of @xmath448 , @xmath488 and , for any @xmath489 , @xmath490 the modulus of continuity of @xmath448 over @xmath155 , @xmath491 let @xmath492 and assume that @xmath493 . then there exist some positive constants @xmath449 , @xmath494 and @xmath421 only depending on @xmath487 , @xmath224 and @xmath495 such that , for any @xmath496 fulfilling @xmath497 , we have , for all @xmath498 in @xmath155 and all integer @xmath499 , @xmath500    denote by @xmath501 the product of matrices appearing in the left - hand side of  ( [ eq : product - univ - bound ] ) .
the proof goes along the same lines as [ moulines , priouret and roueff ( @xcite ) , proposition  13 ] but we use the modulus of continuity instead of the @xmath146-lipschitz norm to control the local oscillation of matrices .    for @xmath502 and any square matrices @xmath503 , adopting the convention @xmath504 if @xmath505 and @xmath506 is the identity matrix if @xmath507 , we have @xmath508\\[-8pt]\nonumber & = & a_{1}^{\ell_{1}}+\sum_{k=1}^{\ell_{1}-1}a_{1}^{\ell_{1}-k } ( a_{\ell_{1}-k+1}-a_{1 } ) \prod_{i=\ell_{1}-k+2}^{\ell _ { 1}}a_{i } . \nonumber\end{aligned}\ ] ]    given a positive integer @xmath494 , using the euclidean division of @xmath509 by @xmath494 , @xmath510 , we decompose the product @xmath501 as @xmath511\\[-8pt]\nonumber & & { } \times\prod_{k=1}^{r}a \biggl(t- \frac{(q\ell+k-1)(t - s)}{p } \biggr ) .\end{aligned}\ ] ]    using ( [ equation : decomposition_product ] ) , we have for any @xmath512 , @xmath513 and @xmath514 , @xmath515\\[-8pt]\nonumber & & \qquad \leq\biggl{\vert}\biggl(a \biggl(t-\frac{j\ell(t - s)}{p } \biggr ) \biggr)^{\ell _ { 1}}\biggr{\vert}+ ( \ell_{1}-1 ) \bigl(a^ { * } \bigr)^{\ell _ { 1}-1}\omega_{h } ( a;i ) .\end{aligned}\ ] ]    take an arbitrary @xmath516 ( say the middle point ) .
the eigenvalues of @xmath448 are at most @xmath224 on @xmath155 and @xmath493 . applying [ @xcite ( @xcite ) , lemma  12 ]
we obtain that there is a constant @xmath517 only depending on @xmath224 , @xmath518 and @xmath519 such that @xmath520 .    from ( [ equation : decomposition_pi_t_s ] ) and
( [ equation : bound_block_size_l_product ] ) , we derive the following inequality : @xmath521 where @xmath522 .    we can choose a positive integer @xmath494 and a positive number @xmath523 only depending on @xmath524 , @xmath495 and @xmath525 such that @xmath526 in the following , we set @xmath527 .
the previous bound gives that for any @xmath496 such that @xmath497 and @xmath528 , @xmath529 hence , we have the result .
we can now provide a proof of proposition  [ proposition : stationary_tvar ] .
equation  ( [ equation : definition_tvar ] ) can be more compactly written as @xmath530 for all @xmath531 , iterating this recursive equation @xmath154 times , we have @xmath532\mathbf x_{t - k-1,t } \nonumber\\[-8pt]\\[-8pt]\nonumber & & { } + \sum_{j=0}^{k } \sigma \biggl ( { \frac{t - j}{t } } \biggr)\mathbf e'_{1 } \biggl[\prod_{i=1}^{j}a \biggl ( { \frac { t - i}{t } } \biggr ) \biggr]\mathbf e_{1}\xi_{t - j } , \end{aligned}\ ] ] where @xmath533'$ ] and @xmath534 . \ ] ]    note that the eigenvalues of @xmath535 are the reciprocals of the roots of the local time varying autoregressive polynomial @xmath536 , and thus are at most @xmath537 . moreover , since @xmath227 is bounded by a constant only depending on @xmath275 and is uniformly continuous on @xmath158 $ ] , so is @xmath448 as a function defined on @xmath155 and we can find @xmath496 such that @xmath538 for any positive @xmath449 . if @xmath539 , this @xmath540 can be chosen depending only on @xmath541 and @xmath219 ( and also on the matrix norm @xmath486 )
.    consider @xmath216 .
lemma  [ lem : product ] gives that there exist some positive constant @xmath449 , @xmath494 and @xmath421 only depending on @xmath487 , @xmath224 and @xmath495 such that , for any @xmath496 fulfilling @xmath497 , we have , for all @xmath54 , @xmath200 and @xmath542 so that @xmath543 , @xmath544 we here consider the @xmath545 operator norm which is the maximum absolute row sum of the matrix , in which case @xmath546 .
hence , by ( [ equation : tvar_initial_condition_noise ] ) we obtain that @xmath547\\[-8pt ] \eqntext{1\leq t\leq t,}\end{aligned}\ ] ] with , provided that @xmath548 , for all @xmath200 , @xmath549 and @xmath550 , @xmath551    the result follows .
denote @xmath552 , so that @xmath553 .
the function @xmath554 is concave on @xmath555 $ ] , so introducing @xmath556 and then using jensen s inequality , we get @xmath557 it only remains to show that @xmath558 , with the assumption that @xmath331 has support on @xmath332 $ ] .
this is verified if @xmath559 , so we now assume @xmath560 .
we write @xmath561 we note that @xmath562 and @xmath563 .
we deduce that the product @xmath564 either take nonpositive values or positive values of the form @xmath565 now , for @xmath566 $ ] with @xmath560 , in the first case , we have @xmath567 since @xmath568 , and , in the second case , @xmath569 .
the lemma follows .
we define @xmath570 as the ( random ) smallest index which minimizes @xmath571 over @xmath572 so that @xmath573 .
note that @xmath574 .
hence , @xmath575 \\ & & \qquad \geq { 1\over4 } \max_{j=0,\ldots , m}{\mathbb{e}}_{\theta_{j } } \bigl[d_{x , t}^2(\theta_{\hat{{\mathrm{j } } } } , \theta_{j } ) \bigr ] \\ & & \qquad \geq{r^2\over4}\max_{j=0,\ldots , m}{\mathbb{p}}_{\theta_{j } } \bigl ( \ { \hat{{\mathrm{j}}}\neq j \ } \cap \bigl\{\min_{i : i\neq j}d_{x , t } ( \theta _ { i},\theta_{j } ) > r \bigr\ } \bigr ) \\ & & \qquad \geq{r^2\over4 } \bigl(1-\min_{j=0,\ldots , m } { \mathbb{p}}_{\theta_{j } } ( \hat{{\mathrm{j}}}= j ) -\max_{j=0,\ldots , m}{\mathbb{p}}_{\theta_{j } } \bigl(\min_{i : i\neq j}d_{x , t}(\theta_{i } , \theta_{j})\leq r \bigr ) \bigr).\end{aligned}\ ] ] birg s lemma [ @xcite ( @xcite ) , corollary  2.18 ] implies that @xmath576 so the lemma follows from condition  ( [ eq : kullback - condition ] ) .      by  ( [ eq : thetastar ] ) , we have @xmath577 for all @xmath578
. decompose the hlder - exponent @xmath152 where @xmath154 is an integer and @xmath151 $ ] . differentiating  ( [ eq : def - thetaj ] ) @xmath154 times
, we have , as in  ( [ eq : thetaj ] ) , @xmath579 . \ ] ] thus , for @xmath580 in the same interval @xmath581 $ ] with @xmath582 , we get @xmath583 the same inequality then follows with @xmath584 replaced by @xmath585 for @xmath580 in two such consecutive intervals . now , if @xmath580 are separated by at least one such interval , we have @xmath586 and , using that @xmath421 has support in @xmath432 , we have that @xmath587 is bounded by @xmath588 .
we thus get in this case that @xmath589 the last two displays and  ( [ eq : l0-def ] ) then yields @xmath590 .
let @xmath591 . recall that @xmath592 by  ( [ eq : vershamov ] ) and  ( [ eq : def - thetaj ] ) . by  ( [ eq : thetaj - zero ] ) and  ( [ eq : lb - tvar - eq ] ) , we have that @xmath197 has the same distribution under @xmath593 and @xmath594 [ which is the distribution of @xmath595 . hence , the likelihood ratio @xmath596 of @xmath597 is given by the corresponding conditional likelihood ratio of @xmath598 given @xmath197 .
hence , under  ( i-3 ) , we obtain that @xmath599 where , in the second equality , we used again that @xmath592 . now , under @xmath600 , we have @xmath601 .
thus , we get @xmath602 \\ & = & \sum_{t=1}^t { \mathbb{e}}_{\theta_{j } } \biggl[\log\frac{f(\xi _ { t})}{f(\theta_{j}((t-1)/t)x_{t-1,t}+\xi_{t } ) } \biggr ] \\ & = & \sum_{t=1}^t { \mathbb{e}}_{\theta_{j}}\int \log \biggl(\frac{f(u)}{f(\theta_{j}((t-1)/t)x_{t-1,t}+u ) } \biggr ) f(u ) \,{\mathrm{d}}u.\end{aligned}\ ] ] using assumption  ( i-3 ) yields @xmath603 \leq\kappa\theta^{*2}\sum _ { t=1}^{t}{\mathbb{e}}_{\theta_{j } } \bigl[x_{t-1,t}^{2 } \bigr].\end{aligned}\ ] ] the series representation  ( [ eq : inf - series - representation - d1 ] ) , the fact that @xmath234 is centered with unit variance and  ( [ eq : thetastar ] ) imply that for all @xmath604 @xmath605 \leq \bigl(1- \theta ^{*2 } \bigr)^{-1 } . \ ] ] using this bound and  ( [ eq : thetastar ] ) in  ( [ eq : etap1:admissible ] ) , we obtain @xmath606 the proof of lemma  [ lemma : admissible - part2 ] now follows by applying the first bound in  ( [ eq : vershamov ] ) .
the proof relies on an upper bound of @xmath607 involving the noise @xmath608 . by the expression of @xmath442 in  ( [ eq : thetaj ] ) , we have @xmath609 where we denoted @xmath610 and @xmath611 . using  ( [ eq : inf - series - representation - d1 ] ) and  ( [ eq : thetastar ] ) , we have , for all
@xmath612 , @xmath613 which implies @xmath614 inserting this bound in  ( [ eq : d2xtdef - thetaj ] ) , we get @xmath615 where @xmath616 thus , with  ( [ eq : bound - d2-remainder ] ) , the left - hand side of inequality  ( [ eq : min - distance ] ) is upper bounded by @xmath617 using that @xmath234 is centered with unit variance and then  ( [ eq : thetastar ] ) , we easily get that @xmath618\leq\frac{2{\mathrm{e}}^{-2}}t \sum_{t=0}^{t-1}\sum _ { j=1}^{\infty}\theta^{*j } \leq \frac{2{\mathrm{e}}^{-2}\theta^*}{1-\theta^ * } \leq\frac{2 r_0{\mathrm{e}}^{-3}}{(1-\delta)m^\beta } .\ ] ] hence , by markov s inequality , to conclude the proof , it now suffices to show that , for @xmath448 well chosen , @xmath619 for @xmath620 we define @xmath621 .
we observe that the cardinality of @xmath622 is @xmath623 where the lower bound is a consequence of the assumption @xmath624 in the lemma .
moreover , it is easy to check that we have @xmath625 for all index @xmath626 and that , for each @xmath627 , the set @xmath628 is included in the set
@xmath629 ( so that , in particular , @xmath630 for @xmath631 ) .
it follows that random variables @xmath632 are i.i.d . by the monotonicity of @xmath421 in @xmath633 and its symmetry
, we have @xmath634 from  ( [ eq : vershamov ] ) , for any @xmath635 there exist at least @xmath636 values of @xmath154 for which @xmath637 equals one in the above sum .
hence , using the order statistics @xmath638 , we thus obtain that @xmath639 where we used @xmath640 for @xmath641 in the last inequality .
let us denote by @xmath642 the cumulative distribution function of @xmath643 , which only depends on @xmath644 and on the distribution of @xmath645 . for @xmath646
, we have @xmath647 gathering the last two bounds , we get that @xmath648 recall that @xmath649 and note that @xmath650 admits a density , since @xmath234 does . by the strong law of large numbers , we further have that the random variable @xmath643 converges to @xmath77 almost surely when @xmath644 goes to infinity , so there exists @xmath651 depending only on the density of @xmath234 such that @xmath652 whatever the value of .
therefore , there exists some @xmath653 , depending only on the distribution of @xmath234 , such that  ( [ eq : bound - d2-remainderrt ] ) holds , which achieves the proof .
we gratefully acknowledge the fruitful comments of the referees .