in the context of a graphical model , the concept of local constancy has been coined recently in honorio et al . , 2009 .
the goal of the present paper is to penetrate this idea by providing a thorough analysis and an extension of this concept .
our approach is using the idea propounded by meinshausen and bhlmann ( 2006 ) for structure learning in gaussian graphical models , and it incorporates an additional fusion penalty term that aims to enforce the structural constraint of _ local constancy_. when the nodes of a graph have spatial information attached , then local constancy can be interpreted as a certain type of spatial regularity ( see below ) .
conceptually , the entire neighborhood graph given by the graphical model is split into two subgraphs : ( a ) a _ local _ graph consisting of nodes that are spatially close , and ( b ) a _ non - local _ graph , where the edges connect nodes of two different spatial clusters of nodes .
our approach then assumes that we have prior structural knowledge about the local graph that we aim to incorporate into the estimation / model selection approach .
it is well known that a @xmath1-dimensional graphical model is given by a @xmath1-dimensional gaussian distribution with non - singular covariance matrix @xmath2 of a random vextor @xmath3 .
the conditional independence of the components of @xmath3 can be represented by occurrence of zero entries of precision matrix @xmath4 .
the non - zero entries of the precision matrix define the edges of the graph corresponding to this multivariate gaussian distribution .
neighborhood selection algorithms aim to find all the neighbors of a node @xmath5 in a graphical model based on an i.i.d
meinshausen & bhlmann  @xcite showed that this problem can be interpreted as an ensemble of @xmath0-penalized regressions , each of which can be solved using the lasso algorithm of tibshirani ( 1997 ) .
our proposal is an extension of this approach . in order to incorporate this additional structural constraint
we assume that the prior structural knowledge can be translated into a _ local _ neighborhood graph , which we think of being comprised of regular graph objects such as chains , cycles , lattices or cliques .
the availability of additional locality information is critical when data is observed in a certain manifold with spatial geometry .
the assumption of local constancy , in some sense , enforces spatial regularization on structure learning and thereby stimulates the search for probabilistic dependencies between local clusters of nodes .
we would like to emphasize that the knowledge of this prior structural information is based on domain knowledge and hence known beforehand .
we propose to use the meinshausen & bhlmann approach and to add a new penalty term which , in essence , generalizes the fused lasso penalty ( see tibshirani , 2005 ) by extending the prenalty term over differences of nodes in the local neighborhood graph , which in turn is given by the prior structural information .
this leads to the _ neighborhood - fused lasso _ procedure , a model selection method for locally constant gaussian graphical models .
we provide theoretical and numerical evidence that our approach outperforms competing model selection algorithms where locality information is ignored .
the paper is organized as follows : in section  [ review ] we first provide an overview of related work .
sections  [ nfl ] and  [ locconstdefn ] then provide a precise definition of our approach descibed above , and a discussion of local constancy , respectively . in section [ optimization ]
we propose our optimization algorithm , before we present both both finite sample and large sample properties of our appraoch in section  [ theory ] .
we prove theoretically that introducing a local penalty term reduces the finite sample type - i error probability in model selection and leads to equivalent accuracy with smaller sample size than competitors .
we also discuss data dependent choice of the penalty parameters , which avoids the use of cross - validation based methods .
we study the asymptotic @xmath0 properties of our estimator to find sufficient conditions on design matrix and regularization parameters to find nice asymptotic bounds in parameter estimation and prediction in terms of @xmath0 and @xmath6 metric , respectively .
in particular , our theoretical analysis reveals that with our assumptions our proposed method displays all the desired properties like sign - consistency and model selection consistency .
some numerical results can be found in section  [ simulations ] .
the proofs are delegated to the appendix .
going back to dempster  @xcite who introduced _ covariance selection _ to discover the conditional independence restrictions ( the graph ) from a set of i.i.d .
observations , many methods have been proposed for sparse estimation of the precision matrix in a gaussian graphical model .
the proposed procedures usually rely on optimization of an objective function  @xcite . while modern technological developments and high computing power enable us to deal with high dimensional models , there still are computational challenges . usually , greedy forward - selection or backward - deletion search is used . in forward ( backward ) search , one starts with the empty ( full ) set and adds ( deletes ) edges iteratively until a suitable stopping criterion is fulfilled .
the selection ( deletion ) of an edge requires an mle fit  @xcite for @xmath7 many models , making it a suboptimal choice for high - dimensional models where @xmath1 is large .
also , the mle might not exist in general for @xmath8 ( see @xcite ) .
in contrast , neighborhood selection using lasso , as proposed by meinshausen and bhlmann  @xcite , relies on optimizing a convex function applied consecutively to each node in the graph , thus fitting @xmath9 many models .
fast lasso - type algorithms and data dependent choices for regularization parameter reduce the computational cost . unlike covariance selection , this algorithm estimates the dependency graph by sequential estimation of individual neighbors and subsequent combination by taking unions or intersections .
other authors have proposed algorithms for the exact maximization of the @xmath0-penalized log - likelihood .
yuan & lin  @xcite proposed an @xmath0-penalty on the off - diagonal elements of the concentration matrix for its sparse estimation with the positive definiteness constraint .
they showed that this problem is similar to the _ maxdet _ problem ( see vandenberghe et al .
@xcite ) , and thus solvable by the interior point algorithm .
a quadratic approximation to the objective function in their proposed method leads to a solution similar to meinshausen & bhlmann .
banerjee et al .
@xcite viewed this as a penalized maximum likelihood estimation problem with the same @xmath0-penalty on the concentration matrix .
constructing the dual transforms the problem into sparse estimation of the covariance matrix instead of the concentration matrix .
they proposed _ block coordinate descent algorithm _ to solve this efficiently for large values of @xmath1 .
they also showed that the dual of the quadratic objective function in the block - coordinate step can be interpreted as a recursive @xmath0-penalized least square solution .
friedman , hastie & tibshirani  @xcite used this idea successfully to develop an algorithm known as _ graphical lasso_. using a coordinate descent approach to solve the lasso problem speeds up the algorithm to a considerable extent , making it quite fast and effective for a large class of high - dimensional problems .    in all the aforementioned methods , information on the local geometry
is not taken into consideration .
often one encounters data that are measured on a certain manifold .
e.g. , data describing some feature on the outline of a ( moving ) silhouette , or pixels or voxels in an 2-d or 3-d image , respectively . in most of these problems , spatially close variables have a structural resemblance in terms of probabilistic dependence .
exploiting this local behavior might lead to faster and/or more efficient estimation .
honorio , ortiz , samaras et al .
@xcite introduced the notion of _ local constancy_. according to them , if one variable @xmath10 is conditionally ( in)dependent of @xmath11 , then a local neighbor of @xmath10 in that manifold is likely to be conditionally ( in)dependent of @xmath11 as well .
they developed a__coordinate direction descent algorithm _ _ to solve a penalized mle problem that penalizes both the @xmath0-norm of the precision matrix and the @xmath0-norm of local differences of the precision matrix , expressed as its  diagonal excluded product " with a local difference matrix .
local geometry has been addressed , although implicitly , by tibshirani et al .
@xcite in the context of fused lasso .
chen et al .
@xcite used this idea and proposed _ graph guided fused lasso _ for structure learning in multi - task regression , where the output space is continuous and the outputs are related by a graph . the input space is high dimensional and outputs that are connected by an edge in the graph are believed to share a common set of inputs .
the goal then is to learn the underlying functional map from the input space to the output space in a way that respects the similar sparsity pattern among the covariates that are believed to affect the output variables that are connected .
local smoothing by penalizing differences of neighboring nodes has been discussed in kovac and smith  @xcite in the context of nonparametric regression based on observations on nodes of a graph . their algorithm aims to split the image into active sets and subsequently merge , split or amalgamate them in order to minimize a penalized weighted distance .
like greenshtein & ritov  @xcite and meinshausen & bhlmann  @xcite we work in a set - up where the number of nodes in the graph @xmath12 and the covariance matrix @xmath13 depend on the sample size .
consider the @xmath12-dimensional multivariate random variable @xmath14 .
the conditional ( in)dependence structure of this distribution can be represented by the graph @xmath15 , where @xmath16 is the set of nodes corresponding to each coordinate variable and @xmath17 the set of edges in @xmath18 .
a pair of nodes @xmath19 lies in @xmath17 if and only if @xmath10 is conditionally dependent of @xmath11 , given all other remaining variables @xmath20 .
the neighborhood @xmath21 of a node @xmath22 is defined as the smallest subset of @xmath23 such that given @xmath21 , @xmath5 is conditionally independent of all the remaining nodes . in other words ,
neighbors of a certain node consists of the coordinates that are conditionally dependent on that particular node . as already mentioned , the conditional independence can be represented by occurrences of zero entries at respective cells of precision matrix @xmath24 , i.e. , @xmath25 iff @xmath26 . the neighborhood selection / model selection algorithms aim to find all the neighbors of a node @xmath5 , given @xmath27 i.i.d .
observations of @xmath3 .
meinshausen & bhlmann considered this as a penalized regression problem , where each variable is regressed on the remaining variables with an @xmath0 penalty on the estimated coefficients .    by the above definition of neighborhood , we have , for all @xmath28 that @xmath29 an alternative definition of neighborhood of @xmath10 is given as the non - zero components of @xmath30 where @xmath30 is given by @xmath31 in light of the above definition , the set of neighbors of a node @xmath28 is @xmath32 given the domain knowledge we construct a regular graph @xmath33 that is representative of the underlying spatial geometry .
we call it a _ local neighborhood graph_. for example , @xmath33 could be a chain graph , a two or three dimensional lattice or more generally a collection of cliques or wheels .
let @xmath34 denote the edge set corresponding to @xmath33 , then define :    [ localgraphdefn ] @xmath10 and @xmath11 are _ local neighbors _ with respect to @xmath33 if the edge connecting them belongs to @xmath35    note that the edge between @xmath10 and @xmath11 need not belong to @xmath17 .
now if one incorporates the _ local constancy _ property to this graph , the neighborhood becomes more structured . by local constancy ,
if @xmath36 is conditionally independent of @xmath5 given the other nodes ( and hence , there is no edge between @xmath22 and @xmath37 ) , it is likely that a _ local neighbor _ @xmath38 of @xmath36 is also conditionally independent of @xmath10 , making both @xmath39 and @xmath40 equal to zero .
thus , the zeroes of @xmath30 are expected to reflect the sparsity pattern .
we now generalize the traditional fused lasso algorithm  @xcite to satisfy our purpose of estimating the zeros and non - zeros of the precision matrix by exploiting the locality information given by the local neighborhood graph @xmath33 . before doing that we carefully define
the _ difference matrix_. consider an @xmath41 matrix @xmath42 where @xmath43 is the total number of pairs of local neighbors . assuming that we have a labelled sequence of nodes @xmath44 , arrange the pairs of local neighbors in a sequence @xmath45 where @xmath46 denotes the set of local neighbors of @xmath47 . the inequality @xmath48 is included to ignore double counting .
note that @xmath49 is nothing but a convenient ordering of edges in @xmath33 , and hence @xmath49 contains the same information as @xmath33 .
it should also be mentioned that our results are not influenced by the particular labelling used .
the @xmath50 row is given by @xmath51 where @xmath52 is the @xmath50 element of the local neighbor sequence and @xmath53 denote two canonical basis vectors of @xmath54 where the @xmath55 s occur at @xmath56 and @xmath57 position , respectively .
this way each pair is represented by a row in the difference matrix .
we denote by @xmath58 the @xmath59 sub - matrix of @xmath42 selecting all the rows with @xmath60 entry being 0 . in other words , @xmath58 is the difference matrix corresponding to all the local neighbor pairs not involving @xmath10 .
the number of local neighbors of @xmath10 in @xmath23 is @xmath61 .
it should be noted that throughout our discussion we shall assume that the local neighborhood structure is known to us , meaning that @xmath42 is known beforehand and it does not depend on the data .
now we define the neighborhood - fused lasso estimate @xmath62 of @xmath63 @xmath64 where @xmath65 denotes the @xmath0-norm of @xmath66 . penalizing both the @xmath0-norms of @xmath67 and @xmath68 implies parsimony , thus ensuring sparsity and local constancy at the same time .
this property helps us in variable selection and thereby leads to neighborhood selection .
note that we are referring to both the non - local and the local neighbors .
the neighborhood estimate of node @xmath22 is defined by the nodes corresponding to non - null coefficients when @xmath10 is regressed on the rest of the variables with the fused lasso penalty in  , in other words @xmath69 where @xmath70 denotes the @xmath37-th component of the vector @xmath62 .
it is clear that the selected neighborhood depends on the value of @xmath71 and @xmath72 chosen .
large values of @xmath71 and @xmath72 will give rise to more sparse solutions .
usually the regularization parameters are chosen by some cross validation criteria .
but in this paper we will find a data driven approach for selection of regularization parameters that speeds up computation and ensures asymptotic consistency in model selection .
meinshausen and bhlmann derived a data driven choice for @xmath71 in their paper .
we will extend their method for simultaneous selection of @xmath71 and @xmath72 from the data .
here we discuss further formalizations of the notion of local constancy . it should be noted again that the definition is linked with the ideas used by honorio et al .
@xcite , but our definition is more general in nature . first we introduce the notion of _ diagonal excluded matrix product_. given a matrix @xmath73 , its zero operator is defined as @xmath74 where @xmath75 is the indicator function .
the diagonal excluded product of two matrices @xmath73 and @xmath76 can now be defined as , @xmath77 where @xmath78 denotes the hadamard product of matrices . although the name does not clearly show how diagonals are removed from the product , usually if @xmath73 is taken to be a difference matrix defined above , this will eventually lead to the exclusion of diagonals of the matrix @xmath76 .
first observe that the absolute values of the non - zero entries of @xmath79 are of the form @xmath80 where @xmath81 corresponds to a pair of local neighbors .
in fact , we have @xmath82 equals the sum of all the absolute values of the differences @xmath83 for local neighborhood pairs @xmath52 . in other words , if we think of the entires @xmath84 as measuring conditional dependence of two nodes , then @xmath82 measures in how different the local neighborhood pairs are with respect to conditional dependence .
a small value of @xmath82 indicates that the local neighbors behave similarly in this respect .
thus , one way to define the local constancy property is to impose a bound on the norm of the difference of local neighbors .
we say a model exhibit @xmath85-local constancy if @xmath86 where @xmath87 denotes the @xmath88 norm . for our purpose ,
the best candidate would be @xmath89 for certain desirable properties like sparsity and convexity .
however , a different choice of norm would lead to a different solution which might be more appropriate for other types of problems . with @xmath90 ,
this definition coincides with honorio s local penalty criterion .
it should be noted that the @xmath91 parameter controls the degree of local constancy so this is closely related to the tuning parameter we are going to use .
this serves as a constraint on certain matrix parameters that we are trying to estimate .
a small @xmath91 would ensure a high level of local constancy where a large @xmath91 would do the opposite .
since the locality information is being used here as a prior information , it makes sense to interpret it from a bayesian perspective .
let @xmath92 denote a probability distribution on the space of positive definite matrices .
then one can say that an @xmath93-local constancy property holds for this model if @xmath94 here we need to assume that @xmath95 .
the @xmath91 parameter controls the degree of local constancy and the @xmath96 parameter is indicative of our prior belief about the local constancy .
one can also think of defining local constancy by replacing @xmath24 in sections [ locconstdefn : quant ] and [ locconstdefn : bayes ] by the partial correlation matrix @xmath97 . even though the definition makes sense , the natural transition from the difference matrix used in @xcite to our analogue @xmath58 will not be consistent with it .
the difference terms corresponding to partial correlation matrix will be @xmath98 but , when we regress @xmath99 on the rest of the nodes , then the local difference we aim to penalize is @xmath100 . in order to be consistent with our definition of neighborhood
fused lasso , we need to replace @xmath101 by @xmath102 , where @xmath103 . assuming that the individual conditional variances are bounded away from zero ( assumption a2 in section  [ ass : variance ] ) , this rescaling does not affect the local constancy and our penalty is consistent with the definition .
partial correlations have the advantage of all being on the same scale .
this might in particular simplify the choice of @xmath104
first we briefly review the existing algorithms used for optimization of fused lasso type problems . broadly speaking , there are two fundamental classes of algorithms used in this type of problems : ( a ) solution path algorithms - which finds the entire solution for all values of the regularization parameters and ( b ) approximation algorithms - which attempt to solve a large scale optimization given a fixed set of regularization parameters using first order approximations .
friedman et al .
@xcite formulated the path - wise optimization method for the standard fused lasso signal approximation problem where the design matrix @xmath105 .
the algorithm was two - step and the final solution is obtained by soft - thresholding the total - variation norm penalized estimate obtained in the first step .
the basic challenge in applying the coordinate descent algorithm to a fused lasso problem is the non - separability of the total variation penalty unlike usual lasso where the @xmath0-penalty is completely separable .
so they used a modified coordinate descent approach where the _ descent _ step was followed by an additional fusion and smoothing step .
however , this method works only for the total variation penalty and does not extend to fused lasso regression problems with generalized fusion penalty like our situation .
hoefling  @xcite proposed a path - wise optimization algorithm for generalized fusion penalties in fused lasso signal approximation problem .
this algorithm uses the fact that when varying the penalty parameter , the corresponding sets of fused coefficients do only change at finitely many values of the penalty parameter .
tibshirani and taylor  @xcite proposed another path algorithm for the fused lasso regression problem with a generalized fused lasso penalty by solving the dual optimization problem .
however , the path algorithm they devised can be applied only when the design matrix is full rank and hence is not applicable for high dimensional problems . in order to resolve the problem when the matrix is not full rank , they proposed to add an infinitesimal perturbation @xmath106 .
however , this does not solve the problem completely as a small @xmath91 leads to ill - conditioning and increasing number of rows in the generalized fused penalty matrix causes inefficient solutions because of an increasing number of dual variables .
approximation algorithms were developed to find efficient solutions to general fused lasso problem with a fixed set of penalty parameters , regardless of its rank and they usually adapt themselves to high dimensional problems quite easily . usually these approximation algorithms are based on first order approximation type methods like gradient descent .
liu et al .
@xcite proposed the efficient fused lasso algorithm ( efla ) that solves standard fused lasso regression problem by replacing the quadratic error term in the optimization function by its first order taylor expansion at an approximate solution followed by an additional quadratic regularization term .
the approximate objective function has a fused lasso signal approximation form and can be solved by applying gradient descent on its dual which is a box - constrained quadratic program .
chen et al .
@xcite proposed the _ smoothing proximal gradient _ method to solve regression problem with structured penalties that closely resemble our objective function .
the basic idea is to approximate the fused penalty term @xmath107 by a smooth function and devise an iterative scheme for an approximate optimization .
they use the smooth function @xmath108 convexity and continuous differentiability of this function follows from nesterov  @xcite .
one may now proceed using standard first order approximation approach like fista  @xcite which works like efla .
however this process is computationally intensive and not a feasible approach in our case where we need to do a nodewise regression .
another algorithm for tackling a similar problem is known as _ split bregman _
algorithm  @xcite .
it was proposed for standard fused lasso regression and later extended to generalized fused lasso regression .
the sb algorithm is derived from augmented lagrangian  @xcite , which adds quadratic penalty terms to the penalized objective function and alternatingly solves the primal and dual starting from an initial estimate .
this is also computationally intensive unless one has a simple structure constraint on the parameters .
we propose a different algorithm to solve our optimization problem .
we show that the neighborhood - fused lasso problem can be reparametrized into a standard lasso problem , thus simplifying the optimization procedure .
the neighborhood - fused lasso estimate @xmath62 of @xmath109 can be written as @xmath110 we note that since @xmath111 has full column rank , @xmath112 has full row rank .
thus , the following lemma can be applied here .
[ optim ] let @xmath113 and @xmath114 also assume that @xmath115 is of full column rank .
then @xmath116    one interesting observation here is that although we started with the assumption that @xmath117 , where @xmath118 denotes the column space of @xmath111 , we did not optimize over @xmath118 but did the same over all @xmath119 .
see the proof of lemma  [ optim ] for details .
the heuristic idea behind this is that we find the minimizer on @xmath118 by projecting the global minimizer onto @xmath118 and the projection operator is given by @xmath120 .
the objective function in   combines the two @xmath121 penalties into a single @xmath121 penalty and thus could be easily solved by any of the standard lasso algorithms .
the parameters @xmath71 and @xmath72 are chosen according to theorem  [ regparam ] .
we show by several simulations that the proposed method performs better than meinshausen - bhlmann s method or graphical lasso in situations where local constancy holds .
from discussion in section 3 , it can be seen that using neighborhood - fused lasso leads to efficient model selection when applied successively to all the nodes . in this section , we show that our proposed method leads to asymptotically consistent model selection similar to the procedure by meinshausen and bhlmann .
the choice of the regularization parameters is crucial in such cases .
moreover , we show that our proposed choice of regularization parameters not only ensures convergence to the  true " model but the convergence is faster than the meinhausen - bhlmann s method when the underlying model is indeed locally constant .
this property is also supported by our simulation results shown in section  [ simulations ] .      in order to prove consistency of our method for gaussian graphical models
, we need to work with the following assumptions .
assumptions [ a1]-[a3 ] and [ a5]-[a7 ] are as in meinshausen and bhlmann s ( see @xcite , section 2.3 ) .
however we need the two additional assumptions [ a4 ] and [ a8 ] to deal with the local constancy .
[ a1 ] : :    [ ass : dimensionality ] * dimensionality * :    @xmath122 , such that    @xmath123 as @xmath124 .
note that @xmath125 is included thus allowing    @xmath126 .
[ a2 ] : :    [ ass : variance ] for all @xmath28 and    @xmath127 , @xmath128 .
there exists    @xmath129 , so that for all @xmath127 and    @xmath28 ,    @xmath130 .
this    means that all the conditional variances are bounded away from 0 .
[ a3 ] : :    [ ass : sparsity ] * sparsity * : there exists some    @xmath131 so that    @xmath132    for @xmath124 .
[ a4 ] : :    [ ass : lnsparse ] * local neighborhood sparsity * : the number of local    neighbors also can grow at polynomial rate of @xmath27 , i.e. ,    @xmath133 such that the maximum number of    local neighbors of a node    @xmath134    for @xmath135 . for convenience we let    @xmath136 be such that @xmath137 [ a5 ] : :    [ ass : bounded ] * @xmath0-boundedness * : there exists some    @xmath138 so that for all neighboring nodes    @xmath139 and all @xmath127 ,    @xmath140 .
[ a6 ] : :    [ ass : parcor ] * magnitude of partial correlation * : there exists a    constant @xmath141 and some @xmath142    with @xmath143 as in [ a3 ] , so that for every    @xmath144 ,    @xmath145 where    @xmath146 denotes the partial correlation of    @xmath10 and @xmath11 .
[ a7 ] : :    [ ass : ns ] * neighborhood stability * : define    @xmath147 .
there exists some @xmath148 so that for all    @xmath149 with    @xmath150 , @xmath151 [ a8 ] : :    [ lnstable ] * local neighborhood stability * : let    @xmath152    and    @xmath153_k \theta^{b,\mathcal{l}_a}_k$ ] .
there exists some @xmath154 so that for all    @xmath155 such that    @xmath156 ,    @xmath157 , where    @xmath158 denotes denotes the    @xmath159 column of @xmath58 .
similar to meinshausen and bhlmann s interpretation , we can describe an intuitive condition which implies the last two assumptions .
define @xmath160 according to the characterization of @xmath161 derived from  , @xmath162 .
one can think of a two - dimensional perturbation approach in which one tweaks the parameters @xmath163 and @xmath164 in a way that the perturbed neighborhood @xmath165 is identical to original neighborhood @xmath166 .
the following proposition shows that the two assumptions of neighborhood stability are fulfilled under this situation .
the terms @xmath167 and @xmath168 measure the sub - gradient of the lasso penalty and the neighborhood - fused lasso penalty respectively . having a small @xmath0 bound on them enforces the stability of the estimated coefficients , and hence stability of neighbors .
see section  [ proofs ] for proof of the proposition .
[ perturb ] if there exists some @xmath169 such that @xmath170
. then , @xmath171 and @xmath172 .
moreover , @xmath173 .
we start the presentation of the theoretical results with a lemma ( proven in section  [ proofs ] ) characterizing the minimizer of our objective function in terms of its subdifferential .
[ baselemma ] given @xmath174 , let @xmath175 be a @xmath12 dimensional vector with elements @xmath176 .
define @xmath177_{b }   \qquad \text{and } \qquad \mathcal{l}^a ( \theta)= \{b:{\cal d}^a_b\ne 0\}.\]]a vector @xmath178 is a solution to the fused lasso problem described above iff    @xmath179    this lemma builds the foundation of several of the following results and will be used frequently to prove them .
sign consistency is one of the major properties that a model selection method should exhibit .
before we study the model selection consistency of our estimators , we show , in the following lemma that the neighborhood - fused lasso estimator is sign - consistent .
[ signconsist ] let @xmath62 be defined for all @xmath22 . under the assumptions [ a1]-[a7 ] , it holds for some @xmath180 that for all @xmath22 , @xmath181    observe that this lemma , in turn , preserves the asymptotic equality of signs of local neighbors .
if @xmath11 and @xmath182 are local neighbors , then with high probability , when regressing @xmath5 on the remaining variables , the coefficients of @xmath36 and @xmath38 will have the same sign .
this is a direct consequence of local constancy of the estimated regression coefficients .
our results show that , just like in meinshausen and bhlmann , a rate slower than @xmath183 is necessary for the regularization parameters for consistent model selection in the high dimensional case where the dimension may increase as a polynomial in the sample size .
specifically if @xmath71 decays at @xmath184 and @xmath72 decays as @xmath185 where @xmath186 are as in assumptions [ a1]-[a8 ] , the estimated neighborhood is almost surely contained in the true neighborhood .
hence the type - i error probability goes to 0 .
this is formally stated in the following theorem ( see section  [ proofs ] for a proof ) .
[ type1 ] let assumptions [ a1]-[a8 ] be fulfilled . let the penalty parameters satisfy @xmath187 and @xmath188 with some @xmath189 and @xmath190 and @xmath191 .
there exists some @xmath192 such that for all @xmath28 ,    @xmath193    the assumptions of neighborhood stability and local neighborhood stability are not redundant .
the following proposition shows that one can not relax the assumptions a7 and a8 .
[ type1problem ] if there exists some @xmath139 with @xmath150 and @xmath194 and @xmath195 , then for @xmath196 as in theorem 4.6 , @xmath197    on the other hand , with the same sets of assumptions , one can show that the type ii probability , i.e. , probability of falsely identifying an edge as a potential connection exponentially goes to 0 .
this has been formally stated and proved in the following theorem ( see section  [ proofs ] for a proof ) .
[ type2 ] with all the assumptions of theorem 4.6 and @xmath198 as before , @xmath199      it follows from the discussion so far that consistent estimation of nodes is possible using our approach . however , one of the original goals of our method is to estimate the entire underlying graphical model , which can be accomplished by combining the estimated neighborhoods in some way .
two different methods of combination have been proposed @xmath200 in our simulations , we combined them following the union method .
it has been observed that the differences vanish asymptotically when a regular lasso is applied .
although we did not theoretically study this for our case but our experiments indicate that they do not vary much asymptotically .
theoretic exploration in asymptotic domain does not cast much light on the choices of regularization parameter in real life , finite sample problem .
it is hard to ensure consistency or absolute containment like theorem  [ type1 ] or  [ type2 ] .
however , following the idea proposed by meinshausen and bhlmann , one can consider the _ connectivity component _ of a node , which is defined as the set of nodes which are connected to it through a chain of edges .
generalizing the results from meinshausen & bhlmann , the following theorem shows ( see proof in section  [ proofs ] ) that the estimated connectivity component derived from neighborhood - fused lasso estimate will belong to the true connectivity component with probability @xmath201 , for any chosen level of @xmath202 .
[ regparam ] with all the assumptions [ a1]-[a8 ] , and the following choices of the penalty parameters , @xmath203 we have    @xmath204    for all @xmath27 . @xmath205 and @xmath206 are the true and estimated connectivity components of @xmath22 , @xmath207 are certain constants and @xmath208 .
the choice of @xmath209 and @xmath210 depends on the rate of growth of the local neighborhood with increasing sample size . in our simulations
, we found that for models with constant dimension and increasing sample size , @xmath211 and @xmath212 works fine .
+ as shown in the simulations , we get a higher convergence speed as compared to meinshausen - bhlmann s method by applying a neighborhood - fused lasso penalty for nodewise regression when the underlying model exhibit local constancy .
this can be proved using the following lemma ( proof in section  [ proofs ] ) .
[ fastconv ] the upper bound of type i error probability in neighborhood fused lasso is smaller than that of the meinshausen & bhlmann procedure .
delves deep into the precise constants of in the proof of theorem [ type1 ] and shows that one could achieve smaller type 1 error probability with a finite sample size if one uses neighborhood fused lasso instead of usual lasso . ]
lemma  [ fastconv ] does not specify how much the maximal false positive probability is reduced .
this can be understood after going through its proof .
roughly speaking , the reduction in the upper bound is given by @xmath213 \cdot e^{-\frac{1}{\sigma^2_*}\left(\frac{d^2_1}{4}(1-\delta_1)^2 n^{\epsilon}\right ) } ,      \end{aligned}\ ] ] where @xmath214 , and @xmath215 , @xmath216 are as in the proof of theorem [ type1 ] . essentially , it is shown in the proof that the neighborhood fused lasso reduces a dominant portion of the type 1 error ( given by the second factor ) by a fraction ( given by the first factor ) shown above .
looking at the proof of theorem  [ type1 ] , it is seen that the term @xmath217 is the principal contributor to the probability of false positives .
the corresponding term using usual lasso is @xmath218 lemma  [ fastconv ] makes use of this fact and it also shows that substantial reduction takes place when the local neighborhood grows .
so , this method invariably performs better than the meinshausen - bhlmann s method with respect to a minimax criterion ( in a sense that is minimizes the maximum probability of false positives ) , and the improvement is more when the local neighborhood grows faster .
it also implies that it can perform as bad as usual lasso regression as the worst case scenario .      in our attempt to gaussian graphical model learning
, we adopt the meinshausen - bhlmann approach , i.e. , do a componentwise penalized regression .
however , one should keep in mind that in the process of doing so , the design matrix ( which is random here ) changes at every iteration . in previous section
we discussed the asymptotic model selection consistency of our estimator . in this section
, we shall go beyond model selection and explore conditions under which our neighborhood - fused lasso estimate exhibit nice asymptotic @xmath0 properties .
we shall carry out our theoretical analysis under the assumption that the linear regression model holds exactly , with some underlying  true " @xmath219 .
most of the theoretical results in this section have been derived in the light of discussions in @xcite .
we start with a quick recapitulation of the notation we are going to use here .
if @xmath220 , one assumes a node - wise regression model @xmath221 where @xmath10 denotes the @xmath22-th component , @xmath222 denotes all the components except @xmath22 and @xmath223 for some @xmath224 .
also assume that @xmath225 and @xmath226 is the precision matrix . if @xmath227 denotes the edge set in the conditional independence graph then @xmath228 .
the design matrix @xmath222 is obtained by deleting the @xmath22-th column of the data matrix .    in the high dimensional
set up , where one generally has lesser number of samples that model dimension @xmath229 , we assume an inherent sparsity in the true @xmath109 .
this sparsity is also ensured if we assume that the underlying conditional independence graph is sparse .
let us assume @xmath230 and @xmath231 .
since @xmath232 is not known , one needs a regularization penalty .
meinshausen and bhlmann chose the @xmath0 penalty , i.e. , the traditional lasso and got the estimate @xmath233.\end{aligned}\ ] ] in our situation , we have added another penalty term @xmath234 along with the lasso penalty term .
hence , our estimator is given by @xmath235.\end{aligned}\ ] ] note that this new definition of our estimator is exactly same as what was defined in  [ nfldefn ] .
as mentioned in the earlier section , we shall develop our theory based on the assumption of a linear truth .
we try to provide an upper bound on the prediction error .
the following lemma forms the basis of our derivation .    * the basic inequality*[basicineq ] @xmath236    it
should be noted that the number of non zero elements in the column of the matrix @xmath58 denotes the number of local neighbors that particular node has ( except node @xmath22 ) .
let us assume that the number of local neighbors is @xmath237 .
fixing @xmath27 , let us also assume that the number of local neighbors is bounded by @xmath76 .
then , it follows from lemma  [ basicineq ] using @xmath238 that @xmath239 the basic objective of using a penalty parameter is to overrule the empirical process term @xmath240 .
it can be easily seen that @xmath241 our goal is to choose a @xmath71 and a @xmath72 such that the probability that the empirical process term in the right hand side exceeds @xmath242 is small , so that with high probability the right hand side could be dominated by @xmath243 . to that effect ,
we define @xmath244 our objective is to show that for some particular choice of @xmath245 and @xmath246 , @xmath247 has high probability .
now one should keep in mind that both @xmath248 and @xmath249 are random here .
one can make the valid assumption of their individual gaussian law and independence .
we formalize these notions in the following proposition .    under the assumption of linear truth for the gaussian graphical model , i.e. , @xmath250 , we have @xmath251    the proof of this proposition is straightforward and hence skipped .
the following results show that for suitable choice of @xmath245 and @xmath246 , @xmath252 is high .    [ highprob ] under the assumption that @xmath253 , @xmath254 , \intertext{in particular , with $ \lambda_0 = \frac{2(t + \log p)}{n } > 0 $ and $ \mu_0 = \frac{2}{b}\sqrt{\frac{2}{n}(t + \log p)}$ , then } p(\lambda_a ) & \geq 1 - 2 e^{-t}.\end{aligned}\ ] ]    [ highprobcor ] assume that @xmath253 and that @xmath255 .
let the regularization parameters be @xmath256 then the following is true @xmath257    the following lemma provides an upper bound on the estimation error .
[ l2ineq ] assume that @xmath253 and that @xmath255 .
let the regularization parameters be @xmath258 also let @xmath259 be the smallest non - zero singular value of @xmath222 .
then we have @xmath260 \geq 1 - e^{-t^2}.\end{aligned}\ ] ]      we now try to derive some oracle inequalities which will provide @xmath0 bound for our neighborhood - fused lasso estimate . following bhlmann s notation ,
let us write , for an index set @xmath261 , @xmath262 we need some more notation . split the matrix @xmath58 as follows : @xmath263,\end{aligned}\ ] ]
where @xmath264 consists of all rows such that both the non zero terms belong to @xmath265 .
@xmath266 consists of all rows and columns such that exactly one of the non - zero term in that row belongs to @xmath265 .
@xmath267 consists of all rows and columns such that exactly one of the non - zero term in that row belongs to @xmath268 .
@xmath269 consists of all rows such that both the non - zero terms belong to @xmath265 .
[ oracle1 ] on @xmath247 , if we choose @xmath270 and @xmath271 , we have @xmath272    it can be easily verified that if @xmath273 for some @xmath274 , we have @xmath275 this helps us to simplify the consequences lemma  [ oracle1 ] , which implies that @xmath276 combining with the aforementioned condition , we get @xmath277 a standard way to relate the @xmath0 penalty @xmath278 to an @xmath6 penalty is to use the cauchy - schwarz inequality @xmath279 where @xmath280 is some constant .
however , @xmath281 being random , this condition can not hold unanimously for all @xmath282 .
bhlmann provided a _ compatibility condition _ so that this is true .
in our situation , we found a similar condition like them to carry out further analysis .
it should be noted that our compatibility condition is weaker than the compatibility condition bhlmann provided .
however , we need to work under the assumption that @xmath283 .
the definition of this _ compatibility condition _ is provided below .
[ compat ] ( compatibility condition ) we say that a collection of nodes @xmath284 satsifies the @xmath285-compatibility condition if for all @xmath67 satisfying @xmath286 we have @xmath287    following bickel et al .
@xcite , we shall refer to @xmath288 as the _ restricted
eigenvalue_. here we provide a detailed explanation of this phenomenon .
observe that the compatibility condition could be re - written as @xmath289 @xmath288 can be assumed to be the minimum eigenvalue over the restricted set @xmath290 . with this compatibility condition imposed , we derive the following oracle inequality    [ oracle2 ] assume that the compatibility condition ( from definition  [ compat ] ) holds for some @xmath291 .
then on @xmath247 , for @xmath270 , @xmath292 and @xmath273 , we have @xmath293    combining theorem  [ oracle2 ] and lemma  [ highprob ] , we get ,    [ finalthm ] let @xmath294 and assume that @xmath295 .
then with probability @xmath296 , @xmath293    theorem  [ finalthm ] provides a unified way to deal with both the squared estimation error and the absolute error of the estimated parameters in neighborhood fused lasso regression .
it can be seen that for increasing @xmath27 , with probability increasing to 1 , the absolute difference of our nfl estimator from underlying truth is bounded by @xmath297 for large values of @xmath27 , this is approximately equal to @xmath298 which is a positive and decreasing function of @xmath299 if @xmath300 .
this is automatically satisfied for all @xmath291 .
so , it can be easily seen , using the assumption @xmath301 , that the above quantity is bounded by @xmath302
in this simulation we repeat the exact scenario presented in honorio s first simulation setting . the gaussian graphical model
consists of 9 variables as shown in figure [ nflvscdd ] .
it deals with both local and non - local interactions .
our method is compared to meinshausen - bhlmann s method and graphical lasso .
we run our simulation for 4 different sample sizes @xmath303 & @xmath304 . for each sample size , we run the simulation 50 times and estimate the neighborhood for each iteration . we construct a weighted graph with edge weight corresponding to the frequency of its occurrence in all of those 50 iterations .
+      in this simulation study we take a 50 dimensional normal random vector with zero mean .
the diagonals of the precision matrix are all @xmath55 and all the nonzero off - diagonal entries are @xmath305 .
the conditional ( in)dependence graph of this vector consists of both spatial ( local ) and non - spatial neighbors where the local neighborhood structure is linear ( one dimensional lattice ) .
there are two groups of distant neighbors .
we generate @xmath27 i.i.d .
samples from the corresponding normal distribution .
we run the simulation for @xmath306 .
we try to reconstruct the conditional ( in)dependence graph from the data using graphical lasso ( we use an oracle version of graphical lasso where the choice of penalty parameter is contingent on the actual number of edges in the true model ) , meinshausen - bhlmann s coordinate - wise lasso and our coordinate - wise generalized fused lasso approach . for each @xmath27
, we run the simulation 50 times and calculate the number of correctly identified edges and that of falsely identified edges for each iteration . the mean and standard deviations
are shown in the table [ bigmodeltable1 ] and table [ bigmodeltable2 ] .
figure [ bigmodelcomp ] shows the relative performance of the competing methods for different sample sizes .
sample size increases from top to bottom . in the figure
we include comparison for two additional sample values .
they are @xmath307 and @xmath308 respectively .
+    .comparison of false positives and true positives [ cols="^,^,^,^,^,^,^,^",options="header " , ]     it is quite evident from the two simulation that our method converges to the true model faster than meinshausen - bhlmann s method .
it is also shown theoretically in lemma  [ fastconv ] .
in this paper , we have successfully extended the meinshausen - bhlmann approach of model selection in gaussian graphical models where the assumption of local constancy holds .
we also provided a rigorous and generalized definition of local constancy and used it to propose neighborhood - fused lasso ( nfl ) , an algorithm to solve the problem by penalizing the differences of local neighbors given by a pre - determined graph @xmath33 .
we have used a generalized version of fused lasso in order to accomplish our objective .
our algorithm reduces the fused lasso problem into a regular lasso problem for a given set of regularizing parameters and thereby fast solutions are readily available .
we were able to provide data dependent choices for the tuning parameters in order to establish asymptotic consistency in model selection .
we substantiated our theoretical discussion on asymptotic model selection consistency with simulations that furnished desired results .
we also proved , both theoretically and experimentally , that incorporating local constancy ensures faster convergence to the underlying truth , meaning that similar accuracy is achieved with smaller sample size .
this phenomenon is somewhat similar to what buhl  @xcite and uhler  @xcite observed while investigating the existence of mle s in gaussian models with certain geometric structures .
they were able to prove the existence of mle of a higher dimensional model with relatively smaller sample size .
our findings are somewhat reminiscent of these results .
we also discussed about the compatibility issue while doing regression with local constancy and were able to derive sufficient conditions under which @xmath0 boundedness of estimated parameters is ensured . on top of that we also provided an upper bound for the quadratic prediction error .
+   + there are many extensions possible for future research .
firstly , instead of using the pre - defined local graph @xmath33 , one can think of alternative ways to grow or shrink the local neighborhood adaptively . secondly , as discussed in section [ locconstdefn ] , one can think of devising an algorithm that penalizes the differences of partial correlations instead of the elements of the precision matrix .
thirdly , ( because of the compatibility issue ) regular lasso or fused lasso might fail to perform up to the mark if the model is ill - conditioned or the compatibility assumptions do not hold .
one can think of using adaptive lasso or modify it accordingly in order to incorporate the local constancy property .
some preliminary simulations in this direction look promising .
thirdly , a bayesian approach could be pursued where one can start with some prior distribution on the inverse covariance matrix respecting the local constancy property and use the bayesian definition of local constancy in this paper ( see  [ locconstdefn : bayes ] ) to come up with a novel alternative method .
let @xmath309 . we will show that @xmath310 . we know from definition of @xmath311 that @xmath312 @xmath313    the proof can be found in [ suppa ] .
the subdifferential of @xmath314 is given by @xmath315 where @xmath316\quad if \quad \theta_{b}=0\}$ ] and @xmath317\quad if \quad \alpha_{b}=0 \quad \mathrm{where } \quad \alpha = d'^{a}\mathrm{sgn}(d^{a}\theta)\}$ ] .
observe that @xmath318 where @xmath319 denotes the @xmath320 column of the matrix @xmath321 .
this is same as the total number of local neighbors of @xmath37 except @xmath22 .
the lemma follows .
this proof can be found in [ suppa ] .
the event @xmath322 is equivalent to the event that there exists some node @xmath323 in the set of non - neighbors of node @xmath22 such that the estimated coefficient @xmath70 is not zero .
thus ,    @xmath324    let @xmath325 be the event that @xmath326 conditional on @xmath325 , it follows from meinshausen - bhlmann s discussion that @xmath327 is also a solution to the fused lasso problem with @xmath328 . as @xmath329 for all @xmath330
, it follows that @xmath331 for all @xmath330 . by  [ baselemma ] , @xmath332 it suffices to show that there exists a constant @xmath192 so that for all @xmath330 ,    @xmath333    writing @xmath334 for any @xmath323 where @xmath335 for some @xmath336 and @xmath337 is independent of @xmath338 .
hence , @xmath339\end{aligned}\ ] ] by lemma  [ signconsist ] , there exists a @xmath192 so that with probability @xmath340 ,    @xmath341 @xmath342    in this case , it holds by lemma  [ baselemma ] that @xmath343_m\theta^{b,\mathrm{ne}_a}_m \mu\right| \\ & \leq \delta_1\lambda + \delta_2 b\mu\end{aligned}\ ] ] the absolute value of the coefficient @xmath344 is hence bounded by    @xmath345    with probability @xmath346 .
conditional on @xmath347 , the random variable @xmath348 is normally distributed with mean @xmath349 and variance @xmath350 . by definition of @xmath351 ,
@xmath352    since @xmath353 , @xmath354 is stochastically smaller than or equal to @xmath355 .
it remains to show that for some @xmath192 and some @xmath356 ,    @xmath357    if @xmath10 and @xmath337 are independent , @xmath358 .
using gaussianity and bounded variance of both @xmath10 and @xmath337 , we obtain existence of some @xmath359 such that @xmath360 . hence using bernstein inequality and boundedness of @xmath71 and @xmath72
, it holds for some @xmath192 that for all @xmath361 ( see lemma  [ fastconv ] for detailed proof ) , @xmath362 which completes the proof .
we follow the proof of theorem  [ type1 ] and claim that with the above assumptions , for all @xmath363 with @xmath150 ,    @xmath364 for @xmath135    following similar arguments afterwards , it can be concluded that with some @xmath365 and @xmath366 as @xmath135    @xmath367    it holds for the third term that for any @xmath368 ,    @xmath369 as @xmath370    which combined with the previous result proves the proposition .
observe that    @xmath371    let @xmath325 be the event    @xmath372    on @xmath325 , following similar arguments as before , we can conclude that @xmath373 .
therefore    @xmath374    it follows from the proof of theorem 4.6 that there exists some @xmath375 so that @xmath376 . using bonferronis inequality , it hence remains to show that there exists some @xmath375 so that for all @xmath377 ,    @xmath378    which follows from lemma  [ signconsist ] .
@xmath379 implies the existence of an edge in the estimated neighborhood that connects two nodes in two different connectivity components of the true underlying graph .
hence ,    @xmath380    going by the same arguments used in proving theorem 4.6 , we have    @xmath381    thus , it is sufficient to show that    @xmath382    now observe that since @xmath11 and @xmath383 are in different connectivity component , they are , in fact , independent . therefore , conditional on @xmath384 , @xmath385 , making it stochastically smaller than @xmath386 .
hence it holds for all @xmath28 and @xmath387 that    @xmath388    where @xmath208 . using the @xmath71 and @xmath72 proposed ,
the rhs becomes @xmath389 .
this is a direct application of bernstein inequality . since @xmath390 and @xmath391 as @xmath135 , for large enough @xmath27 , we have , by a version of bernstein s inequality , that @xmath392\\     & \hspace*{3cm}= \exp\left[-\frac{1}{\sigma^2 _ * } \left(\frac{d_1}{2}(1 - \delta_1)+ \frac{d_2}{2}(1 - \delta_2 ) n^{\beta_0}\right)^2 n^\epsilon\right].\\      \intertext{this proves part ( a ) .
write the last expression as }   & e^{-\frac{1}{\sigma^2_*}\left(\frac{d^2_1}{4}(1-\delta_1)^2 n^{\epsilon}\right ) } \cdot e^{-\frac{1}{\sigma^2_*}\left(\frac{d_1 d_2}{2}(1 - \delta_1)(1 - \delta_2)n^{\beta_0 } + \frac{d^2_2}{4}(1 - \delta_2)^2 n^{2\beta_0}\right)n^{\epsilon } }      \intertext{proves part ( b ) , since the first factor is the upper bound for lasso . }
\end{aligned}\ ] ]    since @xmath281 is the fused lasso minimizer , we get @xmath393    the proof can be found in [ suppa ] .
the proof exploits the fact that @xmath281 is the minimizer of the penalized least square by equalling the sub - differential of the objective function to 0 .
we start by replacing the assumed linear truth , i.e. , @xmath394 .
therefore , we get by using the notation @xmath395 and @xmath396 that @xmath397_{\theta_a = \hat{\theta}^{\lambda,\mu}_a } = 0\\ & \rightarrow\quad \frac{2}{n}\left((x^a)^2\ , ( \hat{\theta}^{\lambda,\mu}_a - \theta_0 ) \right ) - \frac{2}{n}x^{a'}\epsilon_a + \lambda\mathrm{sgn}\left(\hat{\theta}^{\lambda,\mu}_a\right ) + \mu { \mathbb d}^a   = 0\\ & \rightarrow\quad ( x^a)^2 \hat{\theta}^{\lambda,\mu}_a =   ( x^a)^2\theta_0 + \frac{n}{2}\left(\frac{2}{n}x^{a'}\epsilon_a\right ) - \frac{n\lambda}{2}\mathrm{sgn}\left(\hat{\theta}^{\lambda,\mu}_a\right ) - \frac{n\mu}{2 } { \mathbb d}^a \\ & \rightarrow\quad \hat{\theta}^{\lambda,\mu}_a = \left((x^a)^2\right)^+ ( x^a)^2 \theta^0_a + \frac{n}{2}\left((x^a)^2\right)^+ \left(\frac{2}{n}x^{a'}\epsilon_a - \lambda\mathrm{sgn}\left(\hat{\theta}^{\lambda,\mu}_a\right ) - \mu { \mathbb d}^a\right).\end{aligned}\ ] ] with the given choices for @xmath71 and @xmath72 , define @xmath398 , then we have , with probability @xmath399 that @xmath400 where the inequality is meant to be interpreted componentwise and @xmath401 is a vector of size @xmath1 consisting of 1 s .
hence , we get @xmath402 using the facts that @xmath403 , where @xmath404 is the maximum singular value of @xmath73 and that @xmath405 is idempotent , and hence its singular values are either 0 or 1 , we can continue the above sequence of inequalities and obtain @xmath406 plugging in the result obtained from previous corollary , we get @xmath407 which implies that @xmath408    to start with , we derive a series of inequalities and apply them on the basic inequality .
@xmath409 we write @xmath410 and @xmath411 so that @xmath412 .
similarly we write @xmath413 .
hence , we get @xmath414 \left ( \begin{matrix } \hat{\theta}^{\lambda,\mu}_{a , s_0,1}\\ \hat{\theta}^{\lambda,\mu}_{a , s^c_0,1 } \end{matrix } \right)\right|\right|_1 =   \left|\left| \left ( \begin{matrix } d^a_{s_0,s_0 } \hat{\theta}^{\lambda,\mu}_{a , s_0,1}\\ d^a_{s_0,0 } \hat{\theta}^{\lambda,\mu}_{a , s_0,1 } + d^a_{0,s^c_0 } \hat{\theta}^{\lambda,\mu}_{a , s^c_0,1 } \\ d^a_{s^c_0,s^c_0 } \hat{\theta}^{\lambda,\mu}_{a , s^c_0,1 } \end{matrix } \right ) \right|\right|_1\end{aligned}\ ] ] @xmath415 similarly , we get @xmath416 \left ( \begin{matrix } \theta^0_{a , s_0,1}\\ 0 \end{matrix } \right)\right|\right|_1 \quad = \quad \left|\left| \left ( \begin{matrix } d^a_{s_0,s_0 } \theta^0_{a , s_0,1}\\ d^a_{s_0,0 } \theta^0_{a , s_0,1}\\ 0 \end{matrix } \right)\right|\right|_1\\ & = \quad \big\| d^a_{s_0,s_0 } \theta^0_{a , s_0,1}\big\|_1 + \big\| d^a_{s_0,0 } \theta^0_{a , s_0,1 } \big\|_1\end{aligned}\ ] ] plugging into the basic inequality , we get on @xmath247 , with @xmath417 and @xmath292 , @xmath418 from this , we get @xmath419    the proof is basically a continuation of what we have already shown .
we have @xmath420
the authors are thankful to dr .
debashis paul and dr .
ethan anderes who helped shape some ideas presented in this paper with insightful comments and discussions .
banerjee , o. , ghaoui , l. e. , daspremont , a. ( 2008 ) .
 model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data " , _ journal of machine learning research _ , vol .
485 - 516 .
chen , x. , lin , q. , kim , s. , carbonell , j.g . , and
xing , e.p .
 smoothing proximal gradient method for general structured sparse regression " . _
annals of applied statistics _ , vol .
719 - 752 .
tibshirani , r. , saunders , m. , rosset , s. , zhu , ji and knight , k. ( 2005 ) .  sparsity and smoothness via the fused lasso " , _ journal of royal statistical society _ , series b ( methodological ) , vol .
67 , part 1 , pp .