semi - supervised learning has attracted the attention of many researchers because of the importance of combining unlabeled data with labeled data . in many applications
the number of unlabeled data points is so large that labeling training data is expensive and time - consuming .
therefore , the problem of effectively utilizing a combination of unlabeled and labeled information is very important in machine learning research .
this paper concerns the issue of how to address uncertainty quantification in such classification methods . in doing
so we bring together a variety of themes from the mathematical sciences , including optimization , pdes , probability and statistics .
we will show that a variety of different methods , arising in very distinct communities , can all be formulated around a common objective function @xmath0 for a real valued function @xmath1 on the nodes of a graph representing the data points .
the matrix @xmath2 is proportional to a graph laplacian derived from the unlabeled data and the function @xmath3 involves the labelled data . the variable @xmath1 is used for classification .
minimizing this objective function is one approach to such a classification . a probability distribution related to the objective function has density @xmath4 proportional to @xmath5 probability of the labelling variable @xmath1 is high where the objective function is small , and vice - versa .
uncertainty quantification corresponds to using the probability distribution to compute expectations of test functions @xmath6 , defined on the nodes of the graph , which enable us to measure the variability of label variables , such as means and variances : @xmath7 in the settings of interest this will typically be a very high dimensional integral , with the dimension given by the number of unlabelled data points . carrying out this program requires computational algorithms to minimize @xmath8 or to draw samples , via monte carlo markov chain ( mcmc ) for example , from the probability distribution with density @xmath4 .
these algorithms exploit the fact that @xmath9 is a graph analogue of the dirichlet energy and will leverage analogies with pde - based methodologies involving the classical euclidean dirichlet energy in order to derive effective computational methods . in this paper
we will describe this confluence of ideas from different parts of the mathematical sciences , show how our approach builds on a broad range of advances in the field which we will review , and demonstrate the emergence of a problem area with many open challenges for the mathematical sciences .
we emphasize that the variety of probablistic models considered in this paper arise from different assumptions concerning the structure of the data .
our objective is not to assess the validity of these assumptions , which is a modelling question best addressed on a case - by - case basis , but rather we develop an overarching computational framework suitable for all the models arising from these different assumptions .      an effective method for semi - supervised
learning is to construct a similarity graph on both the unlabeled and labeled examples , and classify unknown labels by leveraging the graph structure .
a central conceptual issue in the setting of this problem is that labels are discrete , whilst similarity information is often continuous
. strategies to work with both of these settings simultaneously are at the heart of this subject . in @xcite ,
blum et al . posed the binary semi - supervised classification problem using a graph min - cut problem .
this is equivalent to a maximum a posteriori ( map ) estimator with respect to a bayesian posterior distribution for a markov random field ( mrf ) over the discrete state space of binary labels @xcite ; the resulting optimization problem can be solved exactly in polynomial time . in general , inference for multi - label
discrete mrfs is intractable @xcite .
however , several approximate algorithms exist for the multi - label case @xcite , and have been applied to many imaging tasks @xcite .
the probit classification method , using gaussian process priors , is described in @xcite ; however in that book the prior does not depend on the unlabelled data .
gaussian priors which depend on the unlabelled data may be constructed by using the graph laplacian , an approach undertaken in @xcite .
the model defined in @xcite is equivalent to a continuum relaxation of the discrete state space mrf in @xcite .
the bayesian formulation which underpins our work in this paper was made explicit in @xcite where a variety of likelihood models are used to condition on the labelled data ; the probit approach , for example , could be used to accomplish this .
probit utilizies the same prior as in @xcite but the data is assumed to take binary values , found from thresholding the underlying continuous variable , and thereby providing a link between the combinatorial and continuous state space approaches described in the previous paragraph .
the probit methodology is often implemented via map optimization  that is the posterior probability is maximized rather than sampled  or an approximation to the posterior is computed , in the neighbourhood of the map estimator @xcite . for full posterior exploration , gibbs sampling
is often used @xcite and this methodology has been applied recently in @xcite ; furthermore methods designed to break undesirable dependencies in the gibbs sampler are introduced in @xcite . in the context of map estimation , the graph - based terms act as a regularizer , in the form of the graph dirichlet energy @xmath10 .
a formal framework for graph - based regularization can be found in @xcite .
more recently , other forms of regularization have been considered such as the graph wavelet regularization @xcite .
another link between discrete combinatorial optimization approaches and methods based on optimization over real - valued variables was made in the work of bertozzi et al . @xcite .
the approach is based on the fact that the tv functional , when suitably generalized to weighted graphs , coincides with the graph cut energy .
relaxation of the tv functional is well - understood in the context of partial differential equations ( pde ) and generalizing ideas applicable to the pde laplacian in the context of the graph laplacian leads to new optimization methods .
based on this reasoning , in @xcite the graph ginzburg - landau functional was used as a relaxation of the graph tv functional for the task of binary classification .
this was generalized to multiclass classification in @xcite .
following this line of work , several new algorithms were developed for semi - supervised and unsupervised classification problems on weighted graphs @xcite .
a further connection with pde based methods is the level - set approach to bayesian inversion , introduced recently in @xcite ; this is very closely related to our variant on the probit method , as we will demonstrate .
there are a wide range of methodologies employed in the field of uncertainty quantification , and the reader may consult the books @xcite and the recent article @xcite for details and further references . underlying all of these methods is a bayesian methodology which is attractive both for its clarity with respect to modelling assumptions and its basis for application of a range of computational tools .
nonetheless it is important to be aware of limitations in this approach , in particular with regard to its robustness with respect to the specification of the model , and in particular the prior distribution on the unknown of interest @xcite .
whilst the book @xcite conducts a number of thorough uncertainty quantification studies for a variety of learning problems using gaussian process priors , most of the papers studying graph based learning referred to above primarily use the bayesian approach to learn hyperparameters in an optimization context , and do not consider uncertainty quantification .      in this paper , we focus exclusively on the problem of binary semi - supervised classification ; however the methodology and conclusions will extend beyond this setting .
our focus is on a presentation which puts uncertainty quantification at the heart of the problem formulation , and we make four primary contributions :    * we define a number of different bayesian formulations of the graph - based semi - supervised learning problem and we connect them to one another , to binary classification methods and to a varierty of pde - inspired approaches to classification ; in so doing we provide a single framework for a variety of methods which have arisen in distinct communities and we open up a number of new avenues of study for the problem area ; * we highlight the pcn - mcmc method for posterior sampling which , based on analogies with its use for pde - based inverse problems @xcite , has the potential to sample the posterior distribution in a number of steps which is independent of the number of graph nodes ; * we introduce approximations exploiting the empirical properties of the spectrum of the graph laplacian , generalizing methods used in the optimization context in @xcite , allowing for computations at each mcmc step which scale well with respect to the number of graph nodes ; * we demonstrate , by means of numerical experiments on a range of problems , both the feasibility , and value , of bayesian uncertainty quantification in semi - supervised , graph - based , learning .
the paper is organized as follows . in section [ sec:2 ] ,
we give some background material needed for problem specification . in section [ sec:3 ]
we formulate the four bayesian models used for the classification tasks .
section [ sec:4 ] introduces the mcmc and optimization algorithms that we use . in section [ sec:5 ] ,
we present and discuss results of numerical experiments to illustrate our findings ; these are based on four examples of increasing size : the house voting records from 1984 ( as used in @xcite ) , the tuneable two moons data set @xcite , the mnist digit data base @xcite and the hyperspectral gas plume imaging problem @xcite .
we conclude in section [ sec:6 ] .    to aid the reader , we give here an overview of notation used throughout the paper .
* @xmath11 the set of nodes of the graph , with cardinality @xmath12 ; * @xmath13 the set of nodes where labels are observed , with cardinality @xmath14 ; * @xmath15 , feature vectors ; * @xmath16 latent variable characterizing nodes , with @xmath17 denoting evaluation of @xmath18 at node @xmath19 ; * @xmath20 the thresholding function ; * @xmath21 relaxation of @xmath22 using gradient flow in double - well potential @xmath23 ; * @xmath24 the label value at each node with @xmath25 * @xmath26 or @xmath27 , label data ; * @xmath28 with @xmath29 being a relaxation of the label variable @xmath30 ; * @xmath31 weight matrix of the graph , @xmath32 the resulting symmetric graph laplacian ; * @xmath2 the precision matrix and @xmath33 the covariance matrix , both found from @xmath32 ; * @xmath34 eigenpairs of @xmath32 ; * @xmath35 : orthogonal complement of the null space of the graph laplacian @xmath32 , given by @xmath36 ; * @xmath37 : orthogonal complement of the first @xmath38 eigenfunctions of the graph laplacian @xmath32 .
* @xmath39 denotes the euclidean norm and @xmath40 the corresponding inner - product ; * @xmath41 : ginzburg - landau functional ; * @xmath42 , @xmath43 : prior probability measures ; * @xmath44 and @xmath45 : ( with suffices denoting different models ) ; * the measures denoted @xmath44 typically take argument @xmath18 and are real - valued ; the measures denoted @xmath45 take argument @xmath30 on label space , or argument @xmath29 on a real - valued relaxation of label space ; * @xmath46 denotes a gaussian random variable with mean @xmath47 and covariance @xmath48 ; * @xmath49 and @xmath50 denote the probability of an event , and the expectation of a random variable , respectively ; the underlying probability measure will be made explicit as a subscript when it is necessary to do so .
in subsection [ ssec : ssl ] we formulate semi - supervised learning as a problem on a graph . subsection [ ssec :
graph ] defines the relevant properties of the graph laplacian and in subsection [ ssec : gauss ] these properties are used to construct a gaussian probability distribution ; in section [ sec:3 ] this gaussian will be used to define our prior information about the classification problem . in subsection
[ ssec : thr ] we discuss thresholding which provides a link between the real - valued prior information , and the label data provided for the semi - supervised learning task ; in section [ sec:3 ] this will be used to definine our likelihood .
we are given a set of points denoted by @xmath51 , and a set of features @xmath52 associated with these points ; each feature vector @xmath53 is an element of @xmath54 , so that @xmath55 graph learning starts from the construction of an undirected graph @xmath56 with weights @xmath57 computed from the feature set @xmath58 . for graph semi
- supervised learning , we are also given a partial set of ( possibly noisy ) labels @xmath59 , where @xmath60 has size @xmath14 .
the task is to infer the labels for all nodes in @xmath11 , using the weighted graph @xmath56 and also the set of noisily observed labels @xmath61 . in the bayesian formulation which we adopt the feature set @xmath58 , and hence the graph @xmath56 ,
is viewed as prior information , describing correlations amongst the nodes of the graph , and we combine this with a likelihood based on the noisily observed labels @xmath61 , to obtain a posterior distribution on the labelling of all nodes . various bayesian formulations , which differ in the specification of the observation model and/or the prior , are described in section [ sec:3 ] . in the remainder of this section
we give the background needed to understand all of these formulations , thereby touching on the graph laplacian itself , its link to gaussian probability distributions and , via thresholding , to non - gaussian probability distributions and to the ginzburg - landau functional . an important point to appreciate
is that building our priors from gaussians confers considerable computational advantages for large graphs ; for this reason the non - gaussian priors will be built from gaussians via change of measure or push forward under a nonlinear map .
the graph laplacian is central to many graph - learning algorithms .
there are a number of variants used in the literature ; see for a discussion .
we will work with the symmetric laplacian , defined from the weight matrix @xmath62 as follows .
we define the diagonal matrix @xmath63 with entries @xmath64 if we assume that the graph @xmath56 is connected , then @xmath65 for all nodes @xmath66 .
we can then define the symmetric graph laplacian that we use is that it is symmetric positive semi - definite .
we could therefore use other graph laplacians , such as the unnormalized choice @xmath67 , in most of the paper .
the only exception is the spectral approximation sampling algorithm introduced later ; that particular algorithm exploits empirical properties of the symmetrized graph laplacian .
note , though , that the choice of which graph laplacian to use can make a significant difference
see @xcite , and figure 2.1 therein . to make our exposition more concise we confine our presentation to the symmetric graph laplacian . ] as @xmath68 and the graph dirichlet energy as @xmath69 .
then @xmath70 thus , similarly to the classical dirichlet energy , this quadratic form penalizes nodes from having different function values , with penalty being weighted with respect to the similarity weights from @xmath31 .
furthermore the identity shows that @xmath32 is positive semi - definite .
indeed the vector of ones @xmath71 is in the null - space of @xmath72 by construction , and hence @xmath32 has a zero eigenvalue with corresponding eigenvector @xmath73 .
we let @xmath74 denote the eigenpairs of the matrix @xmath32 , ordered so that @xmath75 the upper bound of @xmath76 may be found in ( * ? ? ?
* lemma 1.7 , chapter 1 ) .
the eigenvector corresponding to @xmath77 is @xmath78 and @xmath79 , assuming a fully connected graph .
then @xmath80 where @xmath81 has columns @xmath82 and @xmath83 is a diagonal matrix with entries @xmath84 . using these eigenpairs the graph dirichlet energy can be written as @xmath85 this is analogous to decomposing the classical dirichlet energy using fourier analysis .
we now show how to build a gaussian distribution with negative log density proportional to @xmath86 such a prior prefers functions that have larger components on the first few eigenvectors of the graph laplacian , where the eigenvalues of @xmath32 are smaller .
the corresponding eigenvectors carry rich geometric information about the weighted graph .
for example , the second eigenvector of @xmath32 is the _
fiedler vector _ and solves a relaxed normalized min - cut problem @xcite .
the gaussian distribution thereby connects geometric intuition embedded within the graph laplacian to a natural probabilistic picture .    to make this connection concrete we define diagonal matrix @xmath48 with entries defined by the vector @xmath87 and define the positive semi - definite covariance matrix @xmath88 ; choice of the scaling @xmath89
will be discussed below .
we let @xmath90 note that the covariance matrix is that of a gaussian with variance proportional to @xmath91 in direction @xmath92 thereby leading to structures which are more likely to favour the fiedler vector ( @xmath93 ) , and lower values of @xmath19 in general , than it does higher values .
the fact that the first eigenvalue of @xmath33 is zero ensures that any draw from @xmath42 changes sign , because it will be orthogonal to @xmath94 to make this intuition explicit we recall the karhunen - loeve expansion which constructs a sample @xmath18 from the prior @xmath42 according to the random sum @xmath95 where the @xmath96 are i.i.d .
@xmath97 since each @xmath92 with @xmath98 is orthogonal to @xmath99 it follows that @xmath18 is orthogonal to @xmath99 and the sign - change property is enforced because @xmath99 is of one sign .
we choose the constant of proportionality @xmath89 as a rescaling which enforces the property @xmath100 for @xmath101 ; in words the per - node variance is @xmath102 .
note that , using the orthogonality of the @xmath103 , @xmath104 thus the normalization implies that @xmath105 we reiterate that the support of the measure @xmath42 is the space @xmath106 and that , on this space , the probability density function is proportional to @xmath107 so that the _ precision matrix _ of the gaussian is @xmath108 in what follows the sign of @xmath18 will be related to the classification ; since all the entries of @xmath109 are positive , working on the space @xmath35 ensures a sign change in @xmath18 , and hence a non - trivial classification .      for the models considered in this paper ,
the label space of the problem is discrete while the latent variable @xmath18 through which we will capture the correlations amongst nodes of the graph , encoded in the feature vectors , is real - valued .
we describe thresholding , and a relaxation of thresholding , to address the need to connect these two differing sources of information about the problem . in
what follows the latent variable @xmath110 is thresholded to obtain the label variable @xmath111 the variable @xmath112 is a real - valued relaxation of the label variable @xmath113 the variable @xmath18 will be endowed with a gaussian probability distribution .
from this the variable @xmath30 ( which lives on a discrete space ) and @xmath29 ( which is real - valued , but concentrates near the discrete space supporting @xmath30 ) will be endowed with non - gaussian probability distributions .
define the ( signum ) function @xmath114 by @xmath115 this will be used to connect the latent variable @xmath18 with the label variable @xmath30 .
the function @xmath22 may be relaxed by defining @xmath116 where @xmath29 solves the gradient flow @xmath117 this will be used , indirectly , to connect the latent variable @xmath18 with the real - valued relaxation of the label variable , @xmath29 .
note that @xmath118 , pointwise , as @xmath119 , on @xmath120 this reflects the fact that the gradient flow minimizes @xmath23 , asymptotically as @xmath121 , whenever started on @xmath120    we have introduced a gaussian measure @xmath42 on the latent variable @xmath18 which lies in @xmath122 ; we now want to introduce two ways of constructing non - gaussian measures on the label space @xmath123 or on real - valued relaxations of label space , building on the measure @xmath124 the first is to consider the push - forward of measure @xmath42 under the map @xmath22 : @xmath125 then @xmath126 thus @xmath127 is a measure on the label space @xmath128 the second approach is to work with a change of measure from the gaussian @xmath42 in such a way that the probability mass on @xmath122 concentrates close to the label space @xmath129 .
we may achieve this by defining the measure @xmath43 via its radon - nykodim derivative @xmath130 we name @xmath43 the ginzburg - landau measure , since the negative log density function of @xmath43 is the graph ginzburg - landau functional @xmath131 the ginzburg - landau distribution defined by @xmath43 can be interpreted as a non - convex ground relaxation of the discrete mrf model @xcite , in contrast to the convex relaxation which is the gaussian field @xcite . since the double well has minima at the label values @xmath132 , the probability mass of @xmath43 is concentrated near the modes @xmath133 , and @xmath134 controls this concentration effect .
[ sec : problem - spec ] in this section we formulate four different bayesian models for the semi - supervised learning problem .
the four models all combine the ideas described in the previous section to define four distinct posterior distributions .
it is important to realize that these different models will give different answers to the same questions about uncertainty quantification , just as different methods based around optimization will give different classifications .
the choice of which bayesian model to use is related to the data itself , and making this choice is beyond the scope of this paper .
currently the choice must be addressed on a case by case basis , as is done when choosing an optimization method for classification .
nonetheless we will demonstrate that the shared structure of the four models mean that a common algorithmic framework can be adopted and we will make some conclusions about the relative costs of applying this framework to the four models .    we denote the latent variable by @xmath17 , @xmath135 , the thresholded value of @xmath17 by @xmath136 which is interpreted as the label assignment at each node @xmath19 , and noisy observations of the binary labels by @xmath137 , @xmath138 .
the variable @xmath139 will be used to denote the real - valued relaxation of @xmath140 used for the ginzburg - landau model .
recall bayes formula which transforms a prior density @xmath141 on a random variable @xmath18 into a posterior density @xmath142 on the conditional random variable @xmath143 : @xmath144 we will now apply this formula to condition our graph latent variable @xmath18 , whose thresholded values correspond to labels , on the noisy label data @xmath61 given at @xmath13 . as prior on @xmath18 we will always use @xmath145 we willdescribe three different likelihoods .
we will also apply the formula to condition relaxed label variable @xmath29 , on the same label data @xmath61 , via the formula @xmath146 we will use as prior the non - gaussian @xmath147    for the probit , level - set and atomic models , we now explicitly state the prior density @xmath141 , the likelihood function @xmath148 , and the posterior density @xmath142 ; in the ginzburg - landau case @xmath29 will replace @xmath18 and we will define the densities @xmath149 and @xmath150 . prior and posterior probability measures associated with letter @xmath44 are on the latent variable @xmath18 ; measures associated with letter @xmath45 are on the label space , or real - valued relaxation of the label space .      the probit method is designed for classification and is described in @xcite ; in that context gaussian process priors are used and these do not depend on label data .
a recent fully bayesian treatment of the methodology using unweighted graph laplacians may be found in the paper @xcite . in detail
our model is as follows .
* prior * we take as prior on @xmath18 the gaussian @xmath124 thus @xmath151    * likelihood * for any @xmath138 @xmath152 with the @xmath153 drawn i.i.d from @xmath154 we let @xmath155 and note that then @xmath156 similarly @xmath157    * posterior * bayes theorem gives posterior @xmath158 with probability density function ( pdf ) @xmath159 where @xmath160 we let @xmath161 denote the push - forward under @xmath22 of @xmath162    * map estimator * this is the minimizer of the negative of the log posterior . thus we minimize the following objective function over @xmath35 : @xmath163 this is a convex function , a fact which is well - known in related contexts , but which we state and prove in the supplementary materials for the sake of completeness . in view of the close relationship between this problem and
the level - set formulation described next , for which there are no minimizers , we expect that minimization may not be entirely straightforward in the @xmath164 limit .
this is manifested in the presence of near - flat regions in the probit log likelihood function when @xmath164 .
our variant on the probit methodology differs from that in @xcite in several ways : ( i ) our prior gaussian is scaled to have per - node variance one , whilst in @xcite the per node variance is a hyper - parameter to be determined ; ( ii ) our prior is supported on @xmath165 whilst in @xcite the prior precision is found by shifting @xmath32 and taking a possibly fractional power of the resulting matrix , resulting in support on the whole of @xmath166 ; ( iii ) we allow for a scale parameter @xmath167 in the observational noise , whilst in @xcite the parameter @xmath168      this method is designed for problems considerably more general than classification on a graph @xcite . for the current application , this model is exactly the same as probit except for the order in which the noise @xmath153 and the thresholding function @xmath169 is applied in the definition of the data .
* prior * we again take as prior for @xmath18 , the gaussian @xmath124 thus @xmath151    * likelihood * for any @xmath138 @xmath170 with the @xmath153 drawn i.i.d from @xmath154 then @xmath171 with pdf @xmath172 where latexmath:[\[{\phi_{\rm ls}}(u;y)=\sum_{j \in z ' } \bigl(\frac{1}{2\gamma^2 }    denote the pushforward under @xmath22 of @xmath174    * map estimator functional * the negative of the log posterior is , in this case , given by @xmath175 however , unlike the probit model , the bayesian level - set method has no map estimator  the infimum of @xmath176 is not attained and this may be seen by noting that , if the infumum was attained at any non - zero point @xmath177 then @xmath178 would reduce the objective function for any @xmath179 ; however the point @xmath180 does not attain the infimum .
this proof is detailed in @xcite for a closely related pde based model , and the proof is easily adapted .
this is a variant on the level - set method , but deals with the fact that categorical data , whilst maybe noisy , will often be discrete .
the resulting data model could also be used within the ginzburg - landau formulation in the next subsection , but we do not describe this explicitly . the prior we employ is the gaussian @xmath42 , and in contrast to the level - set method , but like the probit method , the observation is assumed to take values in the set @xmath181 .
in the atomic noise model the reporting of those values is accompanied by specified error rates @xcite , determined by the parameters @xmath182 .
taking @xmath183 in the following corresponds to exact data with no error and is the same as level - set thresholding , or probit , in the limit @xmath184    * prior * we again take as prior for @xmath18 the gaussian @xmath124 thus @xmath151    * likelihood * we assume that the observation has the sign of @xmath18 with a specified probability . to be precise
we assume that we have , @xmath185 and @xmath186 this defines a piecewise constant function of @xmath18 , with discontinuity at @xmath187 , for each value @xmath188 : we call this function @xmath189 in particular we have @xmath190    * posterior * bayes theorem gives posterior @xmath191 with pdf @xmath192 where @xmath193
we let @xmath194 denote the pushforward under @xmath22 of @xmath195    * map estimator functional * this is the minimizer of the negative of the log posterior .
thus we minimize the objective function : @xmath196 we will not consider numerical methods for the atomic noise model because of space limitations .
however it may be a natural choice for some measurement scenarios , hence its inclusion in this paper .
its behaviour is very similar to the bayesian level set method because its likelihood is also piecewise constant with respect to latent variable @xmath18 ; and as with the bayesian level set there is no minimizer for the map estimation problem .      for this model
, we take as prior the ginzburg - landau measure @xmath43 defined by ( [ eq : gl - meas ] ) , and employ a gaussian likelihood for the observed labels .
this construction gives the bayesian posterior whose map estimator is the objective function introduced and studied in @xcite .
* prior * we define prior on @xmath29 to be the ginzburg - landau measure @xmath43 given by ( [ eq : gl - meas ] ) with density @xmath197    * likelihood * for any @xmath138 @xmath198 with the @xmath153 drawn i.i.d from @xmath154 then @xmath199 we see that bayes theorem gives posterior @xmath200 with pdf latexmath:[\[\begin{aligned } { \mathbb{p}_{\rm gl}}(v|y ) & \propto & \exp\bigl(-\frac12 \langle v , pv \rangle - { \phi_{\rm gl}}(v;y)\bigr),\\ { \phi_{\rm gl}}(v;y)&:=&\sum_{j \in z } w_\epsilon\bigl(v(j)\bigr)+\sum_{j \in z ' } \bigl(\frac{1}{2\gamma^2 }     * map estimator * this is the minimizer of the negative of the log posterior .
thus we minimize the following objective function over @xmath35 : @xmath202 this objective function was introduced in @xcite as a relaxation of the min - cut problem , penalized by data ; the relationship to min - cut was studied rigorously in @xcite .
the minimization problem for @xmath203 is non - convex and has multiple minimizers , reflecting the combinatorial character of the min - cut problem of which it is a relaxation .
in the small label noise limit @xmath204 the probit and level - set posteriors coincide with the atomic noise model in the limit where @xmath183 .
furthermore all models then take the form of the gaussian prior @xmath42 conditioned to be positive on labelled nodes where @xmath205 and to be negative on labelled nodes where @xmath206 this can be linked with the original work of zhu et al @xcite which based classification on the measure @xmath42 conditioned to take the value exactly @xmath102 on labelled nodes where @xmath205 and conditioned to take the value exactly @xmath207 on labelled nodes where @xmath206 thus we see explicit connections between a variety of different bayesian formulations of graph - based semi - supervised learning .      in figure
[ fig : spectral - approx ] we plot the component of the negative log likelihood at a labelled node @xmath19 , as a function of the latent variable @xmath208 with data @xmath209 fixed , for the probit , bayesian level - set , and atomic noise models .
the log likelihood for the ginzburg - landau formulation is not directly comparable as it is a function of the relaxed label variable @xmath139 , with respect to which it is quadratic with minimum at the data point @xmath210    .
we set @xmath211 for probit and bayesian level - set , and @xmath212 , @xmath213 for atomic noise model .
since @xmath214 for probit and bayesian level - set , we omit the plot for @xmath215.,width=264 ]    the probit , bayesian level - set , and atomic noise models lead to posterior distributions @xmath44 ( with different subscripts ) in latent variable space , and pushforwards @xmath45 ( also with different subscripts ) in label space .
the ginzburg - landau formulation leads to a measure @xmath200 in label space .
uncertainty quantification in the widest sense is concerned with completely characterizing these posterior distributions . in practice
this may be acheived by sampling using mcmc methods . in this paper
we will study four measures of uncertainty :    * we will study the empirical pdfs of the latent and label variables at certain nodes ; * we will study the posterior mean of the label variables at certain nodes ; * we will study the posterior variance of the label variables averaged over all nodes ; * we will use the posterior mean or variance to order nodes into those whose classificaions are most uncertain and those which are most certain .    for the probit , level - set and atomic models
, we interpret the thresholded variable @xmath216 as the binary label assignments corresponding to a real - valued configuration @xmath18 .
the node - wise posterior mean of @xmath30 can be used as a useful confidence score of the class assignment of each node .
the node - wise posterior mean @xmath217 is defined as @xmath218 with respect to any of the posterior measures ( pushed forward from latent variable space for probit , level - set and atomic models ) @xmath45 .
note that @xmath219 $ ] and if @xmath220 then @xmath221 . for binary labels
@xmath222 the mean also contains the variance information , and hence the formula captures posterior variance .
specifically we have that @xmath223 later we will find it useful to consider the variance averaged over all nodes and hence define ] @xmath224 note that the maximum value obtained by var(l ) is @xmath102 .
this maximum value is attained under all the prior distributions we use in this paper .
the deviation from this maximum , under the posterior , is a measure of the information content of the labelled data .
note , however , that the prior does contain information about classifications , in the form of correlations between vertices ; this is not captured in .
from section [ sec:3 ] , we see that for all of the models considered , the posterior @xmath225 has the form @xmath226 for some function @xmath3 , different for each of the four models ( acknowledging that in the ginzburg - landau case the independent variable is @xmath227 , real - valued relaxation of label space , where as for the other models @xmath228 an underlying latent variable which may be thresholded by @xmath229 into label space . )
furthermore , the map estimator is the minimizer of @xmath230 note that @xmath3 is differentiable for the ginzburg - landau and probit models , but not for the level - set and atomic noise models .
we introduce algorithms for both sampling ( mcmc ) and map estimation ( optimization ) that apply in this general framework .
the sampler we employ does not use information about the gradient of @xmath3 ; the map estimation algorithm does , but is only employed on the ginzburg - landau and probit models .
the samplers do use properties of the precision matrix @xmath2 , which is proportional to the graph laplcian @xmath32 ; in particular its spectral properties are relevant .
figure [ fig : spectra - all ] demonstrates the spectral properties of @xmath32 for the four examples that we will apply our algorithms to in section [ sec:5 ] .
we sample the posterior probability distribution using mcmc .
to date probit models have typically been sampled by means of a gibbs methodology .    however for three reasons we consider sampling algorithms which apply directly on all the nodes @xmath11 .
these are : ( i ) we wish to highlight methods which apply to the ginzburg - landau , level - set and atomic noise models which precludes the explicit conditionally gaussian , or truncated gaussian , form of the methods described for probit ; ( ii ) all of our posterior distributions have a density with respect to the gaussian @xmath42  that is their densities are proportional to that of @xmath42  and as a result we may use mcmc methods which , in the case where the graph laplacian has a limit @xcite , have the potential for delivering samples from the posterior in a number of steps which is independent of the dimension @xmath12 of the state space , as overviewed in @xcite ; ( iii ) these mcmc methods are well - adapted to the use of approximation methods which exploit structure in the spectral properties of the graph laplacian .
other classes of mcmc methods , such as the gibbs samplers in @xcite ; could be considered ; and other priors , relaxing the gaussian structure , could be considered ; but in taking these directions then the development of mcmc methods with @xmath231independent mixing rates which can also exploit approximations of the spectral properties of the graph laplacian is an open research direction .    in order to induce scalability with respect to size of @xmath11 we use the pcn method described in @xcite and introduced in the context of diffusions by beskos et .
al . in @xcite and by neal in the context of machine learning @xcite .
the standard random walk metropolis ( rwm ) algorithm suffers from the fact that the optimal proposal variance or stepsize scales inverse proportionally to the dimension of the state space @xcite , which is the graph size @xmath12 in this case .
the pcn method is designed so that the proposal variance required to obtain a given acceptance probability scales independently of the dimension of the state space ( here the number of graph nodes @xmath12 ) , hence in practice giving faster convergence of the mcmc when compared with rwm @xcite .
we restate the pcn method as algorithm [ alg : pcnfull ] , and then follow with various variants on it in algorithms [ alg : pcneig ] and [ alg : pcneigfill ] . in
all three algorithms @xmath232 $ ] is the key parameter which determines the efficiency of the mcmc method : small @xmath233 leads to high acceptance probability but small moves ; large @xmath233 leads to low acceptance probability and large moves . somewhere between these extremes
is an optimal choice of @xmath233 which minimizes the asymptotic variance of the algorithm when applied to compute a given expectation .
the value @xmath234 is a sample from the prior @xmath42 .
if the eigenvalues and eigenvectors of @xmath32 are all known then the karhunen - loeve expansion gives @xmath235 where @xmath89 is given by , the @xmath236 are i.i.d centred unit gaussians and the equality is in law .      for graphs with a large number of nodes @xmath12 , it is prohibitively costly to directly sample from the distribution @xmath42 , since doing so involves knowledge of a complete eigen - decomposition of @xmath32 , in order to employ .
a method that is frequently used in classification tasks is to restrict the support of @xmath18 to the eigenspace spanned by the first @xmath38 eigenvectors with the smallest non - zero eigenvalues of @xmath32 ( hence largest precision ) and this idea may be used to approximate the pcn method ; this leads to a low rank approximation .
in particular we approximate samples from @xmath42 by @xmath237 where @xmath238 is given by truncated after @xmath239 , the @xmath240 are i.i.d centred unit gaussians and the equality is in law .
this is a sample from @xmath241 where @xmath242 and the diagonal entries of @xmath243 are set to zero for the entries after @xmath244 in practice , to implement this algorithm , it is only necessary to compute the first @xmath38 eigenvectors of the graph laplacian @xmath32 .
this gives algorithm [ alg : pcneig ] .
@xmath245    the accuracy of algorithm [ alg : pcneig ] as an approximation of algorithm [ alg : pcnfull ] depends to a large extent on the size of the eigenvalues of @xmath32 in the following sense : @xmath246 where the expectation is with respect to the centred unit gaussians @xmath247 note that the examples shown in figure [ fig : spectra - all ] gives an indiciation of the quality of this approximation ; the size and number of the smallest eigenvalues of @xmath32 play a very important role in determining whether the difference in is small .
spectral projection often leads to good classification results , but may lead to reduced posterior variance and a posterior distribution that is overly smooth on the graph domain .
we propose an improvement on the method that preserves the variability of the posterior distribution but still only involves calculating the first @xmath38 eigenvectors of @xmath248 this is based on the empirical observation that in many applications the spectrum of @xmath32 saturates and satisfies , for @xmath249 , @xmath250 for some @xmath251 .
such behaviour may be observed in b ) , c ) and d ) of figure [ fig : spectra - all ] ; in particular note that in the hyperspectal case @xmath252 .
we assume such behaviour in deriving the low rank approximation used in this subsection .
( see supplementary materials for a detailed discussion of the graph laplacian spectrum . )
we define @xmath253 by overwriting the diagonal entries from @xmath38 to @xmath254 with @xmath255 we then set @xmath256 , and generate approximate samples from @xmath42 by setting @xmath257 where @xmath258 is given by with @xmath259 replaced by @xmath251 for @xmath260 , the @xmath96 are centred unit gaussians , and the equality is in law . importantly samples according to can be computed very efficiently . in particular
there is no need to compute @xmath92 for @xmath260 , and the quantity @xmath261 can be computed by first taking a sample @xmath262 , and then projecting @xmath263 onto @xmath264 . moreover ,
projection onto @xmath37 can be computed only using @xmath265 , since the vectors span the orthogonal complement of @xmath37 .
concretely , we have @xmath266 where @xmath262 and equality is in law .
hence the samples @xmath267 can be computed by @xmath268 the error induced by this approximation can be characterized through the formula @xmath269 under the stated empirical properties of the graph laplacian , we expect this to be a better approximation of the prior variance than the approximation leading to . the vector @xmath267 is a sample from @xmath270 and results in algorithm [ alg : pcneigfill ] .
@xmath245      recall that the objective function for the map estimation has the form @xmath271 , where @xmath18 is supported on the space @xmath35 . for ginzburg - landau and probit ,
the function @xmath3 is smooth , and we can use a standard projected gradient method for the optimization .
since @xmath32 is typically ill - conditioned , it is preferable to use a semi - implicit discretization as suggested in @xcite , as convergence to a stationary point can be shown under a graph independent learning rate .
furthermore , the discretization can be performed in terms of the eigenbasis @xmath272 , which allows us to easily apply spectral projection when only a truncated set of eigenvectors is available .
we state the algorithm in terms of the ( possibly truncated ) eigenbasis below . here
@xmath273 is an approximation to @xmath2 found by setting @xmath274 where @xmath275 is the matrix with columns @xmath276 and @xmath277 for @xmath278 , @xmath279 thus @xmath280    , @xmath281 .
in this section we conduct a series of numerical experiments on four different data sets that are representative of the field of graph semi - supervised learning . there are four main purposes for the experiments .
first we perform uncertainty quantification , as explained in subsection [ ssec : uq ] .
secondly , we study the spectral approximation and projection variants on pcn sampling as these scale well to massive graphs .
finally we make some observations about the cost and practical implementation details of these methods , for the different bayesian models we adopt ; these will help guide the reader in making choices about which algorithm to use .
we present the results for map estimations in the supplementary materials , alongside the proof of convexity of the probit map estimator .
the quality of the graph constructed from the feature vectors is central to the performance of any graph learning algorithms . in the experiments below ,
we follow the graph construction procedures used in the previous papers @xcite ; those papers , together , applied graph partitioning to all of the datasets that we use here and so provide important guidance .
moreover , we have verified that for all the reported experiments below , the graph parameters are in a range such that spectral clustering gives a reasonable performance .
the methods we employ lead to refinements over spectral clustering ( improved classification ) and , of course , to uncertainty quantification ( which spectral clustering does not address ) .
we introduce the data sets and describe the graph construction for each data set . in all cases
we numerically construct the weight matrix @xmath31 , and then the graph laplacian @xmath32 . is symmetric in theory ; in practice we find that symmetrizing via the map @xmath282 is helpful . ]      the two moons artificial data set is constructed to give noisy data which lies near a nonlinear low dimensional manifold embedded in a high dimensional space @xcite .
the data set is constructed by sampling @xmath12 data points uniformly from two semi - circles centered at @xmath283 and @xmath284 with radius @xmath102 , embedding the data in @xmath285 , and adding gaussian noise with standard deviation @xmath286 we set @xmath287 and @xmath288 in this paper ; recall that then the graph size is @xmath12 and each feature vector has length @xmath289 .
we will conduct a variety of experiments with different labelled data size @xmath290 , and in particular study variation with @xmath290 .
the default value , when not varied , is @xmath290 at @xmath291 of @xmath12 , with the labelled points chosen at random .
we take each data point as a node on the graph , and construct a fully connected graph using the self - tuning weights of zelnik - manor and perona @xcite , with @xmath292 = 10 .
specifically we let @xmath293 , @xmath53 be the coordinates of the data points @xmath294 and @xmath19 .
then weight @xmath295 from @xmath294 to @xmath19 is defined by @xmath296 where @xmath297 is the distance of the @xmath292-th closest point to the node @xmath19 .
this dataset contains the voting records of 435 u.s .
house of representatives ; for details see @xcite and the references therein .
the votes were recorded in 1984 from the @xmath298 united states congress , @xmath299 session .
the votes for each individual is vectorized by mapping a yes vote to @xmath102 , a no vote to @xmath207 , and an abstention / no - show to @xmath300 .
the data set contains @xmath301 votes that are believed to be well - correlated with partisanship , and we use only these votes as feature vectors for constructing the graph . thus the graph size is @xmath302 , and feature vectors have length @xmath303 .
the goal is to predict the party affiliation of each individual .
we pick @xmath304 democrats and @xmath76 republicans at random to use as the observed class labels ; thus @xmath305 corresponding to less than @xmath306 of fidelity points .
we construct a fully connected graph with weights given by with @xmath307 for all nodes @xmath308      the mnist database consists of @xmath309 images of size @xmath310 pixels containing the handwritten digits @xmath300 through @xmath311 ; see @xcite for details . since in this paper we focus on binary classification , we only consider pairs of digits .
to speed up calculations , we subsample randomly @xmath312 images from each digit to form a graph with @xmath313 nodes ; we use this for all our experiments except in subsection [ ssec : cm ] where we use the full data set of size @xmath314 for digit pair @xmath315 to benchmark computational cost .
the nodes of the graph are the images and as feature vectors we project the images onto the leading 50 principal components given by pca ; thus the feature vectors at each node have length @xmath316 we conduct a variety of experiments with different labelled data dimension @xmath290 , and in particular study variation with @xmath290 .
the default value , when not varied , is @xmath290 at @xmath317 of @xmath12 , with the labelled points chosen at random .
we construct a @xmath292-nearest neighbor graph with @xmath318 for each pair of digits considered .
namely , the weights @xmath319 are non - zero if and only if one of @xmath294 or @xmath19 is in the @xmath292 nearest neighbors of the other .
the non - zero weights are set using with @xmath320 .
this is the only example we consider in this paper that does not have a fully connected graph .
we choose the four pairs @xmath321 , @xmath322 , @xmath323 and @xmath315 .
these four pairs exhibit increasing levels of difficulty for classification .
this fact is demonstrated in figures [ fig : mnist1 ] - [ fig : mnist4 ] , where we visualize the datasets by projecting the dataset onto the second and third eigenvector of the graph laplacian .
namely , each node @xmath294 is mapped to the point @xmath324 , where @xmath325 .
the hyperspectral data set analysed for this project was provided by the applied physics laboratory at johns hopkins university ; see @xcite for details .
it consists of a series of video sequences recording the release of chemical plumes taken at the dugway proving ground .
each layer in the spectral dimension depicts a particular frequency starting at @xmath326 nm and ending with @xmath327 nm , with a channel spacing of @xmath328 nm , giving @xmath329 channels ; thus the feature vector has length @xmath330 the spatial dimension of each frame is @xmath331 pixels .
we select @xmath332 frames from the video sequence as the input data , and consider each spatial pixel as a node on the graph .
thus the graph size is @xmath333 .
the classification problem is to classify pixels that represent the chemical plumes against pixels that are the background .
we construct a fully connected graph with weights given by the cosine distance : @xmath334 this distance is small for vectors that point in the same direction , and is insensitive to their magnitude .
we consider the symmetric laplacian defined in . because it is computationally prohibitive to compute eigenvectors of a laplacian of this size
, we apply the nystrm extension @xcite to obtain an approximation to the true eigenvectors and eigenvalues ; see @xcite for details pertinent to the set - up here .
we emphasize that each pixel in the @xmath332 frames is a node on the graph and that , in particular , pixels across the @xmath332 time - frames are also connected .
since we have no ground truth labels for this dataset , we generate known labels by setting the segmentation results from spectral clustering as ground truth .
the default value of @xmath290 is @xmath335 , and labels are chosen at random .
this corresponds to labelling around @xmath336 of the points .
we only plot results for the last @xmath337 frames of the video sequence since the first frame does not contain the chemical plume .      in this subsection
we demonstrate both the feasibility , and value , of uncertainty quantification in graph classification methods .
we employ the probit and the bayesian level - set model for most of the experiments in this subsection ; we also employ the ginzburg - landau model but since this can be slow to converge , due to the presence of local minima , it is only demonstrated on the voting records dataset .
the atomic noise model has a similar piecewise constant log likelihood as for the bayesian level - set method , and so we omit experiments on the atomic noise model .
the pcn method is used for sampling on various datasets to demonstrate properties and interpretations of the posterior .      in this subsection ,
we contrast the posterior distribution @xmath150 of the ginzburg - landau model with that of the probit and bayesian level - set ( bls ) models .
the graph is constructed from the voting records data with the fidelity points chosen as described in subsection [ subsec : datasets ] . in figure
[ fig : glvsprobitmcmc ] we plot the histograms of the empirical marginal posterior distribution on @xmath338 and @xmath339 for a selection of nodes on the graph . for the top row of figure [ fig : glvsprobitmcmc ] ,
we select @xmath337 nodes with  low confidence " predictions , and plot the empirical marginal distribution of @xmath18 for probit and bls , and that of @xmath29 for the ginzburg - landau model .
note that the same set of nodes is chosen for different models .
the plots in this row demonstrate the multi - modal nature of the ginzburg - landau distribution in contrast to the uni - modal nature of the probit posterior ; this uni - modality is a consequence of proposition [ prop : convex - probit ] . for the bottom row , we plot the same empirical distributions for 6 nodes with  high confidence " predictions .
in contrast with the top row , the ginzburg - landau marginal for high confidence nodes is essentially uni - modal since most samples of @xmath29 evaluated on these nodes have a fixed sign .
we also observe that the pcn algorithm converges far more quickly for probit than for ginzburg - landau , because of the presence of multiple modes in the latter ; this is manifest in the fact that the posteriors for ginzburg - landau are less well converged than for probit , for a similar amount of algorithmic time ; this issue is quantified in subsection [ ssec : cm ] .
this undesirable feature of ginzburg - landau sampling can be ameliorated by choosing a larger value of @xmath134 ; however this then leads to a probability measure which is further from the label space which it relaxes .
the bayesian level - set method behaves similarly to probit as is to be expected from the fact that , in the limit @xmath340 , they are formally identical as discussed in subsection [ ssec : sn ] .
we construct the graph from the mnist @xmath315 dataset following subsection [ subsec : datasets ] .
the noise variance @xmath167 is set to @xmath341 , and @xmath317 of fidelity points are chosen randomly from each class . the probit posterior is used to compute . in figure
[ fig : mnistlowhigh ] we demonstrate that nodes with scores @xmath342 closer to the binary ground truth labels @xmath133 look visually more uniform than nodes with @xmath342 far from those labels .
this shows that the posterior mean contains useful information which differentiates between outliers and inliers that align with human perception .      in this set of experiments
, we show that the posterior distribution of the label variable @xmath216 captures the uncertainty of the classification problem .
we use the posterior variance of @xmath30 , averaged over all nodes , as a measure of the model variance ; specifically formula .
we study the behaviour of this quantity as we vary the level of uncertainty within certain inputs to the problem .
we demonstrate empirically that the posterior variance is approximately monotonic with respect to variations in the levels of uncertainty in the input data , as it should be ; and thus that the posterior variance contains useful information about the classification .
we select quantities that reflect the separability of the classes in the feature space .
figure [ fig : twomoonsfeature ] plots the posterior variance @xmath343 against the standard deviation @xmath344 of the noise appearing in the feature vectors for the two moons dataset ; thus points generated on the two semi - circles overlap more as @xmath344 increases .
we employ a sequence of posterior computations , using probit and bayesian level - set , for @xmath345 .
recall that @xmath346 and we choose @xmath291 of the nodes to have the ground truth labels as observed data . within both models ,
@xmath167 is fixed at @xmath341 .
a total of @xmath347 samples are taken , and the proposal variance @xmath233 is set to @xmath348 .
we see that the mean posterior variance increases with @xmath344 , as is intuitively reasonable . furthermore , because @xmath167 is small , probit and bayesian level - set are very similar models and this is reflected in the similar quantitative values for uncertainty .     for the probit model and
the bls model applied to the two moons dataset with @xmath287 . for each trial ,
a realization of the two moons dataset under the given parameter @xmath344 is generated , and @xmath291 of nodes are randomly chosen as fidelity .
we run @xmath349 trials for each value of @xmath344 , and average the mean posterior variance across the @xmath349 trials in the figure .
we set @xmath350 and @xmath351 for both models .
, width=283 ]    a similar experiment studies the posterior label variance @xmath343 as a function of the pair of digits classified within the mnist data set .
we choose @xmath317 of the nodes as labelled data , and set @xmath350 .
the number of samples employed is @xmath347 and the proposal variance @xmath233 is set to be @xmath348 .
table [ tab : mnistpostvar ] shows the posterior label variance .
recall that figures [ fig : mnist1 ] - [ fig : mnist4 ] suggest that the pairs @xmath352 are increasingly easy to separate , and this is reflected in the decrease of the posterior label variance shown in table [ tab : mnistpostvar ] .
.mean posterior variance of different digit pairs for the probit model and the bls model applied to the mnist dataset .
the pairs are organized from left to right according to the separability of the two classes as shown in fig.[fig : mnist1 ] - [ fig : mnist4 ] . for each trial
, we randomly select @xmath317 of nodes as fidelity .
we run @xmath353 trials for each pairs of digits and average the mean posterior variance across trials .
we set @xmath350 and @xmath351 for both models . [ cols="<,<,<,<,<",options="header " , ]        trials are run .
we use spectral projection with number of eigenvectors @xmath354 .
we plot the median accuracy along with error bars indicating the 25 and 75-th quantile of the classification accuracy of each method .
we set @xmath350 for the probit model , and @xmath355 , @xmath356 for ginzburg - landau.,width=283 ]    the non - convexity of the ginzburg - landau model can result in large variance in classification accuracy ; the extent of this depends on the percentage of observed labels .
the existence of sub - optimal local extrema causes the large variance .
if initialized without information about the classification , ginzburg - landau can perform very badly in comparison with probit . on the other hand we find that the best performance of the ginzburg - landau model , when initialized at the probit minimizer , is typically slightly better than the probit model .
we note that the probit model is convex and theoretically should have results independent of the initialization .
however , we see there are still small variations in the classification result from different initializations .
this is due to slow convergence of gradient methods caused by the flat - bottomed well of the probit log - likelihood .
as mentioned above this can be understood by noting that , for small gamma , probit and level - set are closely related and that the level - set map estimator does not exist  minimizing sequences converge to zero , but the infimum is not attained at zero .