access to positive , negative and unlabeled examples is a standard assumption for most semi - supervised binary classification techniques . in many domains ,
however , a sample from one of the classes ( say , negatives ) may not be available , leading to the setting of learning from positive and unlabeled data @xcite .
positive - unlabeled learning often emerges in sciences and commerce where an observation of a positive example ( say , that a protein catalyzes reactions or that a social network user likes a particular product ) is usually reliable . here ,
however , the absence of a positive observation can not be interpreted as a negative example . in molecular biology , for example , an attempt to label a data point as positive ( say , that a protein is an enzyme ) may be unsuccessful for a variety of experimental and biological reasons , whereas in social networks an explicit dislike of a product may not be possible .
both scenarios lead to a situation where negative examples can not be actively collected .
fortunately , the absence of negatively labeled examples can be tackled by incorporating unlabeled examples as negatives , leading to the development of non - traditional classifiers .
here we follow the terminology by @xcite that a traditional classifier predicts whether an example is positive or negative , whereas a non - traditional classifier predicts whether the example is positive or unlabeled .
positive vs. unlabeled ( non - traditional ) training is reasonable because the class posterior  and also the optimum scoring function for composite losses @xcite  in the traditional setting is monotonically related to the posterior in the non - traditional setting .
however , the true posterior can be fully recovered from the non - traditional posterior only if we know the class prior ; i.e. , the proportion of positives in unlabeled data . the knowledge of the class prior is also necessary for estimation of the performance criteria such as the error rate , balanced error rate or f - measure , and also for finding the right threshold for the non - traditional scoring function that leads to an optimal classifier with respect to some criteria @xcite .    class prior estimation in a nonparametric setting has been actively researched in the past decade offering an extensive theory of identifiability @xcite and a few practical solutions @xcite .
application of these algorithms to real data , however , is limited in that none of the proposed algorithms simultaneously deals with noise in the labels and practical estimation for high - dimensional data .
much of the theory on learning class priors relies on the assumption that either the distribution of positives is known or that the positive sample is clean . in practice , however , labeled data sets contain class - label noise , where an unspecified amount of negative examples contaminates the positive sample .
this is a realistic scenario in experimental sciences where technological advances enabled generation of high - throughput data at a cost of occasional errors .
one example for this comes from the studies of proteins using analytical chemistry technology ; i.e. , mass spectrometry .
for example , in the process of peptide identification @xcite , bioinformatics methods are usually set to report results with specified false discovery rate thresholds ( e.g. , 1% ) .
unfortunately , statistical assumptions in these experiments are sometimes violated thereby leading to substantial noise in reported results , as in the case of identifying protein post - translational modifications .
similar amounts of noise might appear in social networks such as facebook , where some users select ` like ' , even when they do not actually like a particular post .
further , the only approach that does consider similar such noise @xcite requires density estimation , which is known to be problematic for high - dimensional data .    in this work ,
we propose the first classification algorithm , with class prior estimation , designed particularly for high - dimensional data with noise in the labeling of positive data .
we first formalize the problem of class prior estimation from noisy positive and unlabeled data .
we extend the existing identifiability theory for class prior estimation from positive - unlabeled data to this noise setting .
we then show that we can practically estimate class priors and the posterior distributions by first transforming the input space to a univariate space , where density estimation is reliable .
we prove that these transformations preserve class priors and show that they correspond to training a non - traditional classifier .
we derive a parametric algorithm and a nonparametric algorithm to learn the class priors .
finally , we carry out experiments on synthetic and real - life data and provide evidence that the new approaches are sound and effective .
consider a binary classification problem of mapping an input space @xmath0 to an output space @xmath1 .
let @xmath2 be the true distribution of inputs .
it can be represented as the following mixture @xmath3 where @xmath4 , @xmath5 , @xmath6 are distributions over @xmath0 for the positive ( @xmath7 ) and negative ( @xmath8 ) class , respectively ; and @xmath9 is the class prior or the proportion of the positive examples in @xmath2 .
we will refer to a sample from @xmath2 as unlabeled data .
let @xmath10 be the distribution of inputs for the labeled data . because the labeled sample contains some mislabeled examples , the corresponding distribution is also a mixture of @xmath11 and a small proportion , say @xmath12 , of @xmath13 .
that is , @xmath14 where @xmath15 $ ] .
observe that both mixtures have the same components but different mixing proportions .
the simplest scenario is that the mixing components @xmath13 and @xmath11 correspond to the class - conditional distributions @xmath16 and @xmath17 , respectively .
however , our approach also permits transformations of the input space @xmath0 , thus resulting in a more general setup .
the objective of this work is to study the estimation of the class prior @xmath18 and propose practical algorithms for estimating @xmath19 .
the efficacy of this estimation is clearly tied to @xmath20 , where as @xmath20 gets smaller , the noise in the positive labels becomes larger .
we will discuss identifiability of @xmath19 and @xmath20 and give a practical algorithm for estimating @xmath19 ( and @xmath20 ) .
we will then use these results to estimate the posterior distribution of the class variable , @xmath21 , despite the fact that the labeled set does not contain any negative examples .
the class prior is identifiable if there is a unique class prior for a given pair @xmath22 .
much of the identifiability characterization in this section has already been considered as the case of asymmetric noise @xcite ; see section [ sec_related ] on related work .
we recreate these results here , with the aim to introduce required notation , to highlight several important results for later algorithm development and to include a few missing results needed for our approach
. though the proof techniques are themselves quite different and could be of interest , we include them in the appendix due to space .
there are typically two aspects to address with identifiability .
first , one needs to determine if a problem is identifiable , and , second , if it is not , propose a canonical form that is identifiable . in this section
we will see that class prior is not identifiable in general because @xmath23 can be a mixture containing @xmath24 and vice versa . to ensure identifiability ,
it is necessary to choose a canonical form that prefers a class prior that makes the two components as different as possible ; this canonical form was introduced as the mutual irreducibility condition @xcite and is related to the proper novelty distribution @xcite and the max - canonical form @xcite .
we discuss identifiability in terms of measures .
let @xmath25 , @xmath26 , @xmath27 and @xmath28 be probability measures defined on some @xmath29-algebra @xmath30 on @xmath0 , corresponding to @xmath2 , @xmath10 , @xmath13 and @xmath11 , respectively .
it follows that @xmath31 consider a family of pairs of mixtures having the same components @xmath32 where @xmath33 is some set of pairs of probability measures defined on @xmath30 .
the family is parametrized by the quadruple @xmath34 .
the condition @xmath35 means that @xmath26 has a greater proportion of @xmath28 compared to @xmath25 .
this is consistent with our assumption that the labeled sample mainly contains positives .
the most general choice for @xmath33 is @xmath36 where @xmath37 is the set of all probability measures defined on @xmath30 and @xmath38 is the set of pairs with equal distributions .
removing equal pairs prevents @xmath25 and @xmath26 from being identical .
we now define the maximum proportion of a component @xmath39 in a mixture @xmath40 , which is used in the results below and to specify the criterion that enables identifiability ; more specifically , @xmath41:\lambda=\alpha\lambda_1+(1-\alpha)\lambda_0 , \lambda_0 \in { { \ensuremath \mathcal{p}}^{\text{all}}}}.\ ] ] of particular interest is the case when @xmath42 , which should be read as  @xmath40 is not a mixture containing @xmath39 " .
we finally define the set all possible @xmath43 that generate @xmath25 and @xmath26 when @xmath44 varies in @xmath33 : @xmath45 if @xmath46 is a singleton set for all @xmath47 , then @xmath48 is identifiable in @xmath43 .
first , we show that the most general choice for @xmath33 , @xmath49 , leads to unidentifiability ( ) .
fortunately , however , by choosing a restricted set @xmath50 as @xmath33 , we do obtain identifiability ( ) . in words , @xmath51 contains pairs of distributions , where each distribution in a pair can not be expressed as a mixture containing the other .
the proofs of the results below are in the appendix .
lemmaunidentifiability [ lem : fpiall ] given a pair of mixtures @xmath52 , let parameters @xmath34 generate @xmath53 and @xmath54 .
it follows that    1 .
[ st : oneone ] there is a one - to - one relation between @xmath44 and @xmath43 and @xmath55 2 .
[ st : validmeasure ] both expressions on the right - hand side of equation  [ eq : mu01 ] are well defined probability measures if and only if @xmath56 and @xmath57 .
[ st : atwoall ] @xmath58 .
[ st : uniden1 ] @xmath59 is unidentifiable in @xmath43 ; i.e. , @xmath43 is not uniquely determined from @xmath53 .
[ st : uniden2 ] @xmath59 is unidentifiable in @xmath19 and @xmath20 , individually ; i.e. , neither @xmath19 nor @xmath20 is uniquely determined from @xmath53 .
observe that the definition of @xmath60 and @xmath61 imply @xmath62 and , consequently , any @xmath63 satisfies @xmath64 , as expected .
theoremidentifiability [ thm : fpires ] given @xmath52 ,
let @xmath65 and @xmath66 .
let @xmath67 , @xmath68 and @xmath69 it follows that    1 .   [ st : generate ] @xmath70 generate @xmath53 2 .
[ st : frompires ] @xmath71 and , consequently , @xmath72 , @xmath73 .
[ st : samesets ] @xmath74 contains all pairs of mixtures in @xmath59 .
[ st : atwores ] @xmath75 .
[ st : iden]@xmath74 is identifiable in @xmath43 ; i.e. , @xmath43 is uniquely determined from @xmath53 .
we refer to the expressions of @xmath25 and @xmath26 as mixtures of components @xmath27 and @xmath28 as a max - canonical form when @xmath44 is picked from @xmath51 .
this form enforces that @xmath28 is not a mixture containing @xmath27 and vice versa , which leads to @xmath27 and @xmath28 having maximum separation , while still generating @xmath25 and @xmath26 .
each pair of distributions in @xmath74 is represented in this form .
identifiability of @xmath74 in @xmath43 occurs precisely when @xmath75 , i.e. , @xmath76 is the only pair of mixing proportions that can appear in a max - canonical form of @xmath25 and @xmath26
. moreover , statement [ st : generate ] in and statement [ st : oneone ] in imply that the max - canonical form is unique and completely specified by @xmath70 , with @xmath77 following from equation [ eq : correction ] .
thus , using @xmath74 to model the unlabeled and labeled data distributions makes estimation of not only @xmath19 , the class prior , but also @xmath78 a well - posed problem .
moreover , due to statement [ st : samesets ] in , there is no loss in the modeling capability by using @xmath74 instead of @xmath59 .
overall , identifiability , absence of loss of modeling capability and maximum separation between @xmath27 and @xmath28 combine to justify estimating @xmath79 as the class prior .
taking values from @xmath80 . in the first step ,
the sample is randomly selected to attempt labeling , with some probability independent of @xmath81 or @xmath82 . if it is not selected , it is added to the  unlabeled " set .
if it is selected , then labeling is attempted .
if the true label is @xmath83 , then with probability @xmath84 , the labeling will succeed and it is added to  noisy positives " . otherwise , it is added to the  dropped " set . if the true label is @xmath85 , then the attempted labeling is much more likely to fail , but because of noise , could succeed .
the attempted label of @xmath86 succeeds with probability @xmath87 , and is added to  noisy positives " , even though it is actually a negative instance . @xmath88 leads to the no noise case and the noise increases as @xmath87 increases .
@xmath89 , gives the proportion of positives in the  noisy positives " . ]
the theory and algorithms for class prior estimation are agnostic to the dimensionality of the data ; in practice , however , this dimensionality can have important consequences .
parametric gaussian mixture models trained via expectation - maximization ( em ) are known to strongly suffer from co - linearity in high - dimensional data .
nonparametric ( kernel ) density estimation is also known to have curse - of - dimensionality issues , both in theory @xcite and in practice @xcite .
we address the curse of dimensionality by transforming the data to a single dimension .
the transformation @xmath90 , surprisingly , is simply an output of a non - traditional classifier trained to separate labeled sample , @xmath91 , from unlabeled sample , @xmath92 .
the transform is similar to that in @xcite , except that it is not required to be calibrated like a posterior distribution ; as shown below , a good ranking function is sufficient .
first , however , we introduce notation and formalize the data generation steps ( figure [ fig_labeling ] ) .
let @xmath81 be a random variable taking values in @xmath0 , capturing the true distribution of inputs , @xmath25 , and @xmath82 be an unobserved random variable taking values in @xmath93 , giving the true class of the inputs .
it follows that @xmath94 and @xmath95 are distributed according to @xmath27 and @xmath28 , respectively .
let @xmath96 be a selection random variable , whose value in @xmath80 determines the sample to which an input @xmath97 is added ( figure [ fig_labeling ] ) . when @xmath98 , @xmath97 is added to the noisy labeled sample ; when @xmath99 , @xmath97 is added to the unlabeled sample ; and when @xmath100 , @xmath97 is not added to either of the samples .
it follows that @xmath101 and @xmath102 are distributed according to @xmath25 and @xmath26 , respectively .
we make the following assumptions which are consistent with the statements above :    @xmath103 assumptions [ eq : sy0 ] and [ eq : sy1 ] states that the proportion of positives in the unlabeled sample and the labeled sample matches the true proportion in @xmath25 and @xmath26 , respectively .
assumption [ eq : sxy ] states that the distribution of the positive inputs ( and the negative inputs ) in both the unlabeled and the labeled samples is equal and unbiased .
gives the implications of these assumptions .
statement [ st : seq2 ] in is particularly interesting and perhaps counter - intuitive as it states that with non - zero probability some inputs need to be dropped .    lemmaassumptions [ lem : assumptions ] let @xmath81 , @xmath82 and @xmath96 be random variables taking values in @xmath0 , @xmath93 and @xmath104 , respectively , and @xmath101 and @xmath102 . for measures
@xmath105 , satisfying equations [ eq : muu ] and [ eq : mul ] and @xmath106 , let @xmath107 give the distribution of @xmath81 , @xmath94 and @xmath95 , respectively . if @xmath108 and @xmath96 satisfy assumptions [ eq : sy0 ] , [ eq : sy1 ] and [ eq : sxy ] , then    1 .
[ st : sx0 ] @xmath81 is independent of @xmath109 ; i.e. , @xmath110 2 .
[ st : xuxl ] @xmath111 and @xmath112 are distributed according to @xmath25 and @xmath26 , respectively .
st : seq2 ] @xmath113 .
the proof is in the appendix .
next , we highlight the conditions under which the score function @xmath114 preserves @xmath79 . observing that @xmath96 serves as the pseudo class label for labeled vs.  unlabeled classification as well ,
we first give an expression for the posterior : @xmath115    theoremtransform [ thm : transform ] let random variables @xmath116 and measures @xmath105 be as defined in .
let @xmath117 be the posterior as defined in and @xmath118 , where @xmath119 is a 1-to-1 function on @xmath120 $ ] and @xmath121 is the composition operator .
assume    1 .
@xmath122 , 2 .
@xmath111 and @xmath112 are continuous with densities @xmath2 and @xmath10 , respectively , 3 .
@xmath123 are the measures corresponding to @xmath124 , respectively , 4 .
@xmath125 and @xmath126 .
then @xmath127 and so @xmath114 is an @xmath79-preserving transformation .
moreover , @xmath117 can also be used to compute the true posterior probability : @xmath128    the proof is in the appendix .
shows that the @xmath79 is the same for the original data and the transformed data , if the transformation function @xmath114 can be expressed as a composition of @xmath117 and a one - to - one function , @xmath119 , defined on @xmath120 $ ] .
trivially , @xmath117 itself is one such function .
we emphasize , however , that @xmath79-preservation is not limited by the efficacy of the calibration algorithm ; uncalibrated scoring that ranks inputs as @xmath129 also preserves @xmath79 .
further demonstrates how the true posterior , @xmath130 , can be recovered from @xmath117 by plugging in estimates of @xmath117 , @xmath131 , @xmath79 and @xmath132 in .
the posterior probability @xmath117 can be estimated directly by using a probabilistic classifier or by calibrating a classifier s score @xcite ; @xmath133 serves as an estimate of @xmath131 ; gives parametric and nonparametric approaches for estimation of @xmath79 and @xmath132 .
in this section , we derive a parametric and a nonparametric algorithm to estimate @xmath79 and @xmath132 from the unlabeled sample , @xmath134 , and the noisy positive sample , @xmath135 . in theory , both approaches can handle multivariate samples ; in practice , however , to circumvent the curse of dimensionality , we exploit the theory of @xmath79-preserving univariate transforms to transform the samples .
* parametric approach . *
the parametric approach is derived by modeling each sample as a two component gaussian mixture , sharing the same components but having different mixing proportions : @xmath136 where @xmath137 and @xmath138 , the set of all @xmath139 positive definite matrices .
the algorithm is an extension to the em approach for gaussian mixture models ( gmms ) where , instead of estimating the parameters of a single mixture , the parameters of both mixtures @xmath140 are estimated simultaneously by maximizing the combined likelihood over both @xmath141 and @xmath142 .
this approach , which we refer to as a multi - sample gmm ( msgmm ) , exploits the constraint that the two mixtures share the same components .
the update rules and their derivation are given in the appendix .
* nonparametric approach . *
our nonparametric strategy directly exploits the results of and , which give a direct connection between @xmath143 ) and @xmath144 ) .
therefore , for a two - component mixture sample , @xmath145 , and a sample from one of the components , @xmath146 , it only requires an algorithm to estimate the maximum proportion of @xmath147 in @xmath148 . for this purpose
, we use the alphamax  algorithm @xcite , briefly summarized in the appendix .
specifically , our two - step approach for estimating @xmath79 and @xmath132 is as follows : ( @xmath149 ) estimate @xmath150 and @xmath151 as outputs of alphamax@xmath152 and alphamax@xmath153 , respectively ; ( @xmath154 ) estimate @xmath76 from the estimates of @xmath155 by applying .
we refer to our nonparametric algorithm as alphamax - n .
in this section we systematically evaluate the new algorithms in a controlled , synthetic setting as well as on a variety of data sets from the uci machine learning repository @xcite .    * experiments on synthetic data : * we start by evaluating all algorithms in a univariate setting where both mixing proportions , @xmath156 and @xmath157 , are known .
we generate unit - variance gaussian and unit - scale laplace - distributed i.i.d .
samples and explore the impact of mixing proportions , the size of the component sample , and the separation and overlap between the mixing components on the accuracy of estimation .
the class prior @xmath156 was varied from @xmath158 and the noise component @xmath157 from @xmath159 .
the size of the labeled sample @xmath91 was varied from @xmath160 , whereas the size of the unlabeled sample @xmath92 was fixed at @xmath161 .
* experiments on real - life data : * we considered twelve real - life data sets from the uci machine learning repository . to adjust these data to our problems , categorical features were transformed into numerical using sparse binary representation , the regression data sets were transformed into classification based on mean of the target variable , and the multi - class classification problems were converted into binary problems by combining classes . in each data set , a subset of positive and negative examples was randomly selected to provide a labeled sample while the remaining data ( without class labels ) were used as unlabeled data .
the size of the labeled sample was kept at @xmath162 ( or @xmath163 for small data sets ) and the maximum size of unlabeled data was set @xmath161 .    *
algorithms : * we compare the alphamax - n and msgmm algorithms to the elkan - noto algorithm @xcite as well as the noiseless version of alphamax @xcite .
there are several versions of the elkan - noto estimator and each can use any underlying classifier .
we used the @xmath164 alternative estimator combined with the ensembles of 100 two - layer feed - forward neural networks , each with five hidden units .
the out - of - bag scores of the same classifier were used as a class - prior preserving transformation that created an input to the alphamax algorithms .
it is important to mention that neither elkan - noto nor alphamax algorithm was developed to handle noisy labeled data .
in addition , the theory behind the elkan - noto estimator restricts its use to class - conditional distributions with non - overlapping supports .
the algorithm by @xcite minimizes the same objective as the @xmath164 elkan - noto estimator and , thus , was not implemented .
* evaluation : * all experiments were repeated 50 times to be able to draw conclusions with statistical significance . in real - life data , the labeled sample was created randomly by choosing an appropriate number of positive and negative examples to satisfy the condition for @xmath157 and the size of the labeled sample , while the remaining data was used as the unlabeled sample .
therefore , the class prior in the unlabeled data varies with the selection of the noise parameter @xmath157 .
the mean absolute difference between the true and estimated class priors was used as a performance measure .
the best performing algorithm on each data set was determined by multiple hypothesis testing using the p - value of @xmath165 and bonferroni correction .
* results : * the comprehensive results for synthetic data drawn from univariate gaussian and laplace distributions are shown in appendix ( table 2 ) . in these experiments
no transformation was applied prior to running any of the algorithms . as expected ,
the results show excellent performance of the msgmm model on the gaussian data .
these results significantly degrade on laplace - distributed data , suggesting sensitivity to the underlying assumptions .
on the other hand , alphamax - n was accurate over all data sets and also robust to noise .
these results suggest that new parametric and nonparametric algorithms perform well in these controlled settings .
table 1 shows the results on twelve real data sets . here ,
alphamax and alphamax - n algorithms demonstrate significant robustness to noise , although the parametric version msgmm was competitive in some cases .
on the other hand , the elkan - noto algorithm expectedly degrades with noise . finally , we investigated the practical usefulness of the @xmath166-preserving transform .
( appendix ) shows the results of alphamax - n and msgmm  on the real data sets , with and without using the transform .
because of computational and numerical issues , we reduced the dimensionality by using principal component analysis ( the original data caused matrix singularity issues for msgmm and density estimation issues for alphamax - n ) .
msgmm  deteriorates significantly without the transform , whereas alphamax - n  preserves some signal for the class prior .
alphamax - n  with the transform , however , shows superior performance on most data sets .
1.01 |>x| > x| >
x| > x| > x| > x| >
x| > x| > x| > x| > x| & @xmath19 & @xmath20 & auc & @xmath167 & @xmath168 & @xmath169 & & & & + & 0.095 & 1.00 & 0.842 & 13 & 5188 & 45000 & 0.241 & 0.070 & * 0.037 * * & 0.163 + & 0.096 & 0.95 & 0.819 & 13 & 5188 & 45000 & 0.284 & 0.079 & * 0.036 * * & 0.155 + & 0.101 & 0.75 & 0.744 & 13 & 5188 & 45000 & 0.443 & 0.124 & * 0.040 * * & 0.127 + & 0.419 & 1.00 & 0.685 & 8 & 490 & 1030 & 0.329 & 0.141 & 0.181 & * 0.077 * * + & 0.425 & 0.95 & 0.662 & 8 & 490 & 1030 & 0.363 & 0.174 & 0.231 & * 0.095 * * + & 0.446 & 0.75 & 0.567 & 8 & 490 & 1030 & 0.531 & * 0.212 * & 0.272 & 0.233 + & 0.342 & 1.00 & 0.825 & 127 & 2565 & 5574 & 0.017 & 0.011 & 0.017 & * 0.008 * * + & 0.353 & 0.95 & 0.795 & 127 & 2565 & 5574 & 0.078 & 0.016 & * 0.006 * & 0.006 + & 0.397 & 0.75 & 0.672 & 127 & 2565 & 5574 & 0.396 & 0.137 & 0.009 & * 0.006 * * + & 0.268 & 1.00 & 0.810 & 13 & 209 & 506 & 0.159 & * 0.087 * & 0.094 & 0.209 + & 0.281 & 0.95 & 0.777 & 13 & 209 & 506 & 0.226 & * 0.094 * & 0.110 & 0.204 + & 0.330 & 0.75 & 0.651 & 13 & 209 & 506 & 0.501 & * 0.125 * & 0.134 & 0.172 + & 0.093 & 1.00 & 0.933 & 36 & 1508 & 6435 & 0.074 & 0.009 & * 0.007 * * & 0.157 + & 0.103 & 0.95 & 0.904 & 36 & 1508 & 6435 & 0.110 & 0.015 & * 0.008 * * & 0.152 + & 0.139 & 0.75 & 0.788 & 36 & 1508 & 6435 & 0.302 & 0.063 & * 0.012 * * & 0.143 + & 0.409 & 1.00 & 0.792 & 126 & 3916 & 8124 & 0.029 & * 0.015 * * & 0.022 & 0.037 + & 0.416 & 0.95 & 0.766 & 126 & 3916 & 8124 & 0.087 & 0.015 & * 0.008 * * & 0.037 + & 0.444 & 0.75 & 0.648 & 126 & 3916 & 8124 & 0.370 & 0.140 & * 0.020 * & 0.024 + & 0.086 & 1.00 & 0.885 & 10 & 560 & 5473 & 0.116 & * 0.026 * * & 0.044 & 0.129 + & 0.087 & 0.95 & 0.858 & 10 & 560 & 5473 & 0.137 & * 0.031 * * & 0.052 & 0.125 + & 0.090 & 0.75 & 0.768 & 10 & 560 & 5473 & 0.256 & * 0.041 * * & 0.064 & 0.111 + & 0.243 & 1.00 & 0.875 & 16 & 3430 & 10992 & 0.030 & * 0.006 * * & 0.009 & 0.081 + & 0.248 & 0.95 & 0.847 & 16 & 3430 & 10992 & 0.071 & 0.011 & * 0.005 * * & 0.074 + & 0.268 & 0.75 & 0.738 & 16 & 3430 & 10992 & 0.281 & 0.093 & * 0.007 * * & 0.062 + & 0.251 & 1.00 & 0.735 & 8 & 268 & 768 & 0.351 & 0.120 & * 0.111 * & 0.171 + & 0.259 & 0.95 & 0.710 & 8 & 268 & 768 & 0.408 & 0.118 & * 0.110 * & 0.168 + & 0.289 & 0.75 & 0.623 & 8 & 268 & 768 & 0.586 & * 0.144 * & 0.156 & 0.175 + & 0.139 & 1.00 & 0.929 & 9 & 8903 & 58000 & * 0.024 * * & 0.027 & 0.029 & 0.157 + & 0.140 & 0.95 & 0.903 & 9 & 8903 & 58000 & 0.052 & * 0.004 * * & 0.007 & 0.157 + & 0.143 & 0.75 & 0.802 & 9 & 8903 & 58000 & 0.199 & 0.047 & * 0.004 * * & 0.148 + & 0.226 & 1.00 & 0.842 & 57 & 1813 & 4601 & 0.184 & 0.046 & * 0.041 * & 0.059 + & 0.240 & 0.95 & 0.812 & 57 & 1813 & 4601 & 0.246 & 0.059 & * 0.042 * * & 0.063 + & 0.295 & 0.75 & 0.695 & 57 & 1813 & 4601 & 0.515 & 0.155 & * 0.044 * * & 0.059 + & 0.566 & 1.00 & 0.626 & 11 & 4113 & 6497 & 0.290 & 0.083 & * 0.060 * & 0.070 + & 0.575 & 0.95 & 0.610 & 11 & 4113 & 6497 & 0.322 & 0.113 & * 0.063 * & 0.076 + & 0.612 & 0.75 & 0.531 & 11 & 4113 & 6497 & 0.420 & 0.322 & 0.353 & * 0.293 * +
class prior estimation in a semi - supervised setting , including positive - unlabeled learning , has been extensively discussed previously ; see @xcite and references therein . recently , a general setting for label noise has also been introduced , called the mutual contamination model .
the aim under this model is to estimate multiple unknown base distributions , using multiple random samples that are composed of different convex combinations of those base distributions @xcite .
the setting of asymmetric label noise is a subset of this more general setting , treated under general conditions by @xcite , and previously investigated under a more restrictive setting as co - training @xcite .
a natural approach is to use robust estimation to learn in the presence of class noise ; this strategy , however , has been shown to be ineffective , both theoretically @xcite and empirically @xcite , indicating the need to explicitly model the noise .
generative mixture model approaches have also been developed , which explicitly model the noise @xcite ; these algorithms , however , assume labeled data for each class . as the most related work , though @xcite did not explicitly treat the positive - unlabeled learning with noisy positives , their formulation can incorporate this setting by using @xmath170 and @xmath171 . the theoretical and algorithmic treatment , however , is very different .
their focus is on identifiability and analyzing convergence rates and statistical properties , assuming access to some @xmath172 function which can obtain proportions between samples .
they do not explicitly address issues with high - dimensional data nor focus on algorithms to obtain @xmath172 .
in contrast , we focus primarily on the univariate transformation to handle high - dimensional data and practical algorithms for estimating @xmath173 .
supervised learning used for class prior - preserving transformation provides a rich set of techniques to address high - dimensional data .
in this paper , we developed a practical algorithm for classification of positive - unlabeled data with noise in the labeled data set .
in particular , we focused on a strategy for high - dimensional data , providing a univariate transform that reduces the dimension of the data , preserves the class prior so that estimation in this reduced space remains valid and is then further useful for classification .
this approach provides a simple algorithm that simultaneously improves estimation of the class prior and provides a resulting classifier .
we derived a parametric and a nonparametric version of the algorithm and then evaluated its performance on a wide variety of learning scenarios and data sets . to the best of our knowledge
, this algorithm represents one of the first practical and easy - to - use approaches to learning with high - dimensional positive - unlabeled data with noise in the labels .
we thank prof .
michael w. trosset for helpful comments .
grant support : nsf dbi-1458477 , nih r01mh105524 , nih r01gm103725 , and the indiana university precision health initiative .    * *
we will need the following lemma for the proofs .
[ thm : altchart ] let @xmath25 , @xmath28 and @xmath27 be three measures defined on @xmath30 such that @xmath174 for some @xmath175 $ ] .
define @xmath176 it follows that    1 .
[ st : altchart ] @xmath177 2 .
[ st : resmu0 ] if @xmath178 , then @xmath179 .
the proof follows from lemma 4 and theorem 3 of @xcite .    as a consequence of statement 1 in
, we use an alternate characterization of @xmath60 @xmath180 where @xmath181 .
[ [ statement - stoneone ] ] statement [ st : oneone ] : + + + + + + + + + + + + + + + + + + + + + +    because @xmath34 generate @xmath25 and @xmath26 , @xmath182 and equations [ eq : muu ] and [ eq : mul ] hold .
@xmath27 can be expressed in terms of @xmath183 by eliminating @xmath28 ( @xmath184 ( equation [ eq : muu ] ) @xmath185 ( equation [ eq : mul ] ) ) .
similarly @xmath28 can be expressed in terms of @xmath183 and equation [ eq : mu01 ] follows .
thus , given @xmath25 and @xmath26 , @xmath186 can be uniquely determined from @xmath187 .
equation [ eq : muu ] implies that for all @xmath188 such that @xmath189 , @xmath190 .
existence of @xmath191 is guaranteed because @xmath106 from the definition of @xmath49 .
thus , given @xmath25 , @xmath19 is uniquely determined from @xmath192 .
similarly , equation [ eq : mul ] implies that , given @xmath26 , @xmath20 is uniquely determined from @xmath192 .
* statement [ st : validmeasure ] : * it is easy to observe that @xmath193 is a valid probability measure , if and only if @xmath194 for all @xmath188 . because the inequality is trivially true when @xmath195 , the necessary and sufficient condition can be reduced to @xmath194 for all @xmath188 with @xmath196 .
this can be rewritten as @xmath197 for all @xmath188 such that @xmath198 .
equivalently , @xmath199 is a lower bound to @xmath200 ; in other words , @xmath56 due to the alternate characterization given by equation [ eq : altchar ] .
similarly , @xmath201 is a valid probability measure provided @xmath57 .
* statement [ st : atwoall ] : * because system of equations in [ eq : mu01 ] is essentially equivalent to equations [ eq : muu ] and [ eq : mul ] , @xmath63 if and only if , @xmath27 and @xmath28 as defined in equations [ eq : mu01 ] come from @xmath49 . when the conditions of statement [ st : validmeasure ] are not satisfied , @xmath27 , @xmath28 are not well - defined probability measures and consequently @xmath202 . on the other hand , when they are satisfied , @xmath27 , @xmath28 are well - defined probability measures .
moreover , @xmath61 implies @xmath203 and consequently @xmath204 .
* statement [ st : uniden2 ] : * we construct mixtures @xmath205 and @xmath206 such that there are multiple values for @xmath19 that generate the mixtures .
for @xmath207 and probability measures @xmath208 on @xmath30 , let @xmath209 generate mixtures @xmath205 and @xmath206 .
thus , @xmath210 .
it follows that @xmath211 and @xmath212 are both strictly greater than @xmath213 because @xmath214 implies @xmath215 and @xmath216 implies @xmath217 ( using statement [ st : atwoall ] ) , which contradicts our assumption .
now , @xmath218 follows trivially from statement [ st : atwoall ] .
it is easy to observe that @xmath219 for any @xmath220 $ ] .
thus , there are multiple values for @xmath19 that generate @xmath205 and @xmath206 .
similarly , @xmath221 for any @xmath222 $ ] and there are multiple values for @xmath20 that generate @xmath205 and @xmath206 .
* statement [ st : uniden1 ] : * statement [ st : uniden1 ] follows trivially from statement [ st : uniden2 ] and also by observing that @xmath223 can not be a singleton set because @xmath224 .
[ [ statement - stgenerate ] ] statement [ st : generate ] : + + + + + + + + + + + + + + + + + + + + + + + +    first , we show that @xmath225 are well defined probability measures .
+ @xmath226 : suppose @xmath227 .
it follows that @xmath228 and , consequently , from , @xmath229 .
however , @xmath203 because they are picked from @xmath49 .
thus , @xmath230 by contradiction .
similarly @xmath231 .
thus , the denominator in the r.h.s . of @xmath232 and @xmath233 is not @xmath213 .
+ by definition @xmath234 .
thus , @xmath235 when @xmath198 .
consequently , @xmath236 .
the inequality is trivially true when @xmath195 .
thus , @xmath236 for all @xmath188 .
hence , @xmath232 is a probability measure .
similarly , @xmath233 is also a probability measure .
+ second , we show that @xmath70 generate @xmath237 .
+ @xmath238 : observe that @xmath225 can also be expressed as @xmath239 and @xmath240 . moreover , after some algebraic manipulation of equations labeled [ eq : correction ] , @xmath241 and @xmath242 can be derived .
thus , from ( statements [ st : oneone ] and [ st : atwoall ] )
, @xmath70 generate @xmath237 .     * statement [ st : frompires ] : * we show that @xmath243 and @xmath244 , giving @xmath71 . + we start by showing that @xmath245 .
suppose , for some @xmath246 and @xmath247 thus , @xmath248 is a lower bound to @xmath249 and , consequently , @xmath250 .
however , because @xmath251 , @xmath252 .
this is a contradiction .
thus , for all @xmath246 there exists some @xmath253 such that @xmath254 .
we now divide the inequality on both sides by @xmath255 , observing that the divisor is strictly greater than @xmath213 because @xmath256 as @xmath233 is a probability measure .
thus , @xmath257 because the choice of @xmath258 is arbitrary and @xmath231 , any lower bound of @xmath259 can not be greater than @xmath213 .
thus , @xmath260 .
a similar argument shows that @xmath261 therefore , @xmath243 and @xmath244 ; consequently , @xmath71 . because @xmath232 is not a mixture containing @xmath233 and because @xmath262 , from statement [ st : resmu0 ] in we get that @xmath72 .
similarly , @xmath73 .
* statement [ st : samesets ] : * because statements [ st : generate ] and [ st : frompires ] are true for any @xmath52 , statement [ st : samesets ] is true .
* statement [ st : atwores ] : * it follows from statements [ st : generate ] and [ st : frompires ] , that @xmath263 . to show that @xmath264 contains no other element
, we give a proof by contradiction .
suppose @xmath265 and @xmath266 .
let @xmath267 generate @xmath53 , for some @xmath268 .
first , we show that @xmath269 and @xmath270 : + @xmath271 : suppose for some @xmath272 and all @xmath273 where @xmath198 , @xmath274 however , this is a contradiction because @xmath122 .
thus , for every @xmath275 there exists some @xmath276 with @xmath277 such that @xmath278 .
thus , @xmath279 , because @xmath258 can be made arbitrarily small , @xmath280 .
however , because @xmath281 , @xmath43 also belongs to @xmath282 and consequently @xmath283 from ( statement [ st : atwoall ] ) .
thus , @xmath284 .
+ @xmath285 : the proof is similar to @xmath284 supposing @xmath286 for some @xmath287 and all @xmath273 with @xmath288 , reaching a contradiction and following the subsequent steps .
+ @xmath269 , @xmath270 and implies @xmath289 and @xmath290 , which contradicts our assumption .
hence , @xmath76 is the only element in @xmath264 , which proves statement [ st : atwores ] .
* statement [ st : iden ] : * this statement follows by observing that @xmath264 is a singleton set .
we first prove , which we can then use to construct a suitable univariate transform .
[ [ statement - stsx0 ] ] statement [ st : sx0 ] : + + + + + + + + + + + + + + + + + + +    observe that @xmath291 thus , @xmath81 is independent of @xmath109 .
[ [ statement - stxuxl ] ] statement [ st : xuxl ] : + + + + + + + + + + + + + + + + + + + +    from statement [ st : sx0 ] , @xmath111 has the same distribution as @xmath81 , which is @xmath25 .
now , @xmath292 thus , the distribution of @xmath293 is @xmath294 , which is @xmath26 .
[ [ statement-3 ] ] statement 3 : + + + + + + + + + + + +    now , @xmath295 the probability @xmath296 is independent of @xmath97 only if @xmath297 is a constant with respect to @xmath97 .
let @xmath298 , where @xmath299 is some constant . integrating over @xmath97 on both sides gives @xmath300 .
since both integrals are @xmath301 , it follows that @xmath302 .
thus , @xmath303 , which implies @xmath228 ; i.e. , the labeled and unlabeled samples have the same distribution . however , this implies @xmath304 , which contradicts the assumption .
therefore , @xmath305 is not independent of @xmath81 .    to prove the main result about the preservation properties of the univariate transform
, we will make use of the following theorem .
[ thm : transformpdf ] let @xmath81 and @xmath306 be random variables with densities @xmath2 and @xmath11 and measures @xmath25 and @xmath28 respectively . for @xmath307 and an abstract space @xmath308 , given any one - to - one function @xmath309 , define function @xmath310    @xmath311    where @xmath312    let @xmath313 and @xmath314 be the measures for the random variables @xmath315 , @xmath316 respectively for @xmath29-algebra @xmath317 on @xmath308
. then @xmath318 .
first we prove that @xmath319 . to this end , we expand @xmath129 as follows @xmath320 the last step is justified because @xmath321 and @xmath322 .
consider one - to - one functions @xmath323 and @xmath324 defined on @xmath325 .
we can apply to @xmath326 to get @xmath327 and to @xmath328 to get @xmath329 . to satisfy the condition of for @xmath326 ,
let @xmath111 , @xmath112 and @xmath330 play the role of @xmath81 , @xmath306 and @xmath331 , respectively .
now , @xmath332 because @xmath119 and @xmath326 are both one - to - one functions , @xmath330 is one - to - one as well .
thus , all the conditions of are satisfied and , consequently , @xmath327 .
similarly , we can use with @xmath111 , @xmath112 and @xmath333 playing the role of @xmath306 , @xmath81 and @xmath331 , respectively , now giving that @xmath329 . from statement
[ st : frompires ] in and @xmath334 and , thus , @xmath335 .
next we prove .
let @xmath336 . rearranging the terms of , @xmath337 rearranging the terms , @xmath338
let @xmath134 and @xmath135 be the unlabeled sample and the noisy positive sample , respectively .
the parametric approach is derived by modeling each sample as a two component gaussian mixture , sharing the same components but having different mixing proportions : @xmath136 where @xmath137 and @xmath138 , the set of all @xmath139 positive definite matrices .
the algorithm is an extension to the em approach for gaussian mixture models ( gmms ) where , instead of estimating the parameters of a single mixture , the parameters of both mixtures @xmath140 are estimated simultaneously by maximizing the combined likelihood over both @xmath141 and @xmath142 .
this approach , that we refer to as a multi - sample gmm ( msgmm ) , exploits the constraint that the two mixtures share the same components .
+ to derive the update equations , we introduce missing variables @xmath339 that give the true class of the @xmath340 and @xmath341 example in @xmath92 and @xmath91 , respectively . the variables @xmath339 are bernoulli distributed ; i.e. , @xmath342 and @xmath343 .
for @xmath344 the quartet @xmath345 forms the observed and unobserved variables in the em framework .
the complete data log - likelihood , @xmath346 is given by , @xmath347 where @xmath348 is the density of @xmath349 .
our goal is to maximize @xmath350 $ ] .
to do so we take the conditional expectation of @xmath346 with respect to @xmath351 and @xmath352 given @xmath92 and @xmath91 . for @xmath353=\frac{{\alpha}{\phi_1}({x^{u}}_i)}{{\alpha}{\phi_1}({x^{u}}_i)+(1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\alpha}){\phi_0}({x^{u}}_i)},\\      { \bar{w}^l}_i = e[{w^{l}}_i|{{x^{l}}_i={x^{l}}_i}]=\frac{{\beta}{\phi_1}({x^{l}}_i)}{{\beta}{\phi_1}({x^{l}}_i)+(1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\beta}){\phi_0}({x^{l}}_i)},\end{aligned}\ ] ] we obtain @xmath354 & = \sum_{i=1}^{|{u}| } { \bar{w}^u}_i \log\brac*{{\alpha}{\phi_1}({x^{u}}_i ) } + ( 1-{\bar{w}^u}_i ) \log\brac*{(1-{\alpha}){\phi_0}({x^{u}}_i ) } \\      & \hspace{1em }   + \sum_{i=1}^{|{l}| } { \bar{w}^l}_i \log\brac*{{\beta}{\phi_1}({x^{l}}_i ) } + ( 1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\bar{w}^l}_i ) \log\brac*{(1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\beta}){\phi_0}({x^{l}}_i)}\end{aligned}\ ] ] which up to constants , that are ignored in the optimization , can be explicitly written as @xmath355 = & \sum_{i=1}^{|{u}| } { \bar{w}^u}_i \brac * { \log{\alpha}{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}\frac{1}{2}\log |{\sigma_1}| { \raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}({x^{u}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_1})^{t}{\sigma_1}^{-1}({x^{u}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_1 } ) } \\      & +   \sum_{i=1}^{|{u}| } ( 1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\bar{w}^u}_i ) \brac * { \log(1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\alpha } ) { \raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}\frac{1}{2}\log |{\sigma_0}| { \raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}({x^{u}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_0})^{t}{\sigma_0}^{-1}({x^{u}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_0 } ) } \\      & +   \sum_{i=1}^{|{l}| } { \bar{w}^l}_i \brac * { \log{\beta}{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}\frac{1}{2}\log |{\sigma_1}| { \raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}({x^{l}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_1})^{t}{\sigma_1}^{-1}({x^{l}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_1 } ) } \\      & +   \sum_{i=1}^{|{l}| } ( 1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\bar{w}^l}_i ) \brac * { \log(1{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{\beta } ) { \raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}\frac{1}{2}\log |{\sigma_0}| { \raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}({x^{l}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_0})^{t}{\sigma_0}^{-1}({x^{l}}_i{\raisebox{0.05em}{$\scalebox{0.8}{$-$}$}}{u_0})}.\end{aligned}\ ] ]    finally , we obtain the parameter update equations by maximizing @xmath350 $ ] with respect to @xmath356 : @xmath357 the update rules reduce to the standard gmm when the labeled sample is not provided . further generalization to more than two samples and/or
mixing components is straightforward .
for a mixture sample @xmath148 and a component sample @xmath147 , alphamax@xmath358 estimates the maximum proportion of @xmath147 in @xmath148 @xcite .
alphamax  is based on the constrained maximization of the log likelihood of samples @xmath145 and @xmath146 , derived using nonparametric estimates of their densities @xmath359 and @xmath360 , respectively .
we list the main steps of alphamax  below .    1 .
estimate @xmath360 nonparameterically as @xmath361 using sample @xmath146 .
obtain the weights , @xmath362 , and components @xmath363 from nonparametric density estimation of @xmath359 as a @xmath364-component mixture , @xmath365 , using @xmath145 .
2 .   construct two density functions @xmath366 and @xmath367 from @xmath368 and @xmath361 parameterized by a @xmath364-dimensional weight vector @xmath369 $ ] , @xmath370 , which re - weights components @xmath363 : @xmath371 3 .   maximize the log likelihood of @xmath145 and @xmath146 constructed with @xmath372 and @xmath373 under the constraint @xmath374 for many values of @xmath375 equispaced in @xmath120 $ ] .
@xmath376 4 .
estimate the maximum proportion of @xmath360 in @xmath359 , @xmath377 ( minor abuse of notation is a minor abuse of notation ; replacing the densities with the underlying measures makes the notation correct . ] ) , as the @xmath97-coordinate of the elbow in the @xmath378 versus @xmath375 graph .
the densities @xmath367 and @xmath366 are constructed to approximate @xmath359 and @xmath360 .
the efficacy of the approximation depends on the value of @xmath379 ; there exists @xmath379 such that @xmath367 and @xmath366 are good approximations provided @xmath380 , however , the approximation deteriorates progressively , even with the optimum @xmath379 , as @xmath381 moves beyond @xmath377 .
this suggests that the graph of @xmath378 versus @xmath375 should be approximately a flat line from @xmath213 to @xmath377 and decrease progressively beyond @xmath377 exposing an elbow at @xmath377 , which is detected in the last step .
the pseudo code for elbow detection is provided in @xcite .
the practical implementation , backed by the @xmath166-preservation theory , reduces the dimension of the data to a single dimension by using the scoring function of a non - traditional classifier and employs histograms as the nonparametric method to obtain @xmath382 and @xmath361 .
the bin - width is chosen to cover the component sample s ( after the transformation ) range and reveal the shape of its distribution , using the default option in matlab s ` histogram ` function .
more bins with the same bin - width are subsequently added to cover the mixture sample s range .
the total computation includes the time to ( a ) train a classifier ( b ) perform density estimation in 1d and ( c ) perform optimization in alphamax .
the complexity for ( a ) varies ; for neural networks it is @xmath383 per epoch . for ( b )
we used @xmath384 bin histograms , where @xmath384 can be @xmath385 to @xmath386 depending on the bandwidth selection rule , giving @xmath387 complexity . for ( c ) , ( size @xmath384 optimization ) the computation of the objective and gradient is @xmath387 per step ; e.g. , lbfgs is @xmath387 per step .
an execution of alphamax takes about 10 minutes on a laptop computer for the shuttle data set ( for results reported in table 1 ; i.e. , 1000 labeled and 10000 unlabeled examples ) ; 12 hours on the entire data ( 8903 labeled and 49097 unlabeled examples ) .
[ [ results - for - univariate - synthetic - data . ] ] results for univariate synthetic data . + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +      1.01 |>x| > x| > x| > x| > x| > x| > x| > x| > x| >
x| > x| & & & & & & + & @xmath156 & @xmath157 & 100 & 1000 & 100 & 1000 & 100 & 1000 & 100 & 1000 + & 0.05 & 1.00 & 0.473 & 0.460 & 0.079 & 0.102 & 0.064 & 0.085 & * 0.034 * * & * 0.020 * * + & 0.25 & 1.00 & 0.484 & 0.435 & 0.132 & 0.160 & 0.123 & 0.122 & * 0.063 * * & * 0.044 * * + & 0.50 & 1.00 & 0.395 & 0.347 & 0.155 & 0.125 & 0.173 & 0.096 & * 0.073 * * & * 0.040 * * + & 0.05 & 0.95 & 0.484 & 0.496 & 0.099 & 0.124 & 0.076 & 0.099 & * 0.039 * * & * 0.022 * * + & 0.25 & 0.95 & 0.510 & 0.469 & 0.127 & 0.167 & 0.115 & 0.111 & * 0.074 * * & * 0.037 * * + & 0.50 & 0.95 & 0.433 & 0.378 & 0.180 & 0.152 & 0.186 & 0.087 & * 0.068 * * & * 0.048 * * + & 0.05 & 0.75 & 0.663 & 0.630 & 0.124 & 0.135 & 0.089 & 0.076 & * 0.056 * * & * 0.024 * * + & 0.25 & 0.75 & 0.641 & 0.608 & 0.152 & 0.209 & 0.141 & 0.120 & * 0.141 * & * 0.060 * * + & 0.50 & 0.75 & 0.548 & 0.485 & 0.219 & 0.218 & 0.244 & 0.137 & * 0.040 * * & * 0.068 * * +    & 0.05 & 1.00 & 0.112 & 0.136 & 0.018 & 0.016 & 0.017 & 0.015 & * 0.006 * * & * 0.004 * * + & 0.25 & 1.00 & 0.177 & 0.168 & 0.050 & 0.049 & 0.049 & 0.042 & * 0.018 * * & * 0.010 * * + & 0.50 & 1.00 & 0.219 & 0.153 & 0.104 & 0.053 & 0.109 & 0.043 & * 0.024 * * & * 0.015 * * + & 0.05 & 0.95 & 0.125 & 0.162 & 0.015 & 0.019 & 0.015 & 0.015 & * 0.009 * * & * 0.004 * * + & 0.25 & 0.95 & 0.212 & 0.208 & 0.063 & 0.061 & 0.060 & 0.043 & * 0.025 * * & * 0.011 * * + & 0.50 & 0.95 & 0.271 & 0.211 & 0.099 & 0.077 & 0.106 & 0.046 & * 0.023 * * & * 0.018 * * +    & 0.05 & 0.75 & 0.215 & 0.285 & 0.030 & 0.036 & * 0.022 * & 0.012 & 0.034 & * 0.005 * * + & 0.25 & 0.75 & 0.405 & 0.403 & 0.100 & 0.139 & 0.084 & 0.036 & * 0.025 * * & * 0.014 * * + & 0.50 & 0.75 & 0.457 & 0.415 & 0.156 & 0.184 & 0.159 & 0.048 & * 0.030 * * & * 0.020 * * + & 0.05 & 1.00 & 0.010 & 0.013 & 0.022 & 0.019 & 0.022 & 0.019 & * 0.001 * * & * 0.001 * * + & 0.25 & 1.00 & 0.032 & 0.018 & 0.022 & 0.004 & 0.027 & 0.008 & * 0.002 * * & * 0.002 * * + & 0.50 & 1.00 & 0.070 & 0.020 & 0.032 & 0.005 & 0.039 & 0.012 & * 0.002 * * & * 0.002 * * + & 0.05 & 0.95 & 0.018 & 0.029 & 0.018 & 0.014 & 0.019 & 0.015 & * 0.001 * * & * 0.001 * * + & 0.25 & 0.95 & 0.063 & 0.056 & 0.010 & 0.017 & 0.013 & 0.009 & * 0.002 * * & * 0.001 * * + & 0.50 & 0.95 & 0.115 & 0.083 & 0.040 & 0.032 & 0.045 & 0.010 & * 0.002 * * & * 0.001 * * +    & 0.05 & 0.75 & 0.062 & 0.114 & 0.011 & 0.006 & 0.018 & 0.015 & * 0.001 * * & * 0.001 * * + & 0.25 & 0.75 & 0.236 & 0.249 & 0.063 & 0.090 & 0.026 & 0.005 & * 0.002 * * & * 0.002 * * + & 0.50 & 0.75 & 0.380 & 0.353 & 0.130 & 0.168 & 0.106 & 0.021 & * 0.002 * * & * 0.002 * * + & 0.05 & 1.00 & 0.410 & 0.389 & 0.195 & 0.256 & * 0.147 * * & * 0.190 * * & 0.418 & 0.390 + & 0.25 & 1.00 & 0.410 & 0.356 & 0.151 & 0.209 & * 0.103 * * & * 0.117 * * & 0.148 & 0.183 + & 0.50 & 1.00 & 0.367 & 0.299 & 0.195 & 0.154 & * 0.190 * & * 0.038 * * & 0.243 & 0.236 + & 0.05 & 0.95 & 0.455 & 0.430 & 0.204 & 0.271 & * 0.140 * * & * 0.192 * * & 0.424 & 0.406 + & 0.25 & 0.95 & 0.455 & 0.401 & 0.180 & 0.230 & * 0.115 * * & * 0.112 * * & 0.157 & 0.187 + & 0.50 & 0.95 & 0.412 & 0.331 & 0.222 & 0.179 & * 0.214 * & * 0.047 * * & 0.249 & 0.241 + & 0.05 & 0.75 & 0.593 & 0.578 & 0.225 & 0.325 & * 0.133 * * & * 0.181 * * & 0.415 & 0.440 + & 0.25 & 0.75 & 0.602 & 0.562 & 0.191 & 0.327 & * 0.119 * * & * 0.123 * * & 0.210 & 0.230 + & 0.50 & 0.75 & 0.520 & 0.470 & 0.264 & 0.278 & 0.272 & * 0.053 * * & * 0.254 * & 0.247 +    & 0.05 & 1.00 & 0.116 & 0.123 & 0.052 & 0.061 & * 0.045 * & 0.056 & 0.448 & * 0.014 * * + & 0.25 & 1.00 & 0.158 & 0.132 & 0.054 & 0.067 & 0.041 & 0.051 & * 0.037 * & * 0.027 * * + & 0.50 & 1.00 & 0.186 & 0.125 & 0.077 & 0.049 & * 0.074 * & * 0.021 * * & 0.130 & 0.068 + & 0.05 & 0.95 & 0.131 & 0.146 & 0.053 & 0.066 & * 0.042 * * & 0.053 & 0.455 & * 0.016 * * + & 0.25 & 0.95 & 0.186 & 0.175 & 0.071 & 0.082 & * 0.049 * & 0.049 & 0.055 & * 0.021 * * + & 0.50 & 0.95 & 0.239 & 0.183 & 0.082 & 0.082 & * 0.073 * & * 0.024 * * & 0.134 & 0.075 +    & 0.05 & 0.75 & 0.230 & 0.268 & 0.079 & 0.097 & * 0.046 * * & * 0.052 * * & 0.461 & 0.261 + & 0.25 & 0.75 & 0.375 & 0.377 & 0.123 & 0.159 & * 0.090 * * & * 0.039 * & 0.191 & 0.050 + & 0.50 & 0.75 & 0.450 & 0.406 & 0.201 & 0.212 & 0.216 & * 0.045 * & * 0.101 * * & 0.067 + & 0.05 & 1.00 & 0.015 & 0.020 & 0.024 & 0.020 & 0.025 & 0.021 & * 0.015 * & * 0.011 * * + & 0.25 & 1.00 & 0.040 & 0.025 & 0.018 & * 0.005 * & 0.022 & 0.005 & * 0.009 * * & 0.009 + & 0.50 & 1.00 & 0.094 & 0.027 & 0.022 & 0.005 & 0.028 & 0.006 & * 0.002 * * & * 0.002 * * + & 0.05 & 0.95 & 0.024 & 0.037 & * 0.019 * & 0.017 & 0.020 & 0.018 & 0.022 & * 0.012 * * + & 0.25 & 0.95 & 0.074 & 0.063 & 0.013 & 0.021 & 0.011 & 0.011 & * 0.009 * & * 0.009 * * + & 0.50 & 0.95 & 0.141 & 0.089 & 0.028 & 0.034 & 0.026 & 0.008 & * 0.002 * * & * 0.002 * * +    & 0.05 & 0.75 & 0.072 & 0.124 & * 0.008 * * & * 0.005 * * & 0.015 & 0.010 & 0.034 & 0.012 + & 0.25 & 0.75 & 0.244 & 0.258 & 0.076 & 0.095 & 0.033 & 0.009 & * 0.009 * * & * 0.008 * + & 0.50 & 0.75 & 0.389 & 0.355 & 0.128 & 0.173 & 0.106 & 0.014 & * 0.002 * * & * 0.002 * * +      to demonstrate the efficacy of the class prior preserving transform , we implemented the multivariate versions of alphamax - n  and msgmm   and evaluated them on the twelve real data sets without applying the transform .
there were significant stability and computational issues related to the high - dimensional nature of the data sets .
msgmm  was numerically unstable because of singular / nearly - singular covariance matrix , whereas alphamax - n  became computationally demanding because the number of bins ( for histogram based density estimation ) grow exponentially with the dimension , resulting in a large parameter vector @xmath379 and , consequently , a large optimization problem , even after removing the zero - count bins .
this is expected , as density estimation for multivariate data is known to be problematic , which is one of the main reasons for introducing our transform . to make estimation feasible under these stability and computational issues , we used dimensionality reduction .
though not all data sets posed the same level of difficulty , to have a standard approach and permit effective density estimation , we used the top three principal components , obtained via principal component analysis on the z - score normalized data ( mixture and component samples combined ) , as input to the two algorithms .
we also attempted using top @xmath384 principal components that preserve 75 percent of the total variance , however , for some of the data sets , the dimension was still too high .    in the same manner as in the univariate case , we used histograms in the multivariate implementation of alphamax - n .
the bin - width for a dimension was selected to minimize the asymptotic mean integrated squared error ( amise ) with a normal reference rule , using the component sample , @xmath147 .
the formula for the bin - width of dimension @xmath384 is given by : @xmath388 where @xmath167 is the total number of dimensions , @xmath389 is the standard deviation of the @xmath390 dimension and @xmath391 is the size of the component sample .
bins were added to cover the range of the entire data , mixture and component combined , and empty bins were removed to reduce the size of the optimization problem .
contains the results of alphamax - n   and msgmm  on the real - life data sets , using the top three principal components under column headings alphamax - nm  ( m for multi - dimensional ) and msgmm , respectively .
the results of alphamax - n   and msgmm - t  with the class - prior preserving transform are also provided for comparison .
notice that , though alphamax - nm  ( without transform ) performs well ,
alphamax - n  ( with transform ) is significantly better in terms of estimation error , despite having a lower computational cost .
also notice the deterioration in the performance of msgmm  ( without transform ) compared to msgmm - t  ( with transform ) .
1.01 |>x| > x| > x| > x| > x| > x| > x| > x| & @xmath19 & @xmath20 & % variance & & & & + & 0.095 & 1.00 & & 0.037 & * 0.028 * * & 0.163 & 0.528 + & 0.096 & 0.95 & 40 & 0.036 & * 0.032 * & 0.155 & 0.574 + & 0.101 & 0.75 & & * 0.040 * & 0.047 & 0.127 & 0.580 + & 0.419 & 1.00 & & 0.181 & 0.276 & 0.077 & * 0.020 * * + & 0.425 & 0.95 & 60 & 0.231 & 0.269 & 0.095 & * 0.028 * * + & 0.446 & 0.75 & & 0.272 & 0.320 & 0.233 & * 0.063 * * + & 0.342 & 1.00 & & 0.017 & 0.030 & * 0.008 * * & 0.585 + & 0.353 & 0.95 & 81 & * 0.006 * & 0.021 & 0.006 & 0.575 + & 0.397 & 0.75 & & 0.009 & 0.064 & * 0.006 * * & 0.533 + & 0.268 & 1.00 & & * 0.094 * * & 0.132 & 0.209 & 0.316 + & 0.281 & 0.95 & 69 & 0.110 & * 0.110 * & 0.204 & 0.308 + & 0.330 & 0.75 & & * 0.134 * & 0.205 & 0.172 & 0.283 + & 0.093 & 1.00 & & * 0.007 * & 0.008 & 0.157 & 0.443 + & 0.103 & 0.95 & 89 & * 0.008 * * & 0.028 & 0.152 & 0.298 + & 0.139 & 0.75 & & * 0.012 * * & 0.053 & 0.143 & 0.270
+ & 0.409 & 1.00 & & * 0.022 * * & 0.075 & 0.037 & 0.432 + & 0.416 & 0.95 & 24 & * 0.008 * * & 0.021 & 0.037 & 0.398 + & 0.444 & 0.75 & & * 0.020 * & 0.050 & 0.024 & 0.375 + & 0.086 & 1.00 & & * 0.044 * & 0.046 & 0.129 & 0.178 + & 0.087 & 0.95 & 72 & 0.052 & * 0.040 * * & 0.125 & 0.178 + & 0.090 & 0.75 & & 0.064 & * 0.031 * * & 0.111 & 0.188 + & 0.243 & 1.00 & & * 0.009 * * & 0.071 & 0.081 & 0.289 + & 0.248 & 0.95 & 65 & * 0.005 * * & 0.070 & 0.074 & 0.286 + & 0.268 & 0.75 & & * 0.007 * * & 0.092 & 0.062 & 0.260 + & 0.251 & 1.00 & & * 0.111 * & 0.123 & 0.171 & 0.299 + & 0.259 & 0.95 & 60 & * 0.110 * * & 0.156 & 0.168 & 0.292 + & 0.289 & 0.75 & & * 0.156 * & 0.178 & 0.175 & 0.286 + & 0.139 & 1.00 & & * 0.029 * * & 0.064 & 0.157 & 0.232 + & 0.140 & 0.95 & 67 & * 0.007 * * & 0.055 & 0.157 & 0.227 + & 0.143 & 0.75 & & * 0.004 * * & 0.015 & 0.148 & 0.356 + & 0.226 & 1.00 & & 0.041 & *
0.034 * & 0.059 & 0.487 + & 0.240 & 0.95 & 22 & 0.042 & * 0.041 * & 0.063 & 0.485 + & 0.295 & 0.75 & & * 0.044 * * & 0.072 & 0.059 & 0.434 + & 0.566 & 1.00 & & * 0.060 * & 0.258 & 0.070 & 0.134 + & 0.575 & 0.95 & 65 & * 0.063 * & 0.256 & 0.076 & 0.126 + & 0.612 & 0.75 & & 0.353 & 0.302 & 0.293 & * 0.096 * * +