mixture models are common tools in statistical pattern recognition @xcite .
they offer a mathematical basis to explain data in fields as diverse as astronomy , biology , ecology , engineering , and economics , amongst many others @xcite . a mixture model is composed of component probabilistic models ; a component may variously correspond to a subtype , kind , species , or subpopulation of the observed data .
these models aid in the identification of hidden patterns in the data through sound probabilistic formalisms .
mixture models have been extensively used in machine learning tasks such as classification and unsupervised learning @xcite .
formally , mixture modelling involves representing a distribution of data as a weighted sum of individual probability distributions .
specifically , the problem we consider here is to model the observed data using a mixture @xmath0 of probability distributions of the form : @xmath1 where @xmath2 is a @xmath3-dimensional datum , @xmath4 is the number of mixture components , @xmath5 and @xmath6 are the weight and probability density of the @xmath7 component respectively ; the weights are positive and sum to one .
the problem of modelling some observed data using a mixture distribution involves determining the number of components , @xmath4 , and estimating the mixture parameters . inferring the optimal number of mixture components involves the difficult problem of balancing the trade - off between two conflicting objectives : low _ hypothesis complexity _ as determined by the number of components _ and _ their respective parameters , versus good quality of _ fit _ to the observed data . generally a hypothesis with more free parameters
can fit observed data better than a hypothesis with fewer free parameters .
a number of strategies have been used to control this balance as discussed in section [ sec : mixture_existing_methods ] .
these methods provide varied formulations to assess the mixture components and their ability to explain the data .
methods using the minimum message length criterion @xcite , a bayesian method of inductive inference , have been proved to be effective in achieving a reliable balance between these conflicting aims @xcite .
although much of the literature concerns the theory and application of gaussian mixtures @xcite , mixture modelling using other probability distributions has been widely used .
some examples are : poisson models @xcite , exponential mixtures @xcite , laplace @xcite , t - distribution @xcite , weibull @xcite , kent @xcite , von mises - fisher @xcite , and many more .
@xcite provide a comprehensive summary of finite mixture models and their variations .
the use of gaussian mixtures in several research disciplines has been partly motivated by its computational tractability @xcite . for datasets where the _ direction _ of the constituent vectors is important ,
gaussian mixtures are inappropriate and distributions such as the von mises - fisher may be used @xcite . in any case
, whatever the kind of distribution used for an individual component , one needs to estimate the parameters of the mixture , and provide a sound justification for selecting the appropriate number of mixture components .
software for mixture modelling relies on the following elements :    1 .
an _ estimator _ of the parameters of each component of a mixture , 2 .   an _ objective function
_ , that is a cost or score , that can be used to compare two hypothetical mixtures and decide which is better , and 3 .   a _ search strategy _ for the best number of components and their weights .
traditionally , parameter estimation is done by strategies such as maximum likelihood ( ml ) or bayesian maximum _ a posteriori _ probability ( map ) estimation . in this work ,
we use the bayesian minimum message length ( mml ) principle . unlike map ,
mml estimators are invariant under non - linear transformations of the data @xcite , and unlike ml , mml considers the number and precision of a model s parameters .
it has been used in the inference of several probability distributions @xcite .
mml - based inference operates by considering the problem as encoding first the parameter estimates and then the data given those estimates . the parameter values that result in the least _ overall _ message length to explain the whole data are taken as the mml estimates for an inference problem .
the mml scheme thus incorporates the cost of stating parameters into model selection .
it is self evident that a continuous parameter value can only be stated to some finite precision ; the cost of encoding a parameter is determined by its prior and the precision .
ml estimation ignores the cost of stating a parameter and map based estimation uses the probability _ density _ of a parameter instead of its probability measure .
in contrast , the mml inference process calculates the optimal precision to which parameters should be stated and a probability value of the corresponding parameters is then computed .
this is used in the computation of the message length corresponding to the parameter estimates .
thus , models with varying parameters are evaluated based on their resultant total message lengths .
we use this characteristic property of mml to evaluate mixtures containing different numbers of components .
although there have been several attempts to address the challenges of mixture modelling , the existing methods are shown to have some limitations in their formalisms ( see section [ sec : mixture_existing_methods ] ) .
in particular , some methods based on mml are incomplete in their formulation .
we aim to rectify these drawbacks by proposing a comprehensive mml formulation and develop a search heuristic that selects the number of mixture components based on the proposed formulation . to demonstrate the effectiveness of the proposed search and parameter estimation , we first consider modelling problems using gaussian mixtures and extend the work to include relevant discussion on mixture modelling of directional data using von mises - fisher distributions .
the importance of gaussian mixtures in practical applications is well established . for a _ given _
number of components , the conventional method of estimating the parameters of a mixture relies on the expectation - maximization ( em ) algorithm @xcite .
the standard em is a local optimization method , is sensitive to initialization , and in certain cases may converge to the boundary of parameter space @xcite .
previous attempts to infer gaussian mixtures based on the mml framework have been undertaken using simplifying assumptions , such as the covariance matrices being diagonal @xcite , or coarsely approximating the probabilities of mixture parameters @xcite .
further , the search heuristic adopted in some of these methods is to run the em several times for different numbers of components , @xmath4 , and select the @xmath4 with the best em outcome @xcite . a search method based on iteratively _ deleting _ components has been proposed by @xcite .
it begins by assuming a very large number of components and selectively eliminates components deemed redundant ; there is no provision for recovering from deleting a component in error .    in this work ,
we propose a search method which selectively _ splits , deletes _ , or _ merges _ components depending on improvement to the mml objective function .
the operations , combined with em steps , result in a sensible redistribution of data between the mixture components . as an example
, a component may be split into two children , and at a later stage , one of the children may be merged with another component . unlike the method of @xcite ,
our method starts with a one - component mixture and alters the number of components in subsequent iterations .
this avoids the overhead of dealing with a large number of components unless required .
the proposed search heuristic can be used with probability distributions for which the mml expressions to calculate message lengths for estimates and for data given those estimates are known . as an instance of this , section [ sec : search_method ] discusses modelling directional data using the von mises - fisher distributions .
directional statistics have garnered support recently in real data mining problems where the _ direction _ , not magnitude , of vectors is significant .
examples of such scenarios are found in earth sciences , meteorology , physics , biology , and other fields @xcite .
the statistical properties of directional data have been studied using several types of distributions @xcite , often described on surfaces of compact manifolds , such as the sphere , ellipsoid , torus _
etc_. the most fundamental of these is the von mises - fisher ( vmf ) distribution which is analogous to a symmetric multivariate gaussian distribution , wrapped around a unit hypersphere @xcite .
the probability density function of a vmf distribution with parameters @xmath8 ( mean direction , concentration parameter ) for a random unit vector @xmath9 on a @xmath10- dimensional hypersphere @xmath11 is given by : @xmath12 where @xmath13 is the normalization constant and @xmath14 is a modified bessel function of the first kind and order @xmath15 .
the estimation of the parameters of the vmf distribution is often done using maximum likelihood .
however , the complex nature of the mathematical form presents difficulty in estimating the concentration parameter @xmath16 .
this has lead to researchers using many different approximations , as discussed in section  [ sec : vmf_existing_methods ] .
most of these methods perform well when the amount of data is large . at smaller sample sizes
, they result in inaccurate estimates of @xmath16 , and are thus unreliable .
we demonstrate this by the experiments conducted on a range of sample sizes .
the problem is particularly evident when the dimensionality of the data is large , also affecting the application in which it is used , such as mixture modelling .
we aim to rectify this issue by using mml estimates for @xmath16 .
our experiments section demonstrates that the mml estimate of @xmath16 provides a more reliable answer and is an improvement on the current state of the art .
these mml estimates are subsequently used in mixture modelling of vmf distributions ( see sections [ sec : search_method ] and [ sec : vmf_applications ] ) .
previous studies have established the importance of von mises circular ( two - dimensional ) and von mises - fisher ( three - dimensional and higher ) mixtures , and demonstrated applications to clustering of protein dihedral angles @xcite , large - scale text clustering @xcite , and gene expression analyses @xcite .
the merit of using cosine based similarity metrics , which are closely related to the vmf , for clustering high dimensional text data has been investigated in @xcite . for text clustering
, there is evidence that vmf mixture models have a superior performance compared to other statistical distributions such as normal , multinomial , and bernoulli @xcite .
movmf is a widely used package to perform clustering using vmf distributions @xcite . +
* contributions : * the main contributions of this paper are as follows :    * we derive the analytical estimates of the parameters of a multivariate gaussian distribution with full covariance matrix , using the mml principle @xcite .
* we derive the expression to infer the concentration parameter @xmath16 of a generic @xmath3-dimensional vmf distribution using mml - based estimation .
we demonstrate , through a series of experiments , that this estimate outperforms the previous ones , therefore making it a reliable candidate to be used in mixture modelling .
* a generalized mml - based search heuristic is proposed to infer the optimal number of mixture components that would best explain the observed data ; it is based on the search used in various versions of the snob classification program @xcite .
we compare it with the work of @xcite and demonstrate its effectiveness .
* the search implements a generic approach to mixture modelling and allows , in this instance , the use of @xmath3-dimensional gaussian and vmf distributions under the mml framework .
it infers the optimal number of mixture components , and their corresponding parameters . *
further , we demonstrate the effectiveness of mml mixture modelling through its application to high dimensional text clustering and clustering of directional data that arises out of protein conformations .    the rest of the paper is organized as follows : sections [ sec : gaussian_estimates ] and [ sec : vmf_existing_methods ] describe the respective estimators of gaussian and vmf distributions that are commonly used .
section [ sec : mml_framework ] introduces the mml framework for parameter estimation .
section [ sec : mml_est_derivations ] outlines the derivation of the mml parameter estimates of multivariate gaussian and vmf distributions .
section [ sec : mml_mixture_modelling ] describes the formulation of a mixture model using mml and the estimation of the mixture parameters under the framework . section  [ sec : mixture_existing_methods ] reviews the existing methods for selecting the mixture components .
section [ sec : search_method ] describes our proposed approach to determine the number of mixture components .
section [ sec : gaussian_experiments ] depicts the competitive performance of the proposed mml - based search through experiments conducted with gaussian mixtures .
section [ sec : vmf_experiments ] presents the results for mml - based vmf parameter estimation and mixture modelling followed by results supporting its applications to text clustering and protein structural data in section [ sec : vmf_applications ] .
the probability density function @xmath17 of a @xmath3-variate gaussian distribution is given as @xmath18 where @xmath19 , @xmath20 are the respective mean , ( symmetric ) covariance matrix of the distribution , and @xmath21 is the determinant of the covariance matrix . the traditional method to estimate the parameters of a gaussian distribution is by maximum likelihood . given data @xmath22 , where @xmath23 , the log - likelihood @xmath24 is given by @xmath25 to compute the maximum likelihood estimates , equation needs to be _ maximized_. this is achieved by computing the gradient of the log - likelihood function with respect to the parameters and solving the resultant equations .
the _ gradient vector _ of @xmath24 with respect to @xmath19 and the _ gradient matrix _ of @xmath24 with respect to @xmath20 are given below .
@xmath26 the maximum likelihood estimates are then computed by solving @xmath27 and @xmath28 and are given as : @xmath29    @xmath30 is known to be a biased estimate of the covariance matrix @xcite and issues related with its use in mixture modelling have been documented in @xcite and @xcite .
an unbiased estimator of @xmath20 was proposed by @xcite and is given below .
@xmath31    in addition to the maximum likelihood estimates ,
bayesian inference of gaussian parameters involving conjugate priors over the parameters has also been dealt with in the literature @xcite .
however , the unbiased estimate of the covariance matrix , as determined by the sample covariance ( equation ) , is typically used in the analysis of gaussian distributions .
for a von mises - fisher ( vmf ) distribution @xmath17 characterized by equation , and given data @xmath22 , such that @xmath32 , the log - likelihood @xmath24 is given by @xmath33 where @xmath34 is the sample size and @xmath35 ( the vector sum ) .
let @xmath36 denote the magnitude of the resultant vector @xmath37 and let @xmath38 and @xmath39 be the maximum likelihood estimators of @xmath40 and @xmath16 respectively . under the condition that @xmath38 is a unit vector , the maximum likelihood estimates are obtained by maximizing @xmath24 as follows : @xmath41 solving the non - linear equation : @xmath42 yields the corresponding maximum likelihood estimate where @xmath43 represents the ratio of bessel functions . because of the difficulties in analytically solving equation ( [ eqn : ml_estimates ] ) , there have been several approaches to approximating @xmath39 @xcite .
each of these methods is an improvement over their respective predecessors .
@xcite is an improvement over the estimate proposed by @xcite .
@xcite is an improvement over @xcite and @xcite fares better when compared to @xcite .
the methods are summarized below .
the approximation given by equation ( [ eqn : banerjee_approx ] ) is due to @xcite and provides an easy to use expression for @xmath39 .
the formula is very appealing as it eliminates the need to evaluate complex bessel functions .
@xcite demonstrated that this approximation yields better results compared to the ones suggested in @xcite .
it is an empirical approximation which can be used as a starting point in estimating the root of equation ( [ eqn : ml_estimates ] ) . @xmath44
the approximation given by equation ( [ eqn : tanabe_approx ] ) is due to @xcite .
the method utilizes the properties of bessel functions to determine the lower and upper bounds for @xmath39 and uses a fixed point iteration function in conjunction with linear interpolation to approximate @xmath39 .
the bounds for @xmath39 are given by @xmath45 @xcite proposed to use a fixed point iteration function defined as @xmath46 and used this to approximate @xmath39 as @xmath47      this a heuristic approximation provided by @xcite .
it involves refining the approximation given by @xcite ( equation ( [ eqn : banerjee_approx ] ) ) by performing two iterations of newton s method .
@xcite demonstrate that this approximation fares well when compared to the approximation proposed by @xcite .
the following two iterations result in @xmath48 , the approximation proposed by @xcite : @xmath49 @xmath50      this approximation provided by @xcite uses halley s method which is the second order expansion of taylor s series of a given function @xmath51 . the higher order approximation results in a more accurate estimate as demonstrated by @xcite .
the iterative halley s method is truncated after iterating through two steps of the root finding algorithm ( similar to that done by @xcite ) .
the following two iterations result in @xmath52 , the approximation proposed by @xcite : @xmath53 @xmath54    the common theme in all these methods is that they try to approximate the maximum likelihood estimate governed by equation ( [ eqn : ml_estimates ] ) .
it is to be noted that the maximum likelihood estimators ( of concentration parameter @xmath16 ) have considerable bias @xcite . to counter this effect
, we explore the minimum message length based estimation procedure .
this bayesian method of estimation not only results in an unbiased estimate but also provides a framework to choose from several competing models @xcite . through a series of empirical tests ,
we demonstrate that the mml estimate is more reliable than any of the contemporary methods . @xcite
have demonstrated the superior performance of the mml estimate for a three - dimensional vmf distribution .
we extend their work to derive the mml estimators for a generic @xmath3-dimensional vmf distribution and compare its performance with the existing methods .
@xcite developed the first practical criterion for model selection to be based on information theory .
the resulting framework provides a rigorous means to objectively compare two competing hypotheses and , hence , choose the best one .
as per bayes s theorem , @xmath55 where @xmath56 denotes some observed data , and @xmath57 some hypothesis about that data .
further , @xmath58 is the joint probability of data @xmath56 and hypothesis @xmath57 , @xmath59 is the prior probability of hypothesis @xmath57 , @xmath60 is the prior probability of data @xmath56 , @xmath61 is the posterior probability of @xmath57 given @xmath56 , and @xmath62 is the likelihood .
mml uses the following result from information theory @xcite : given an event or outcome @xmath63 whose probability is @xmath64 , the length of the optimal lossless code @xmath65 to represent that event requires @xmath66 bits . applying shannon s insight to bayes s theorem , @xcite got the following relationship between conditional probabilities in terms of optimal message lengths : @xmath67 as a result , given two competing hypotheses @xmath57 and @xmath68 , @xmath69 @xmath70 gives the posterior log - odds ratio between the two competing hypotheses . equation ( [ eqn : shannon_bayes_msg ] ) can be intrepreted as the _ total _ cost to encode a message comprising the hypothesis @xmath57 and data @xmath56 . this message is composed over into two parts :    1 .   _ first part : _ the hypothesis @xmath57 , which takes @xmath71 bits , 2 .   _
second part : _ the observed data @xmath56 using knowledge of @xmath57 , which takes @xmath72 bits .
clearly , the message length can vary depending on the complexity of @xmath57 and how well it can explain @xmath56 .
a more complex @xmath57 may fit ( i.e. , explain ) @xmath56 better but take more bits to be stated itself .
the trade - off comes from the fact that ( hypothetically ) transmitting the message requires the encoding of both the hypothesis and the data given the hypothesis , that is , the model complexity @xmath71 and the goodness of fit @xmath72 .      our proposed method of parameter estimation uses the mml inference paradigm .
it is a bayesian method which has been applied to infer the parameters of several statistical distributions @xcite .
we apply it to infer the parameter estimates of multivariate gaussian and vmf distributions .
@xcite introduced a generalized scheme to estimate a vector of parameters @xmath73 of any distribution @xmath17 given data @xmath56 .
the method involves choosing a reasonable prior @xmath74 on the hypothesis and evaluating the _ determinant _ of the fisher information matrix @xmath75 of the _ expected _ second - order partial derivatives of the negative log - likelihood function , @xmath76 .
the parameter vector @xmath73 that minimizes the message length expression ( equation ( [ eqn : two_part_msg ] ) ) is the mml estimate according to @xcite .
@xmath77 where @xmath78 is the number of free parameters in the model , and @xmath79 is the lattice quantization constant @xcite in @xmath78-dimensional space .
the total message length @xmath80 in mml framework is composed of two parts :    1 .
the statement cost of encoding the parameters , @xmath81 and 2 .
the cost of encoding the data given the parameters , @xmath82 .
a concise description of the mml method is presented in @xcite .
we note here a few key differences between mml and ml / map based estimation methods . in maximum likelihood estimation ,
the statement cost of parameters is ignored , in effect considered constant , and minimizing the message length corresponds to minimizing the negative log - likelihood of the data ( the second part ) . in map based estimation , a probability _ density _ rather than the probability measure is used .
continuous parameters can necessarily only be stated only to finite precision .
mml incorporates this in the framework by determining the region of uncertainty in which the parameter is located .
the value of @xmath83 gives a measure of the volume of the region of uncertainty in which the parameter @xmath73 is centered .
this multiplied by the probability density @xmath74 gives the _ probability _ of a particular @xmath73 and is _ proportional _ to @xmath84 .
this probability is used to compute the message length associated with encoding the continuous valued parameters ( to a finite precision ) .
based on the mml inference process discussed in section [ sec : mml_framework ] , we now proceed to formulate the message length expressions and derive the parameter estimates of gaussian and von mises - fisher distributions .
the mml framework requires the statement of parameters to a finite precision .
the optimal precision is related to the fisher information and in conjunction with a reasonable prior , the probability of parameters is computed .      a flat prior is usually chosen on each of the @xmath3 dimensions of @xmath19 @xcite and a conjugate inverted wishart prior
is chosen for the covariance matrix @xmath20 @xcite .
the joint prior density of the parameters is then given as @xmath85      the computation of the fisher information requires the evaluation of the second order partial derivatives of @xmath86 .
let @xmath87 represent the determinant of the fisher information matrix .
this is approximated as the product of @xmath88 and @xmath89 @xcite , where @xmath88 and @xmath89 are the respective determinants of fisher information matrices due to the parameters @xmath19 and @xmath20 . differentiating the gradient vector in equation with respect to @xmath19
, we have : @xmath90 to compute @xmath89 , @xcite derived an analytical expression using the theory of matrix derivatives based on matrix vectorization @xcite .
let @xmath91\ , \forall 1 \leq i , j \leq d$ ] where @xmath92 denotes the element corresponding to the @xmath93 row and @xmath94 column of the covariance matrix .
let @xmath95 be the vector containing the @xmath96 free parameters that completely describe the symmetric matrix @xmath20 .
then , the fisher information due to the vector of parameters @xmath97 is equal to @xmath89 and is given by equation @xcite .
@xmath98 multiplying equations and , we have @xmath99      to derive the message length expression to encode data using a certain @xmath100 , substitute equations , , and , in equation using the number of free parameters of the distribution as @xmath101 . hence , @xmath102    to obtain the mml estimates of @xmath19 and @xmath20 , equation needs to be minimized .
the mml estimate of @xmath19 is same as the maximum likelihood estimate ( given in equation ) . to compute the mml estimate of @xmath20
, we need to compute the gradient matrix of @xmath103 with respect to @xmath20 and is given by equation @xmath104 the mml estimate of @xmath20 is obtained by solving @xmath105 ( given in equation ) .
@xmath106 we observe that the mml estimate @xmath107 is same as the _ unbiased _ estimate of the covariance matrix @xmath20 , thus , lending credibility for its preference over the traditional ml estimate ( equation ) .
parameter estimates for two and three - dimensional vmf have been explored previously @xcite .
mml estimators of three - dimensional vmf were explored in @xcite , where they demonstrate that the mml - based inference compares favourably against the traditional ml and map based estimation methods .
we use the @xcite method to formulate the objective function ( equation ) corresponding to a generic vmf distribution .      regarding choosing a reasonable prior ( in the absence of any supporting evidence ) for the parameters @xmath108 of a vmf distribution , @xcite and @xcite suggest the use of the following  _ colourless _ prior that is uniform in direction , normalizable and locally uniform at the cartesian origin in @xmath16 " :
@xmath109      regarding evaluating the fisher information , @xcite argue that in the general @xmath3-dimensional case , @xmath110 where @xmath111 and @xmath112 are described by equations and respectively .      substituting equations , and in equation with number of free parameters @xmath113
, we have the net message length expression : @xmath114 to obtain the mml estimates of @xmath19 and @xmath16 , equation needs to be minimized . the estimate for @xmath19
is same as the maximum likelihood estimate ( equation ) . the resultant equation in @xmath16 that needs to be minimized is then given by : @xmath115 to obtain the mml estimate of @xmath16
, we need to differentiate equation ( [ eqn : i_kappa ] ) and set it to zero . @xmath116
the non - linear equation : @xmath117 does not have a closed form solution .
we try both the newton and halley s method to find an approximate solution .
we discuss both variants and comment on the effects of the two approximations in the experimental results . to be fair and consistent with @xcite and @xcite , we use the initial guess of the root as @xmath118 ( equation ( [ eqn : banerjee_approx ] ) ) and iterate twice to obtain the mml estimate .    1 .
_ approximation using newton s method : _
@xmath119 2 .
_ approximation using halley s method : _ @xmath120 the details of evaluating @xmath121 and @xmath122 are discussed in appendix  [ subsec : appendix_derivations ] .
equation ( [ eqn : mml_newton_approx ] ) gives the mml estimate ( @xmath123 ) using newton s method and equation ( [ eqn : mml_halley_approx ] ) gives the mml estimate ( @xmath124 ) using halley s method .
we used these values of mml estimates in mixture modelling using vmf distributions .
[ sec : mml_gaussian_est ]
mixture modelling involves representing an observed distribution of data as a weighted sum of individual probability density functions . specifically , the problem we consider here is to model the mixture distribution @xmath0 as defined in equation . for some observed data @xmath125 ( @xmath34 is the sample size ) , and a mixture @xmath0 , the log - likelihood using the mixture distribution is as follows : @xmath126 where @xmath127 , @xmath5 and @xmath6 are the weight and probability density of the @xmath7 component respectively . for a fixed @xmath4 ,
the mixture parameters @xmath128 are traditionally estimated using a standard _ expectation - maximization_(em ) algorithm @xcite .
this is briefly discussed below .
the standard em algorithm is based on maximizing the log - likelihood function of the data ( equation ) .
the maximum likelihood estimates are then given as @xmath129 .
because of the absence of a closed form solution for @xmath130 , a gradient descent method is employed where the parameter estimates are iteratively updated until convergence to some local optimum is achieved @xcite .
the em method consists of two steps :    * _ e - step _ : each datum @xmath131 has fractional membership to each of the mixture components .
these partial memberships of the data points to each of the components are defined using the _ responsibility matrix _
@xmath132 where @xmath133 denotes the conditional probability of a datum @xmath131 belonging to the @xmath7 component . the effective membership associated with each component is then given by @xmath134 * _ m - step _ : assuming @xmath135 be the estimates at some iteration @xmath136
, the expectation of the log - likelihood using @xmath135 and the partial memberships is then _ maximized _ which is tantamount to computing @xmath137 , the updated maximum likelihood estimates for the next iteration @xmath138 .
the weights are updated as @xmath139 .
the above sequence of steps are repeated until a certain convergence criterion is satisfied . at some intermediate iteration @xmath136 ,
the mixture parameters are updated using the corresponding ml estimates and are given below .    * _ gaussian : _ the ml updates of the mean and covariance matrix are @xmath140 * _ von mises - fisher : _ the resultant vector sum is updated as @xmath141 .
if @xmath142 represents the magnitude of vector @xmath143 , then the updated mean and concentration parameter are @xmath144      we will first describe the methodology involved in formulating the mml - based objective function .
we will then discuss how em is applied in this context .
we refer to the discussion in  @xcite to briefly describe the intuition behind mixture modelling using mml .
encoding of a message using mml requires the encoding of ( 1 ) the model parameters and then ( 2 ) the data using the parameters .
the statement costs for encoding the mixture model and the data can be decomposed into :    1 .
encoding the _ number of components _ @xmath4 : in order to encode the message losslessly , it is required to initially state the number of components . in the absence of background knowledge
, one would like to model the prior belief in such a way that the probability decreases for increasing number of components . if @xmath145 , then @xmath146 .
the prior reflects that there is a difference of one bit in encoding the _ numbers _ @xmath4 and @xmath147 .
alternatively , one could assume a uniform prior over @xmath4 within some predefined range .
the chosen prior has little effect as its contribution is minimal when compared to the magnitude of the total message length @xcite .
2 .   encoding the _ weights _ @xmath148 which are treated as parameters of a multinomial distribution with sample size @xmath149 .
the length of encoding all the weights is then given by the expression @xcite : + @xmath150 3 .
encoding each of the _ component parameters _ @xmath151 as given by @xmath152 ( discussed in section  [ subsec : mml_parameter_estimation ] ) .
4 .   encoding the _ data _ : each datum @xmath131 can be stated to a finite precision which is dictated by the accuracy of measurement is a constant value and has no effect on the overall inference process .
it is used in order to maintain the theoretical validity when making the distinction between _ probability _ and _ probability density_. ] .
if the precision to which each element of a @xmath3-dimensional vector can be stated is @xmath153 , then the _ probability _ of a datum @xmath154 is given as @xmath155 where @xmath156 is the _ probability density _ given by equation  .
hence , the _ total _ length of its encoding is given by @xmath157 the entire data @xmath56 can now be encoded as : @xmath158    thus , the total message length of a @xmath4 component mixture is given by equation  .
@xmath159    note that the _ constant _ term includes the lattice quantization constant ( resulting from stating all the model parameters ) in a @xmath78-dimensional space , where @xmath78 is equal to the number of free parameters in the mixture model .
the parameters of the mixture model are those that _
minimize _ equation . to achieve this we use the standard em algorithm ( section [ subsec : em_ml ] ) , where , iteratively , the parameters are updated using their respective _
mml estimates_. the component weights are obtained by differentiating equation with respect to @xmath5 under the constraint @xmath160 .
the derivation of the mml updates of the weights is shown in appendix  [ subsec : wts_mml ] and are given as : @xmath161    the parameters of the @xmath7 component are updated using @xmath162 and @xmath163 ( equations , ) , the partial memberships assigned to the @xmath7 component at some intermediate iteration @xmath136 and and are given below .    * _ gaussian : _ the mml updates of the mean and covariance matrix are @xmath164 * _ von mises - fisher : _ the resultant vector sum is updated as @xmath165 . if @xmath142 represents the magnitude of vector @xmath143 , then the updated mean is given by equation . @xmath166 the mml update of the concentration parameter @xmath167 is obtained by solving @xmath168 after substituting @xmath169 and @xmath170 in equation .
the em is terminated when the change in the total message length ( improvement rate ) between successive iterations is less than some predefined threshold .
the difference between the two variants of standard em discussed above is firstly the objective function that is being optimized . in section  [ subsec : em_ml ] , the log - likelihood function is _ maximized _ which corresponds to @xmath171 term in section  [ subsec : em_mml ] .
equation ( [ eqn : mixture_msglen ] ) includes additional terms that correspond to the cost associated with stating the mixture parameters .
secondly , in the m - step , in section  [ subsec : em_ml ] , the components are updated using their ml estimates whereas in section  [ subsec : em_mml ] , the components are updated using their mml estimates .
the standard em algorithms outlined above can be used only when the number of mixture components @xmath4 is fixed or known _ a priori_. even when the number of components are fixed , em has potential pitfalls .
the method is sensitive to the initialization conditions . to overcome this
, some reasonable start state for the em may be determined by initially clustering the data @xcite .
another strategy is to run the em a few times and choose the best amongst all the trials .
@xcite point out that , in the case of gaussian mixture modelling , em can converge to the boundary of the parameter space when the corresponding covariance matrix is nearly singular or when there are few initial members assigned to that component .
inferring the  right " number of mixture components for unlabelled data has proven to be a thorny issue @xcite and there have been numerous approaches proposed that attempt to tackle this problem @xcite . given some observed data , there are infinitely many mixtures that one can fit to the data .
any method that aims to selectively determine the optimal number of components should be able to factor the cost associated with the mixture parameters . to this end ,
several methods based on information theory have been proposed where there is some form of penalty associated with choosing a certain parameter value @xcite .
we briefly review some of these methods and discuss the state of the art and then proceed to explain our proposed method .
aic in the simplest form adds the _ number _ of free parameters @xmath78 to the negative log - likelihood expression .
there are some variants of aic suggested @xcite .
however , these variants introduce the same penalty constants for each additional parameter : @xmath172 bic , similar to aic , adds a constant multiple of @xmath173 ( @xmath34 being the sample size ) , for each free parameter in the model .
@xcite formulated minimum description length ( mdl ) which formally coincides with bic @xcite .
aic and bic / mdl serve as scoring functions to evaluate a model and its corresponding fit to the data .
the formulations suggest that the parameter cost associated with adopting a model is dependent only on the number of free parameters and _ not _ on the parameter values themselves . in other words ,
the criteria consider all models of a particular type ( of probability distribution ) to have the same statement cost associated with the parameters .
for example , a generic @xmath3-dimensional gaussian distribution has @xmath175 free parameters .
all such distributions will have the same parameter costs regardless of their characterizing means and covariance matrices , which is an oversimplifying assumption which can hinder proper inference .
the criteria can be interpreted under the mml framework wherein the first part of the message is a constant multiplied by the number of free parameters .
aic and bic formulations can be obtained as approximations to the two - part mml formulation governed by equation @xcite .
it has been argued that for tasks such as mixture modelling , where the number of free parameters potentially grows in proportion to the data , mml is known in theory to give consistent results as compared to aic and bic @xcite .      to determine the optimal number of mixture components @xmath4 , the aic or bic scores are computed for mixtures with varying values of @xmath4 . the mixture model with the least score
is selected as per these criteria .
a @xmath3-variate gaussian mixture with @xmath4 number of components has @xmath176 free parameters .
all mixtures with a set number of components have the same cost associated with their parameters using these criteria .
the mixture complexity is therefore treated as independent of the constituent mixture parameters .
in contrast , the mml formulation incorporates the statement cost of losslessly encoding mixture parameters by calculating their relevant probabilities as discussed in section [ sec : mml_mixture_modelling ] .
a mml - based scoring function akin to the one shown in equation was used to model gaussian mixtures .
however , the authors only consider the specific case of gaussians with diagonal covariance matrices , and fail to provide a general method dealing with full covariance matrices .      a rigorous treatment on the selection of number of mixture components @xmath4 is lacking .
@xcite experiment with different values of @xmath4 and choose the one which results in the minimum message length . for each @xmath4 ,
the standard em algorithm ( section  [ subsec : em_ml ] ) was used to attain local convergence .
the method , also referred to as _ laplace - empirical criterion _
( lec ) @xcite , uses a scoring function derived using bayesian inference and serves to provide a tradeoff between model complexity and the quality of fit .
the parameter estimates @xmath128 are those that result in the minimum value of the following scoring function . @xmath177 where @xmath56 is the dataset
, @xmath178 is the negative log - likelihood given the mixture parameters @xmath128 , @xmath4 is the number of mixture components , @xmath3 the dimensionality of the data , @xmath179 are hyperparameters ( which are set to 1 in their experiments ) , @xmath180 is a pre - defined constant or is pre - computed using the entire data , @xmath181 is the hessian matrix which is equivalent to the empirical fisher matrix for the set of component parameters , and @xmath78 is the number of free parameters in the model .
the formulation in equation can be obtained as an approximation to the message length expression in equation by identifying the following related terms in both equations .    1 .
@xmath182 2 .   for a @xmath3-variate gaussian with mean
@xmath19 and covariance matrix @xmath20 , the joint prior @xmath183 is calculated as follows : * _ prior on @xmath19 : _ each of the @xmath3 parameters of the mean direction are assumed to be have uniform priors in the range @xmath184 , so that the prior density of the mean is @xmath185 .
* _ prior on @xmath20 : _ it is assumed that the prior density is dependent only on the diagonal elements in @xmath20 .
each diagonal covariance element is assumed to have a prior in the range @xmath186 so that the prior on @xmath20 is considered to be @xmath187 + the joint prior , is therefore , assumed to be @xmath188 .
+ thus , @xmath189 + 3 .   @xmath190 + 4 .
@xmath191    although the formulation is an improvement over the previously discussed methods , there are some limitations due to the assumptions made while proposing the scoring function :    * while computing the prior density of the covariance matrix , the off - diagonal elements are ignored . *
the computation of the determinant of the fisher matrix is approximated by computing the hessain @xmath192 .
it is to be noted that while the hessian is the _ observed information _ ( data dependent ) , the fisher information is the _ expectation _ of the observed information .
mml formulation requires the use of the expected value .
* further , the approximated hessian was derived for gaussians with diagonal covariances . for gaussians with full covariance matrices ,
the hessian was approximated by replacing the diagonal elements with the corresponding eigen values in the hessian expression .
the empirical fisher computed in this form does not guarantee the charactersitic invariance property of the classic mml method @xcite .
the search method used to select the optimal number of components is rudimentary .
the optimal number of mixture components is chosen by running the em 10 times for every value of @xmath4 within a given range .
an optimal @xmath4 is selected as the one for which the best of the 10 trials results in the least value of the scoring function .
the icl criterion _ maximizes _ the _ complete log - likelihood _ ( cl ) given by @xmath193 where @xmath194 is the log - likelihood ( equation ) , @xmath133 is the responsibility term ( equation  ) , and @xmath195 if @xmath131 arises from component @xmath196 and zero otherwise .
the term @xmath197 is explained as the estimated mean entropy .
the icl criterion is then defined as : @xmath198where @xmath78 is the number of free parameters in the model .
we observe that similar to bic , the icl scoring function penalizes each free parameter by a constant value and does not account for the model parameters .
the search method adopted in this work is similar to the one used by @xcite .
the em algorithm is initiated 20 times for each value of @xmath4 with random starting points and the best amongst those is chosen .
the method uses the mml criterion to formulate the scoring function given by equation .
the formulation can be intrepreted as a two - part message for encoding the model parameters and the observed data .
@xmath199 where @xmath200 is the _ number _ of free parameters per component and @xmath5 is the component weight .
the scoring function is derived from equation by assuming the prior density of the component parameters to be a jeffreys prior . if @xmath151 is the vector of parameters describing the @xmath7 component , then the prior density @xmath201 @xcite .
similarly , a prior for weights would result in @xmath202 .
these assumptions are used in the encoding of the parameters which correspond to the first part of the message .
we note that the scoring function is consistent with the mml scheme of encoding parameters and the data using those parameters .
however , the formulation can be improved by amending the assumptions as detailed in in section [ sec : mml_est_derivations ] .
further , the assumptions made in @xcite have the following side effects :    * the value of @xmath203 gives the cost of encoding the component parameters .
by assuming @xmath201 , the message length associated with using any vector of parameters @xmath151 is essentially treated the same . to avoid this ,
the use of independent uniform priors over non - informative jeffreys s priors was advocated previously @xcite .
the use of jeffreys prior offers certain advantages , for example , not having to compute the fisher information @xcite .
however , this is crucial and can not be ignored as it dictates the _ precision of encoding the parameter vector_. @xcite states that
jeffreys , while noting the interesting properties of the prior formulation did not advocate its use as a genuine expression of prior knowledge . "
+ by making this assumption , @xcite  _ sidestep _ " the difficulty associated with explicitly computing the fisher information associated with the component parameters . hence , for encoding the parameters of the entire mixture , _ only _ the cost associated with encoding the component weights is considered . *
the code length to state each @xmath151 is , therefore , greatly simplified as @xmath204 ( notice the sole dependence on weight @xmath5 ) .
@xcite interpret this as being similar to a mdl formulation because @xmath205 gives the expected number of data points generated by the @xmath7 component .
this is equivalent to the bic criterion discussed earlier .
we note that mdl / bic are highly simplified versions of mml formulation and therefore , equation does not capture the entire essence of complexity and goodness of fit accurately .
the method begins by assuming a large number of components and updates the weights iteratively in the em steps as @xmath206 where @xmath207 is the effective membership of data points in @xmath7 component ( equation ) .
a component is annihilated when its weight becomes zero and consequently the number of mixture components decreases .
we note that the search method proposed by @xcite using the mml criterion is an improvement over the methods they compare against .
however , we make the following remarks about their search method .
* the method updates the weights as given by equation . during any iteration ,
if the amount of data allocated to a component is less than @xmath208 , its weight is updated as zero and this component is ignored in subsequent iterations .
this imposes a lower bound on the amount of data that can be assigned to each component . as an example , for a gaussian mixture in 10-dimensions , the number of free parameters per component is @xmath209 , and hence the lower bound is 33 .
hence , in this exampe , if a component has @xmath210 data , the mixture size is reduced and these data are assigned to some other component(s ) .
consider a scenario where there are 50 observed 10 dimensional data points originally generated by a mixture with two components with equal mixing proportions .
the method would always infer that there is only one component regardless of the separation between the two components .
this is clearly a wrong inference ! ( see section [ subsec : fj_weight_updates_exp2b ] for the relevant experiments ) .
* once a component is discarded , the mixture size decreases by one , and it can not be recovered . because the memberships @xmath207 are updated iteratively using an em algorithm and because em might not always lead to global optimum , it is conceivable that the updated values need not always be optimal .
this might lead to situations where a component is deleted owing to its low prominence .
there is no provision to increase the mixture size in the subsequent stages of the algorithm to account for such behaviour . *
the method assumes a large number of initial components in an attempt to be robust with respect to em initialization . however , this places a significant overhead on the computation due to handling several components .
_ summary : _ we observe that while all these methods ( and many more ) work well within their defined scope , they are incomplete in achieving the true objective that is to rigorously score models and their ability to fit the data . the methods discussed above can be seen as different approximations to the mml framework .
they adopted various simplifying assumptions and approximations .
to avoid such limitations , we developed a classic mml formulation , giving the complete message length formulations for gaussian and von mises - fisher distributions in section [ sec : mml_est_derivations ] .
secondly , in most of these methods , the search for the optimal number of mixture components is achieved by selecting the mixture that results in the best em outcome out of many trials @xcite .
this is not an elegant solution and @xcite proposed a search heuristic which integrates estimation and model selection .
a comparative study of these methods is presented in @xcite .
their analysis suggested the superior performance of icl @xcite and lec @xcite .
later , @xcite demonstrated that their proposed method outperforms the contemporary methods based on icl and lec and is regarded as the current state of the art .
we , therefore , compare our method against that of @xcite and demonstrate its effectiveness .    with this background
, we formulate an alternate search heuristic to infer the optimal number of mixture components which aims to address the above limitations .
the space of candidate mixture models to explain the given data is infinitely large . as per the mml criterion ( equation  ) , the goal is to search for the mixture that has the smallest overall message length .
we have seen in section  [ subsec : em_mml ] that if the number of mixture components are fixed , then the em algorithm can be used to estimate the mixture parameters , namely the component weights and the parameters of each component .
however , here it is required to search for the optimal _ number _ of mixture components along with the corresponding mixture parameters .
our proposed search heuristic extends the mml - based snob program @xcite for unsupervised learning .
we define three operations , namely _ split , delete , _ and _ merge _ that can be applied to any component in the mixture .
[ algm ]    @xmath211    the pseudocode of our search method is presented in algorithm [ algm ] .
the basic idea behind the search strategy is to _ perturb _ a mixture from its current suboptimal state to obtain a new state ( if the perturbed mixture results in a smaller message length ) . in general ,
if a ( current ) mixture has @xmath4 components , it is perturbed using a series of _ split , delete _ , and _ merge _ operations to check for improvement
. each component is split and the new @xmath212-component mixture is re - estimated .
if there is an improvement ( _ i.e. , _ if there is a decrease in message length with respect to the current mixture ) , the new @xmath212-component mixture is retained .
there are @xmath4 splits possible and the one that results in the greatest improvement is recorded ( lines 5  7 in algorithm  [ algm ] ) . a component is first split into two sub - components ( children ) which are locally optimized by the em algorithm on the data that belongs to that sole component .
the child components are then integrated with the others and the mixture is then optimized to generate a @xmath147 component mixture .
the reason for this is , rather than use random initial values for the em , it is better if we start from some already optimized state to reach to a better state .
similarly , each of the components is then deleted , one after the other , and the @xmath213-component mixture is compared against the current mixture .
there are @xmath4 possible deletions and the best amongst these is recorded ( lines 8  11 in algorithm  [ algm ] ) . finally , the components in the current mixture are merged with their closest matches ( determined by calculating the kl - divergence ) and each of the resultant @xmath213-component mixtures are evaluated against the @xmath4 component mixture . the best among these merged mixtures
is then retained ( lines 12  15 in algorithm  [ algm ] ) .
we initially start by assuming a one component mixture .
this component is split into two children which are locally optimized .
if the split results in a better model , it is retained . for any given @xmath4-component mixture , there might be improvement due to splitting , deleting and/or merging its components .
we select the perturbation that best improves the current mixture .
this process is repeated until there is no further improvement possible . and
the algorithm is continued .
the notion of _ best _ or improved mixture is based on the amount of reduction of message length that the perturbed mixture provides . in the current state , the observed data have partial memberships in each of the @xmath4 components . before the execution of each operation , these memberships need to be adjusted and a em is subsequently carried out to achieve an optimum with a different number of components
. we will now examine each operation in detail and see how the memberships are affected after each operation .
let @xmath214 $ ] be the @xmath215 responsibility ( membership ) matrix and @xmath5 be the weight of @xmath7 component in mixture @xmath0 .    1 .
_ split ( line 6 in algorithm  [ algm ] ) : _ as an example , assume a component with index @xmath216 and weight @xmath217 in the current mixture @xmath0 is split to generate two child components .
the goal is to find two distinct clusters amongst the data associated with component @xmath218 .
it is to be noted that the data have fractional memberships to component @xmath218 .
the em is therefore , carried out _ within _ the component @xmath218 assuming a _ two - component sub - mixture _ with the data weighted as per their current memberships @xmath219 .
the remaining @xmath220 components are untouched .
an em is carried out to optimize the two - component sub - mixture . the initial state and the subsequent updates in the maximization - step are described below .
+ _ parameter initialization of the two - component sub - mixture : _ the goal is to identify two distinct clusters within the component @xmath218 . for _ gaussian
_ mixtures , to provide a reasonable starting point , we compute the direction of maximum variance of the parent component and locate two points which are one standard deviation away on either side of its mean ( along this direction ) . these points
serve as the initial means for the two children generated due to splitting the parent component . selecting the initial means in this manner
ensures they are reasonably apart from each other and serves as a good starting point for optimizing the two - component sub - mixture .
the memberships are initialized by allocating the data points to the closest of the two means .
once the means and the memberships are initialized , the covariance matrices of the two child components are computed .
+ there are conceivably several variations to how the two - component sub - mixture can be initialized .
these include random initialization , selecting two data points as the initial component means , and many others . however , the reason for selecting the direction of maximum variance is to utilize the available characteristic of data , _
i.e. , _ the distribution within the component @xmath218 . for _ von mises - fisher _ mixtures ,
the maximum variance strategy ( as for gaussian mixtures ) can not be easily adopted , as the data is distributed on the hypershpere .
hence , in this work , we randomly allocate data memberships and compute the components ( initial ) parameters .
+ once the parameters of the sub - mixture are initialized , an em algorithm is carried out ( just for the sub - mixture ) with the following maximization - step updates .
let @xmath221 $ ] be the @xmath222 responsibility matrix for the two - component sub - mixture .
for @xmath223 , let @xmath224 be the effective memberships of data belonging to the two child components , let @xmath225 be the weights of the child components within the sub - mixture , and let @xmath226 be the parameters describing the child components . *
the effective memberships are updated as given by equation .
@xmath227 * as the sub - mixture comprises of two child components , substitute @xmath228 in equation to obtain the updates for the weights .
these are given by equation .
@xmath229 * for _ gaussian _ mixtures , the component parameters @xmath230 are updated as follows : @xmath231 * for _ von mises - fisher _ mixtures , the component parameters @xmath232 are updated as follows : @xmath233 @xmath234 represents the magnitude of vector @xmath235 .
the update of the concentration parameter @xmath236 is obtained by solving @xmath237 after substituting @xmath238 and @xmath239 in equation .
+ the difference between the em updates in equations , and equations , is the presence of the coefficient @xmath240 with each @xmath131 .
since we are considering the sub - mixture , the original responsibility @xmath241 is multiplied by the responsibility within the sub - mixture @xmath242 to quantify the influence of datum @xmath131 to each of the child components .
+ after the sub - mixture is locally optimized , it is integrated with the untouched @xmath220 components of @xmath0 to result in a @xmath147 component mixture @xmath243 .
an em is finally carried out on the combined @xmath147 components to estimate the parameters of @xmath243 and result in an optimized @xmath212-component mixture as follows .
+ _ em initialization for @xmath243 : _ usually , the em is started by a random initialization of the members . however , because the two - component sub - mixture is now optimal and the @xmath220 components in @xmath0 are also in an optimal state , we exploit this situation to initialize the em ( for @xmath243 ) with a reasonable starting point .
as mentioned above , the component with index @xmath218 with component weight @xmath217 is split . upon integration , the ( child ) components that replaced component @xmath218
will now correspond to indices @xmath244 and @xmath245 in the new mixture @xmath243 .
let @xmath246 \,\forall 1 \leq
i \leq n , 1 \leq j \leq m+1 $ ] be the responsibility matrix for the new mixture @xmath243 and let @xmath247 be the component weights in @xmath243 . * _ component weights : _ the weights are initialized as follows : @xmath248 * _ memberships : _ the responsibility matrix @xmath249 is initialized for all data @xmath250 as follows : @xmath251 where @xmath252 are the effective memberships of the components in @xmath243 . + with these starting points , the parameters of @xmath243 are estimated using the traditional em algorithm with updates in the maximization - step given by equations , , and .
the em results in local convergence of the @xmath212-component mixture .
if the resultant message length of encoding data using @xmath243 is lower than that due to @xmath0 , that means the perturbation of @xmath0 because of splitting component @xmath218 resulted in a new mixture @xmath243 that compresses the data better , and hence , is a better mixture model to explain the data .
2 .   _ delete ( line 10 in algorithm  [ algm ] ) : _ the goal here is to remove a component from the current mixture and check whether it results in a better mixture model to explain the observed data . assume the component with index @xmath218 and the corresponding weight @xmath217 is to be deleted from @xmath0 to generate a @xmath220 component mixture @xmath243 .
once deleted , the data memberships of the component need to be redistributed between the remaining components .
the redistribution of data results in a good starting point to employ the em algorithm to estimate the parameters of @xmath243 as follows .
+ _ em initialization for @xmath243 : _ let @xmath246 $ ] be the @xmath253 responsibility matrix for the new mixture @xmath243 and let @xmath247 be the weight of @xmath7 component in @xmath243 . * _ component weights : _ the weights are initialized as follows : @xmath254 it is to be noted that @xmath255 because the mml update expression in the m - step for the component weights always ensures non - zero weights during every iteration of the em algorithm ( see equation ) . * _ memberships : _ the responsibility matrix @xmath249 is initialized for all data @xmath250 as follows : @xmath256 where @xmath252 are the effective memberships of the components in @xmath243 .
it is possible for a datum @xmath131 to have complete membership in component @xmath218 ( _ i.e. , _
@xmath257 ) , in which case , its membership is equally distributed among the other @xmath220 components ( _ i.e. , _
@xmath258 ) .
+ with these readjusted weights and memberships , and the constituent @xmath220 components , the traditional em algorithm is used to estimate the parameters of the new mixture @xmath243 . if the resultant message length of encoding data using @xmath243 is lower than that due to @xmath0 , that means the perturbation of @xmath0 because of deleting component @xmath218 resulted in a new mixture @xmath243 with better explanatory power , which is an improvement over the current mixture .
3 .   _ merge ( line 14 in algorithm  [ algm ] ) : _ the idea is to join a pair of components of @xmath0 and determine whether the resulting @xmath213-component mixture @xmath243 is any better than the current mixture @xmath0 .
one strategy to identify an improved mixture model would be to consider merging all possible pairs of components and choose the one which results in the greatest improvement .
this would , however , lead to a runtime complexity of @xmath259 , which could be significant for large values of @xmath4 .
another strategy is to consider merging components which are  close " to each other .
for a given component , we identify its _ closest _ component by computing the kullback - leibler ( kl ) distance with all others and selecting the one with the least value .
this would result in a linear runtime complexity of @xmath260 as computation of kl - divergence is a constant time operation .
for every component in @xmath0 , its closest match is identified and they are merged to obtain a @xmath220 component mixture @xmath243 . merging the pair involves reassigning the component weights and the memberships .
an em algorithm is then employed to optimize @xmath243 .
+ assume components with indices @xmath218 and @xmath261 are merged .
let their weights be @xmath217 and @xmath262 ; and their responsibility terms be @xmath219 and @xmath263 respectively .
the component that is formed by merging the pair is determined first .
it is then integrated with the @xmath264 remaining components of @xmath0 to produce a @xmath213-component mixture @xmath243 .
+ _ em initialization for @xmath243 : _ let @xmath265 and @xmath266 be the weight and responsibility vector of the merged component @xmath267 respectively .
they are given as follows : @xmath268 the parameters of this merged component are estimated as follows : * _ gaussian : _ the parameters @xmath269 are : @xmath270 * _ von mises - fisher : _ the parameters @xmath271 are : @xmath272 the concentration parameter @xmath273 is obtained by solving @xmath274 after substituting @xmath275 and @xmath276 in equation .
+ the merged component @xmath267 with weight @xmath265 , responsibility vector @xmath277 , and parameters @xmath278 is then integrated with the @xmath264 components . the merged component and
its associated memberships along with the @xmath264 other components serve as the starting point for optimizing the new mixture @xmath243 .
if @xmath243 results in a lower message length compared to @xmath0 that means the perturbation of @xmath0 because of merging the pair of components resulted in an improvement to the current mixture .
r0.35     we explain the proposed inference of mixture components through the following example that was also considered by @xcite . consider a bivariate gaussian mixture shown in fig .
[ fig : mix1 ]
. the mixture has three components with equal weights of 1/3 each and their means at ( -2,0 ) , ( 0,0 ) , and ( 2,0 ) .
the covariance matrices of the three components are the same and are equal to @xmath279 .
we simulate 900 data points from this mixture ( as done by @xcite ) and employ the proposed search strategy .
the progression of the search method using various operations is detailed below .    _
search for the optimal mixture model : _ the method begins by inferring a one - component mixture @xmath280 ( see fig .  [
fig : mix1_iter_1_splits](a ) ) .
it then splits this component ( as described in _ split _
step of section  [ subsec : search_operations ] ) and checks whether there is an improvement in explanation .
the red ellipse in fig  [ fig : mix1_iter_1_splits](b ) depicts the component being split .
the direction of maximum variance ( dotted black line ) is first identified , and the means ( shown by black dots at the end of the dotted line ) are initialized .
an em algorithm is then used to optimize the two children and this results in a mixture @xmath281 shown in fig  [ fig : mix1_iter_1_splits](c ) .
since the new mixture has a lower message length , the current is updated as @xmath281 .    in the second iteration ,
each component in @xmath281 is iteratively split , deleted , and merged .
[ fig : mix1_iter_2_splits ] shows the splitting ( red ) of the first component . on splitting , the new mixture @xmath282 results in a lower message length .
deletion of the first component is shown in fig .
[ fig : mix1_iter_2_deletions ] . before merging the first component
, we identify its closest component ( the one with the least kl - divergence ) ( see fig .
[ fig : mix1_iter_2_merges ] ) . deletion and merging operations , in this case , do not result in an improvement .
these two operations have different intermediate em initializations ( figures  [ fig : mix1_iter_2_deletions](b ) and [ fig : mix1_iter_2_merges](b ) ) but result in the same optimized one - component mixture .
the same set of operations are performed on the second component in @xmath281 . in this particular case ,
splitting results in an improved mixture ( same as @xmath282 ) .
@xmath282 is updated as the new parent and the series of split , delete , and merge operations are carried out on all components in @xmath282 .
[ fig : mix1_iter_3 ] shows these operations on the first component .
we see that splitting the first component in @xmath282 results in @xmath283 ( see fig .
[ fig : mix1_iter_3](c ) ) .
however , @xmath283 is not an improvement over @xmath282 as seen by the message lengths and is , therfore , discarded .
similarly , deletion and merging of the components do not yield improvements to @xmath282 .
the operations are carried out on the remaining two components in @xmath282 ( not shown in the figure ) too .
these perturbations do not produce improved mixtures in terms of the total message length .
since the third iteration does not result in any further improvement , the search terminates and the parent @xmath282 is considered to be the best mixture .     +   +    in different stages of the search method , we have different intermediate mixtures .
em is a gradient descent technique and it can get trapped in a local optimum . by employing the suggested search , we are exhaustively considering the possible options , and aiming to reduce the possibility of the em getting stuck in a local optimum .
the proposed method infers a mixture by balancing the tradeoff due to model complexity and the fit to the data .
this is particularly useful when there is no prior knowledge pertaining to the nature of the data .
in such a case , this method provides an objective way to infer a mixture with suitable components that best explains the data through lossless compression .
another example is shown in appendix [ subsec : appendix_mix2 ] , where the evolution of the inferred mixture is explored in the case of a mixture with overlapping components .
+ _ variation of the two - part message length : _ the search method infers three components and terminates . in order to demonstrate that the inferred number of components is the optimum number , we infer mixtures with increasing number of components ( until @xmath284 as an example ) and plot their resultant message lengths . for each @xmath285 , the standard em algorithm ( section  [ subsec : em_mml ] ) is employed to infer the mixture parameters .
r0.5     fig .
[ fig : individual_msglens_gaussian ] shows the total message lengths to which the em algorithm converges for varying number of components @xmath4 .
as expected , the total message length ( green curve ) drastically decreases initially until @xmath286 components are inferred . starting from @xmath287 ,
the total message length gradually increases , clearly suggesting that the inferred models are over - fitting the data with increasing statement cost to encode the additional parameters of these ( more complex ) models .
we further elaborate on the reason for the initial decrease and subsequent increase in the total message length .
as per mml evaluation criterion , the message length comprises of two parts  statement cost for the parameters and the cost for stating the data using those parameters .
the model complexity ( which corresponds to the mixture parameters ) increases with increasing @xmath4 .
therefore , the first part of the message to encode parameters increases with an increase in the number of parameters .
this behaviour is illustrated by the red curve in fig .
[ fig : individual_msglens_gaussian ] .
the first part message lengths are shown in red on the right side y - axis in the figure .
as the mixture model becomes increasingly more complex , the error of fitting the data decreases .
this corresponds to the second part of the message in the mml encoding framework .
this behaviour is consistent with what is observed in fig .
[ fig : individual_msglens_gaussian ] ( blue curve ) .
there is a sharp fall until @xmath286 ; then onwards increasing the model complexity does not lower the error significantly . the error saturates and
there is minimal gain with regards to encoding the data ( the case of overfitting ) .
however , the model complexity dominates after @xmath286 .
the optimal balance is achieved when @xmath286 . in summary ,
the message length at @xmath286 components was rightly observed to be the optimum for this example .
we note that for a fixed number of mixture components , the em algorithm for the mml metric is monotonically decreasing .
however , while searching for the number of components , mml continues to decrease until some optimum is found and then steadily increases as illustrated through this example .
we compare our proposed inference methodology against the widely cited method of @xcite . the performance of their method is compared against that of bayesian information criterion ( bic ) , integrated complete likelihood ( icl ) , and approximate bayesian ( lec ) methods ( discussed in section [ sec : mixture_existing_methods ] ) .
it was shown that the method of @xcite was far superior than bic , icl and lec ( using gaussian mixtures ) . in the following sections , we demonstrate through a series of experiments that our proposed approach to infer mixtures fares better when compared to that of @xcite .
the experimental setup is as follows : we use a gaussian mixture @xmath288 ( true distribution ) , generate a random sample from it , and infer the mixture using the data .
this is repeated 50 times and we compare the performance of our method against that of @xcite . as part of our analysis , we compare the number of inferred mixture components as well as the quality of mixtures .      _ comparing message lengths : _ the mml framework allows us to objectively compare mixture models by computing the total message length used to encode the data . the difference in message lengths
gives the log - odds posterior ratio of any two mixtures ( equation ) .
given some observed data , and any two mixtures , one can determine which of the two best explains the data .
our search methodology uses the scoring function ( @xmath289 ) defined in equation .
as elaborated in section  [ subsec : fj ] , @xcite use an approximated mml - like scoring function ( @xmath290 ) given by equation .
we employ our search method and the method of @xcite to infer the mixtures using the same data ; let the inferred mixtures be @xmath291 and @xmath292 respectively .
we compute two quantities : @xmath293 we use the two different scoring functions to compute the differences in message lengths of the resulting mixtures @xmath292 and @xmath291 .
since the search method used to obtain @xmath291 optimizes the scoring function @xmath289 , it is expected that @xmath294 and consequently @xmath295 .
this implies that our method is performing better using our defined objective function .
however , if @xmath296 , this indicates that our inferred mixture @xmath291 results in a lower value of the scoring function that is defined by @xcite .
such an evaluation not only demonstrates the superior performance of our search ( leading to @xmath291 ) using our defined scoring function but also proves it is better using the scoring function as defined by @xcite .
_ kullback leibler ( kl ) divergence : _ in addition to using message length based evaluation criterion , we also compare the mixtures using kl - divergence @xcite .
the metric gives a measure of the similarity between two distributions ( the lower the value , the more similar the distributions ) . for a mixture probability distribution
, there is no analytical form to compute the metric .
however , one can calculate its empirical value ( which asymptotically converges to the kl - divergence ) . in experiments relating to mixture simulations ,
we know the true mixture @xmath288 from which the data @xmath297 is being sampled .
the kl - divergence is given by the following expression : @xmath298               \approx \frac{1}{n } \sum_{i=1}^n\log\frac{\pr(\mathbf{x}_i,{\mathcal{m}}^t)}{\pr(\mathbf{x}_i,{\mathcal{m}})}\end{aligned}\ ] ] where @xmath0 is a mixture distribution ( @xmath291 or @xmath292 ) whose _ closeness _ to the true mixture @xmath288 is to be determined .
an experiment conducted by @xcite was to randomly generate @xmath299 data points from a two - component ( with equal mixing proportions ) bivariate mixture @xmath288 whose means are at @xmath300 and @xmath301 , and equal covariance matrices : @xmath302 ( the identity matrix ) , and compare the number of inferred components .
we repeat the same experiment here and compare with the results of @xcite .
the separation @xmath303 between the means is gradually increased and the percentage of the correct selections ( over 50 simulations ) as determined by the two search methods is plotted .
[ fig : gaussian_2d_mixture_inference](a ) shows the results of this experiment . as the separation between the component means is increased , the number of correctly inferred components increases .
we conducted another experiment where we fix the separation between the two components and increase the amount of data being sampled from the mixture .
[ fig : gaussian_2d_mixture_inference](b ) illustrates the results for a separation of @xmath304 . as expected , increasing the sample size results in an increase in the number of correct selections .
both the search methods eventually infer the true number of components at sample size @xmath305 .
we note that in both these cases , the differences in message lengths @xmath306 and @xmath307 are close to zero .
the kl - divergences for the mixtures inferred by the two search methods are also the same .
therefore , for this experimental setup , the performance of both the methods is roughly similar .    as the difference between the two search methods is not apparent from these experiments , we wanted to investigate the behaviour of the methods with smaller sample sizes .
we repeated the experiment similar to that shown in fig .
[ fig : gaussian_2d_mixture_inference](a ) but with a sample size of @xmath308 .
our search method results in a mean value close to 1 for different values of @xmath303 ( see table  [ tab : gaussian_2d_mixture_inference_boxplot ] ) .
the mean value of the number of inferred components using the search method of @xcite fluctuates between 2 and 3 .
however , there is significant variance in the number of inferred components ( see table  [ tab : gaussian_2d_mixture_inference_boxplot ] ) .
these results are also depicted through a boxplot ( fig .
[ fig : gaussian_2d_mixture_inference_boxplot ] ) .
there are many instances where the number of inferred components is more than 3 .
the results indicate that the search method ( fj ) is overfitting the data .
r0.5     further , we evaluate the correctness of the mixtures inferred by the two search methods by comparisons using the message length formulations and kl-divergence.fig .
[ fig : gaussian_2d_msglen_comparisons ] shows the boxplot of the difference in message lengths of the mixtures @xmath291 inferred using our proposed search method and the mixtures @xmath292 inferred using that of @xcite .
@xmath295 across all values of @xmath303 for the 50 simulations . as per equation
, we have @xmath294 .
this implies that @xmath291 has a lower message length compared to @xmath292 when evaluated using our scoring function .
similarly , we have @xmath309 , _
i.e. , _ @xmath310 .
this implies that @xmath292 has a lower message length compared to @xmath291 when evaluated using fj s scoring function .
these results are not surprising as @xmath291 and @xmath292 are obtained using the search methods which optimize the respective mml and mml - like scoring functions .
we then analyzed the kl - divergence of @xmath291 and @xmath292 with respect to the true bivariate mixture @xmath288 over all 50 simulations and across all values of @xmath303 .
ideally , the kl - divergence should be close to zero .
[ fig : gaussian_2d_kldiv_comparisons](a ) shows the kl - divergence of the mixtures inferred using the two search methods with respect to @xmath288 when the separation is @xmath304 . the proposed search method infers mixtures whose kl - divergence ( denoted by red lines ) is close to zero , and more importantly less than the kl - divergence of mixtures inferred by the search method of @xcite ( denoted by blue lines ) . the same type of behaviour is noticed with other values of @xmath303 .
[ fig : gaussian_2d_kldiv_comparisons](b ) compares the kl - divergence for varying values of @xmath303 .
the median value of the kl - divergence due to the proposed search method is close to zero with not much variation .
the search method of @xcite always result in kl - divergence higher than that of ours .
the results suggest that , in this case , mixtures @xmath292 inferred by employing the search method of @xcite deviate significantly from the true mixture distribution @xmath288 .
this can also be explained by the fact that there is a wide spectrum of the number of inferred components ( fig .
[ fig : gaussian_2d_mixture_inference_boxplot ] ) .
this suggests that the mml - like scoring function is failing to control the tradeoff between complexity and quality of fit , and hence , is selecting overly complex mixture models .      along the lines of the previous experiment
, @xcite conducted another experiment for a 10-variate two - component mixture @xmath288 with equal mixing proportions .
the means are at @xmath311 and @xmath312 , so that the euclidean distance between them is @xmath313 .
the covariances of the two components are @xmath302 ( the identity matrix ) .
random samples of size @xmath299 were generated from the mixture and the number of inferred components are plotted .
the experiment is repeated for different values of @xmath303 and over 50 simulations .
[ fig : gaussian_10d_mixture_inference](a ) shows the number of inferred components using the two search methods . at lower values of @xmath303 ,
the components are close to each other , and hence , it is relatively more difficult to correctly infer the true number of components .
we observe that our proposed method performs clearly better than that of @xcite across all values of @xmath303 .
we also compared the quality of these inferred mixtures by calculating the difference in message lengths using the two scoring functions and the kl - divergence with respect to @xmath288 .
for all values of @xmath303 , @xmath295 , _
i.e. , _ our inferred mixtures @xmath291 have a lower message length compared to @xmath292 when evaluated using our scoring function .
more interestingly , we also note that @xmath314 ( see fig .
[ fig : gaussian_10d_comparisons](a ) ) .
this reflects that @xmath291 have a lower message length compared to @xmath292 when evaluated using the scoring function of @xcite .
this suggests that their search method results in a sub - optimal mixture @xmath292 and fails to infer the better @xmath291 .
in addition to the message lengths , we analyze the mixtures using kl - divergence .
similar to the bivariate example in fig .
[ fig : gaussian_2d_kldiv_comparisons](a ) , the kl - divergence of our inferred mixtures @xmath291 is lower than @xmath292 , the mixtures inferred by @xcite .
[ fig : gaussian_10d_comparisons](b ) shows the boxplot of kl - divergence of the inferred mixtures @xmath291 and @xmath292 . at higher values of @xmath315 ,
the median value of kl - divergence is close to zero , as the number of correctly inferred components ( fig .
[ fig : gaussian_10d_mixture_inference](a ) ) is more than 90%
. however , our method always infers mixtures @xmath291 with lower kl - divergence compared to @xmath292 .
these experimental results demonstrate the superior performance of our proposed search method .
another experiment was carried out where the @xmath316 was held constant ( extremely close components ) , gradually increased the sample size @xmath34 , and plotted the average number of inferred components by running 50 simulations for each @xmath34 .
[ fig : gaussian_10d_mixture_inference](b ) shows the results for the average number of inferred components as the amount of data increases .
our search method , on average , infers the true mixture when the sample size is @xmath317 .
however , the search method of @xcite requires larger amounts of data ; even with a sample size of 2000 , the average number of inferred components is @xmath318 . in fig .
[ fig : gaussian_10d_mixture_inference](b ) , the red curve reaches the true number of 2 and saturates more rapidly than the blue curve .
one of the drawbacks associated with the search method of @xcite is due to the form of the updating expression for the component weights ( equation  ) . as discussed in section  [ subsubsec : fj_search_drawbacks ] , a particular instance of wrong inference
is bound to happen when the net membership of a ( valid ) component is less than @xmath208 , where @xmath200 is the number of free parameters per component .
in such a case , the component weight is updated as zero , and it is eliminated , effectively reducing the mixture size by one .
we conducted the following experiment to demonstrate this behaviour : we considered the two - component 10-variate mixture @xmath288 as before and randomly generate samples of size 50 from the mixture . since the constituent components of @xmath288 have equal weights , on average , each component has a membership of 25 .
we used @xmath319 , so that the two components are well apart from each other . for each @xmath303 , we run 50 simulations and analyze the number of inferred components . as expected , the search method of @xcite always infer a mixture with one component regardless of the separation @xmath303 .
our method always infers the correct number of components . in order to test the validity of mixtures inferred by our proposed method
, we analyze the resultant mixtures by comparing the message lengths as discussed in section [ subsec : comparison_mixtures ] .
[ fig : gaussian_10d_msglens_comparisons_exp2b](a ) shows the difference in message lengths @xmath306 given in equation .
we observe that @xmath295 for all @xmath303 .
this demonstrates that our search based mixtures @xmath291 have lower message lengths compared to mixtures @xmath292 using our scoring function .
the same phenomenon is observed when using the mml - like scoring function of @xcite . in fig .
[ fig : gaussian_10d_msglens_comparisons_exp2b](b ) , we observe that @xmath314 , which means our search based mixtures @xmath291 have lower message lengths compared to mixtures @xmath292 when evaluated using their scoring function .
r0.45     this demonstrates that @xmath291 is a better mixture as compared to @xmath292 and their search method is unable to infer it .
we also note that the differences in message lengths increases with increasing @xmath303 .
this is because for the one - component inferred mixture @xmath292 , the second part of the message ( equation  ) which corresponds to the negative log - likelihood term increases because of poorer fit to the data .
the two modes in the data become increasingly pronounced as the separation between constituent components of the true mixture increases , and hence , modelling such a distribution using a one - component mixture results in a poorer fit .
this is clearly an incorrect inference .
we further strengthen our case by comparing the kl - divergence of the inferred mixtures @xmath291 and @xmath292 with respect to the true mixture .
[ fig : gaussian_10d_kldiv_comparisons_exp2b ] illustrates the results . as @xmath303 increases , the blue coloured plots shift higher .
these correspond to mixtures @xmath292 inferred by @xcite .
our search method , however , infers mixtures @xmath291 which have lower kl - divergence .
the figure indicates that the inferred mixtures @xmath291 are more similar to the true distribution as compared to mixtures @xmath292 .
these experiments demonstrate the ability of our search method to perform better than the widely used method of @xcite .
we compared the resulting mixtures using our proposed mml formulation and the mml - like formulation of @xcite , showing the advantages of the former over the latter .
we also used a neutral metric , kl - divergence , to establish the closeness of our inferred mixtures to the true distributions .
we will now illustrate the behaviour of our search method on two real world datasets .      at any intermediate stage of the search procedure ,
current _ mixture with @xmath4 components requires @xmath4 number of split , delete , and merge operations before it is updated .
each of the perturbations involve performing an em to optimize the corresponding mixture parameters .
to determine the convergence of em , we used a threshold of @xmath320 which was the same as used by @xcite .
fj s method also requires to start from an initial large number of components .
we used 25 as an initial number based on what was suggested in @xcite .
we investigate the number of times the em routine is called and compare it with that of @xcite .
we examine with respect to the simulations that were carried out previously . for the bivariate mixture discussed in section  [ subsec : bivariate_mix_simulation ] , the number of resulting em iterations when the sample sizes are @xmath299 and @xmath308 are compared in fig .
[ fig : em_iterations](a ) , ( b ) respectively .    as per the discussion in section  [ subsec : bivariate_mix_simulation
] , at @xmath299 , the average number of components inferred by the two methods are about the same ( fig .
[ fig : gaussian_2d_mixture_inference](a ) ) .
however , the number of em iterations required by fj s method is greater than 200 across all values of @xmath303 ( fig .
[ fig : em_iterations](a ) ) .
in contrast , the proposed method , on average , requires fewer than 50 iterations . in this case
, both methods produce a similar result with fj s method requiring more number of em iterations . when the bivariate mixture simulation is carried out using @xmath308 , the number of em iterations required by fj s method , on average , is greater than 100 , while the proposed method requires fewer than 40 iterations ( fig .
[ fig : em_iterations](b ) ) . in this case , the proposed method not only infers better mixtures ( as discussed in section  [ subsec : bivariate_mix_simulation ] ) but is also conservative with respect to computational cost .    for the simulation results corresponding to the 10-variate mixtures in section  [ subsec:10d_mix_simulation ] , the proposed method requires close to 50 iterations on average , while fj s method requires about 20 ( fig .
[ fig : em_iterations](c ) ) .
however , the mixtures inferred by the proposed method fare better when compared to that of fj ( figs .
[ fig : gaussian_10d_mixture_inference ] , [ fig : gaussian_10d_comparisons ] ) .
furthermore , for the simulation results explained in section  [ subsec : fj_weight_updates_exp2b ] , fj s method stops after 3 em iterations .
this is because their program does not accommodate components when the memberships are less than @xmath208 .
the proposed method requires 18 em iterations on average and infers the correct mixture components . in these two cases ,
our method infers better quality mixtures , with no significant overhead with regard to the computational cost .
the first example is the univariate _ acidity _ data set which contains 155 points .
our proposed search method infers a mixture @xmath291 with 2 components whereas the search method of @xcite infers a mixture @xmath292 with 3 components .
the inferred mixtures are shown in fig .
[ fig : acidity ] and their corresponding parameter estimates are given in table  [ tab : acidity_mixtures_inference ] . in order to compare the mixtures inferred by the two search methods , we compute the message lengths of the inferred mixtures using our complete mml and the approximated mml - like scoring functions .    when evaluated using our mml scoring function , our inferred mixture results in a gain of @xmath321 bits ( see table  [ tab : acidity_comparison ] ) .
based on the mml framework , our two - component mixture @xmath291 is @xmath322 times more likely than the three - component mixture @xmath292 ( as per equation  ) .
furthermore , when the inferred mixtures are evaluated as per the mml - like scoring function , @xmath291 is still considered better ( @xmath323 bits ) than @xmath292 ( @xmath324 bits ) .
thus , using both forms of scoring function , @xmath291 is the better mixture model of this data set .
[ tab : acidity_mixtures_inference ]    .message lengths ( measured in bits ) of the mixtures ( in fig .
[ fig : acidity ] ) as evaluated using the mml and mml - like scoring functions . [ cols="^,^,^ " , ]     [ tab : null_models_comparison ]    the per residue statistic is calculated by dividing the total message length by the sample size ( the number of @xmath325 pairs ) . this statistic shows that close to 2.5 bits can be saved ( on average ) if the protein data is encoded using the vmf null model .
the vmf null model thus supercedes the naive model of encoding .
this can potentially improve the accuracy of statistical inference that is central to the various protein modelling tasks briefly introduced above .
we presented a statistically robust approach for inferring mixtures of ( i )  multivariate gaussian distributions , and ( ii )  von mises - fisher distributions for @xmath3-dimensional directional data .
it is based on the general information - theoretic framework of minimum message length inference .
this provides an objective tradeoff between the hypothesis complexity and the quality of fit to the data .
an associated search procedure for an optimal mixture model of given data chooses the number of component distributions , @xmath4 , by minimizing the total message length .
we established the better performance of the proposed search algorithm by comparing with a popularly used search method @xcite .
we demonstrated the effectiveness of our approach through extensive experimentation and validation of our results .
we also applied our method to real - world high dimensonal text data and to directional data that arises from protein chain conformations .
the experimental results demonstrate that our proposed method fares better when compared with the current state of the art techniques .
the authors would like to thank arun konagurthu for discussions pertaining to protein data clustering , and maria garcia de la banda for stimulating discussions and providing interesting insights .
the authors would also like to acknowledge wray buntine s inputs with regard to text clustering .
91 [ 1]#1 [ 1]#1 urlstyle [ 1]doi  # 1    [ 2][]#2    agusta y , dowe dl ( 2003 ) unsupervised learning of correlated multivariate gaussian mixture models using mml . in : ai 2003 : advances in artificial intelligence , springer , berlin , heidelberg , pp 477489    akaike h ( 1974 ) a new look at the statistical model identification .
ieee transactions on automatic control , 19(6):716723    anderson e ( 1935 ) the irises of the gasp peninsula .
bulletin of the american iris society 59:25    banerjee a , dhillon i , ghosh j , sra s ( 2003 ) generative model - based clustering of directional data . in : proceedings of the 9th international conference on knowledge discovery and data mining , acm , new york , pp 1928    banerjee a , dhillon is , ghosh j , sra s ( 2005 ) clustering on the unit hypersphere using von mises - fisher distributions . journal of machine learning research 6:13451382    barton de ( 1961 ) unbiased estimation of a set of probabilities .
biometrika 48(1 - 2):227229    basu ap ( 1964 ) estimates of reliability for some distributions useful in life testing .
technometrics 6(2):215219    best d , fisher n ( 1981 ) the bias of the maximum likelihood estimators of the von mises - fisher concentration parameters .
communications in statistics - simulation and computation 10(5):493502    biernacki c , celeux g , govaert g
( 2000 ) assessing a mixture model for clustering with the integrated completed likelihood .
ieee transactions on pattern analysis and machine intelligence 22(7):719725    bishop cm ( 2006 ) pattern recognition and machine learning , vol  1 .
springer , new york    boulton d , wallace c ( 1969 ) the information content of a multistate distribution .
journal of theoretical biology 23:269278    bozdogan h ( 1983 ) determining the number of component clusters in the standard multivariate normal mixture model using model - selection criteria .
tech . rep .
, dtic document    bozdogan h ( 1990 ) on the information - based measure of covariance complexity and its application to the evaluation of multivariate linear models .
communications in statistics - theory and methods 19(1):221278    bozdogan h ( 1993 ) choosing the number of component clusters in the mixture - model using a new informational complexity criterion of the inverse - fisher information matrix .
springer , berlin , heidelberg    burnham kp , anderson dr ( 2002 ) model selection and multimodel inference : a practical information - theoretic approach .
springer , new york    collier jh , allison l , lesk am , de  la banda mg , konagurthu as ( 2014 ) a new statistical framework to assess structural alignment quality using information compression .
bioinformatics 30(17):i512i518    conway jh , sloane nja ( 1984 ) on the voronoi regions of certain lattices .
siam journal on algebraic and discrete methods 5:294305    cordeiro gm , vasconcellos kl ( 1999 ) theory & methods : second - order biases of the maximum likelihood estimates in von mises regression models .
australian & new zealand journal of statistics 41(2):189198    dempster ap , laird nm , rubin db ( 1977 ) maximum likelihood from incomplete data via the em algorithm .
journal of the royal statistical society : series b ( methodological ) 39(1):138    dowe dl , allison l , dix ti , hunter l , wallace cs , edgoose t ( 1996 ) circular clustering of protein dihedral angles by minimum message length . in : pacific symposium on biocomputing , vol  96 , pp 242255    dowe dl , oliver jj , baxter ra , wallace cs ( 1996 ) bayesian estimation of the von mises concentration parameter . in : maximum entropy and bayesian methods , springer , netherlands , pp 5160    dowe dl , oliver jj , wallace cs ( 1996 ) mml estimation of the parameters of the spherical fisher distribution . in : algorithmic learning theory , springer , berlin , heidelberg , pp 213227    drton m , sturmfels b , sullivant s ( 2009 ) lectures on algebraic statistics . in : oberwolfach seminars ( 39 ) ,
birkhuser , basel    dwyer ps ( 1967 ) some applications of matrix derivatives in multivariate analysis . journal of the american statistical association 62(318):607625    eaton ml , morris cn ( 1970 ) the application of invariance to unbiased estimation .
the annals of mathematical statistics 41(5):17081716    figueiredo ams ( 2012 ) goodness - of - fit for a concentrated von mises - fisher distribution .
computational statistics 27(1):6982    figueiredo ma , jain ak ( 2002 ) unsupervised learning of finite mixture models .
ieee transactions on pattern analysis and machine intelligence 24(3):381396    fisher ni ( 1993 ) statistical analysis of spherical data .
cambridge university press , cambridge    fisher r ( 1953 ) dispersion on a sphere .
proceedings of the royal society of london : series a ( mathematical and physical sciences ) 217(1130):295305    fisher ra ( 1936 ) the use of multiple measurements in taxonomic problems .
annals of eugenics 7(2):179188    gauvain j , lee ch ( 1994 ) maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains .
ieee transactions on speech and audio processing 2(2):291298    gray g ( 1994 ) bias in misspecified mixtures . biometrics 50(2):457470    hornik k , grn b ( 2013 ) movmf : an r package for fitting mixtures of von mises - fisher distributions .
r package version 01 - 2    jain ak , dubes rc ( 1988 ) algorithms for clustering data .
prentice - hall , inc .
, upper saddle river , nj , usa    jain ak , duin rpw , mao j ( 2000 ) statistical pattern recognition : a review .
ieee transactions on pattern analysis and machine intelligence 22(1):437    jeffreys h ( 1946 ) an invariant form for the prior probability in estimation problems
. proceedings of the royal society of london : series a ( mathematical and physical sciences ) 186(1007):453461    jones p , mclachlan g ( 1990 ) laplace - normal mixtures fitted to wind shear data .
journal of applied statistics 17(2):271276    jorgensen ma , mclachlan gj ( 2008 ) wallace s approach to unsupervised learning : the snob program .
the computer journal 51(5):571578    kent jt ( 1982 ) the fisher - bingham distribution on the sphere .
journal of the royal statistical society : series b ( methodological ) 44(1):7180    konagurthu as , lesk am , allison l ( 2012 ) minimum message length inference of secondary structure from protein coordinate data .
bioinformatics 28(12):i97i105    konagurthu as ,
allison l , abramson d , stuckey pj , lesk am ( 2013 ) statistical inference of protein  lego bricks " . in : 2013 ieee 13th international conference on data mining ( icdm ) , ieee , pp 10911096    krishnan t , mclachlan g ( 1997 ) the em algorithm and extensions .
wiley , new york    kullback s , leibler ra ( 1951 ) on information and sufficiency .
the annals of mathematical statistics pp 7986    lee p ( 1997 ) bayesian statistics : an introduction .
arnold , london .
lo y ( 2011 ) bias from misspecification of the component variances in a normal mixture .
computational statistics and data anaysis 55(9):27392747    magnus jr , neudecker h ( 1988 ) matrix differential calculus with applications in statistics and econometrics .
wiley , new york    mardia k , jupp p ( 2000 ) directional statistics .
wiley , hoboken , nj , usa    mardia k , holmes d , kent j ( 1984 ) a goodness - of - fit test for the von mises - fisher distribution .
journal of the royal statistical society : series b ( methodological ) 46(1):7278    mardia kv , kent jt , bibby jm ( 1979 ) multivariate analysis . academic press , london    mardia kv , taylor cc , subramaniam gk ( 2007 ) protein bioinformatics and mixtures of bivariate von mises distributions for angular data .
biometrics 63(2):505512    mclachlan g , peel d ( 1997 ) contribution to the discussion of paper by s. richardson and p.j .
journal of the royal statistical society b 59:779780
mclachlan g , peel d ( 2000 ) finite mixture models .
wiley , new york    mclachlan gj , basford ke ( 1988 ) mixture models : inference and applications to clustering ( statistics : textbooks and monographs ) .
dekker , new york    murzin a , brenner s , hubbard t , chothia c , et  al .
( 1995 ) scop : a structural classification of proteins database for the investigation of sequences and structures .
journal of molecular biology 247(4):536540    oliver j , baxter r ( 1994 ) mml and bayesianism : similarities and differences .
dept comput sci monash univ , clayton , victoria , australia , tech rep 206
oliver jj , baxter ra , wallace cs ( 1996 ) unsupervised learning using mml . in : machine learning : proceedings of the 13th international conference , pp 364372    patra k , dey dk ( 1999 ) a multivariate mixture of weibull distributions in reliability modeling .
statistics & probability letters 45(3):225235    peel d , mclachlan gj ( 2000 ) robust mixture modelling using the t - distribution .
statistics and computing 10(4):339348    peel d , whiten wj , mclachlan gj ( 2001 ) fitting mixtures of kent distributions to aid in joint set identification
. journal of the american statistical association 96(453):5663    rao cr ( 1973 ) linear statistical inference and its applications .
wiley , new york    richardson s , green pj ( 1997 ) on bayesian analysis of mixtures with an unknown number of components .
journal of the royal statistical society : series b ( methodological ) 59(4):731792    rissanen j ( 1978 ) modeling by shortest data description .
automatica 14(5):465471    rissanen j ( 1989 ) stochastic complexity in statistical inquiry theory .
world scientific publishing co. , inc . , river edge , nj , usa    roberts s , husmeier d , rezek i , penny w ( 1998 ) bayesian approaches to gaussian mixture modeling .
ieee transactions on pattern analysis and machine intelligence 20(11):11331142    robertson s , zaragoza h ( 2009 ) the probabilistic relevance framework : bm25 and beyond .
now publishers inc .
, hanover , ma , usa    salton g , buckley c ( 1988 ) term - weighting approaches in automatic text retrieval .
information processing & management 24(5):513523    salton g , mcgill mj ( 1986 ) introduction to modern information retrieval .
mcgraw - hill , inc .
, new york , ny , usa    schou g ( 1978 ) estimation of the concentration parameter in von mises  fisher distributions .
biometrika 65(2):369377    schwarz g , et  al .
( 1978 ) estimating the dimension of a model .
the annals of statistics 6(2):461464    seidel w , mosler k , alker m ( 2000 ) a cautionary note on likelihood ratio tests in mixture models .
annals of the institute of statistical mathematics 52(3):481487    shannon ce ( 1948 ) a mathematical theory of communication .
the bell system technical journal 27:379423    song h , liu j , wang g ( 2012 ) high - order parameter approximation for von mises  fisher distributions . applied mathematics and computation 218(24):11,88011,890    sra s ( 2012 ) a short note on parameter approximation for von mises - fisher distributions : and a fast implementation of @xmath326 .
computational statistics 27(1):177190    strehl a , ghosh j , mooney r ( 2000 ) impact of similarity measures on web - page clustering . in : workshop on artificial intelligence for web search ( aaai 2000 ) , pp 5864    taboga m ( 2012 ) lectures on probability theory and mathematical statistics .
createspace independent pub .
tanabe a , fukumizu k , oba s , takenouchi t , ishii s ( 2007 ) parameter estimation for von mises  fisher distributions .
computational statistics 22(1):145157    titterington dm , smith af , makov ue , et  al .
( 1985 ) statistical analysis of finite mixture distributions .
wiley , new york    wallace c ( 1986 ) an improved program for classification .
in : proceedings of the 9th australian computer science conference , pp 357366    wallace c ( 1990 ) classification by minimum - message - length inference .
lecture notes in computer science , vol 468 , springer berlin heidelberg , pp 7281    wallace c , dowe d ( 1994 ) estimation of the von mises concentration parameter using minimum message length . in : proceedings of the 12th australian statistical society conference , monash university , australia    wallace cs ( 2005 ) statistical and inductive inference using minimum message length . information science and statistics , springer - verlag , secaucus , nj , usa    wallace cs , boulton dm ( 1968 ) an information measure for classification .
computer journal 11(2):185194    wallace cs , dowe dl ( 1999 ) minimum message length and kolmogorov complexity .
computer journal 42:270283    wallace cs , freeman pr ( 1987 ) estimation and inference by compact coding .
journal of the royal statistical society : series b ( methodological ) 49(3):240265    wang p , puterman ml , cockburn i , le n ( 1996 ) mixed poisson regression models with covariate dependent rates .
biometrics 52(2):381400    watson g , williams e ( 1956 ) on the construction of significance tests on the circle and the sphere .
biometrika 43(3 - 4):344352    wedel m , desarbo ws , bult jr , ramaswamy v ( 1993 ) a latent class poisson regression model for heterogeneous count data .
journal of applied econometrics 8(4):397411    white h ( 1982 ) maximum likelihood estimation of misspecified models .
econometrica 50(1):125    wood at ( 1994 ) simulation of the von mises fisher distribution .
communications in statistics - simulation and computation 23(1):157164    xu l , jordan mi ( 1996 ) on convergence properties of the em algorithm for gaussian mixtures .
neural computation 8(1):129151    zhong s , ghosh j ( 2003 ) a comparative study of generative models for document clustering . in : proceedings of the workshop on clustering high dimensional data and its applications in siam data mining conference
for brevity , we represent @xmath327 and @xmath328 as @xmath329 and @xmath330 respectively .
expressions to evaluate @xmath331 and @xmath332 are given in equations ( [ eqn : ratio_bessels ] ) , ( [ eqn : ratio_first_derivative ] ) , and ( [ eqn : ratio_second_derivative ] ) respectively .
we require @xmath330 for its use in the remainder of the derivation and we provide its expression below : @xmath333    now we discuss the derivation of @xmath121 and @xmath122 that are required for computing the mml estimate @xmath334 ( equations ( [ eqn : mml_newton_approx ] ) and ( [ eqn : mml_halley_approx ] ) ) . differentiating equation  [ eqn : i_first_derivative ]
, we have @xmath335 @xmath336        equations ( [ eqn : tmp1 ] ) and ( [ eqn : tmp2 ] ) can be used to evaluate @xmath121 and @xmath122 which can then be used to approximate the mml estimates @xmath123 and @xmath124 ( equations ( [ eqn : mml_newton_approx ] ) and ( [ eqn : mml_halley_approx ] ) respectively ) .        to obtain the optimal weights under the constraint @xmath160 , the above equation is optimized using the _ lagrangian _ objective function defined below using some _ lagrangian multiplier _ @xmath341 .
@xmath342 for some @xmath343 , the equation resulting from computing the partial derivative of @xmath344 with respect to @xmath345 and equating it to zero gives the optimal weight @xmath345 .
@xmath346 @xmath347 where @xmath348 and @xmath349 are the responsibility and effective membership terms given as per equations   and respectively . substituting the above value in equation  , we have @xmath350 there are @xmath4 equations similar to equation   for values of @xmath343 .
adding all these equations together , we get @xmath351 substituting the above value of @xmath341 in equation  , we get @xmath352    [ [ subsec : vmf_kldiv ] ] derivation of the kullback - leibler ( kl ) distance between two von mises - fisher distributions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~    the closed form expression to calculate the kl divergence between two vmf distributions is presented below .
let @xmath353 and @xmath354 be two von mises - fisher distributions with mean directions @xmath355 and concentration parameters @xmath356 .
the kl distance between any two distributions is given by @xmath357 \label{eqn : kldiv}\ ] ] where @xmath358 $ ] is the expectation of the quantity @xmath359 $ ] using the probability density function @xmath17 .
@xmath360 using the fact that @xmath361 = a_d(\kappa_1)\boldsymbol{\mu_1}$ ] @xcite , we have the following expression : @xmath362 & = \log\frac{c_d(\kappa_1)}{c_d(\kappa_2 ) } + ( \kappa_1\boldsymbol{\mu_1 } -\kappa_2\boldsymbol{\mu_2})^t a_d(\kappa_1 ) \boldsymbol{\mu_1 } \notag\\ d_{kl}(f||g ) & = \log\frac{c_d(\kappa_1)}{c_d(\kappa_2 ) } + a_d(\kappa_1 ) ( \kappa_1 -\kappa_2\boldsymbol{\mu_1}^t\boldsymbol{\mu_2 } ) \label{eqn : vmf_kldiv}\end{aligned}\ ] ]        we consider an example of a mixture with overlapping components and employ the proposed search method to determine the number of mixture components . @xcite considered the mixture shown in fig .
[ fig : mix2 ] which is described as follows : let @xmath363 be the weight , mean , and covariance matrix of the @xmath364 component respectively .
then , the mixture parameters are given as @xmath365                    0.5 & 1                             \end{bmatrix } , \mathbf{c}_2 =   \begin{bmatrix }                    6    & -2        \\[0.3em ]                    -2   & 6                             \end{bmatrix},\ , \mathbf{c}_3 =   \begin{bmatrix }                    2   & -1        \\[0.3em ]                    -1   & 2                             \end{bmatrix},\ , \mathbf{c}_4 =   \begin{bmatrix }                    0.125    & 0        \\[0.3em ]                    0   & 0.125                             \end{bmatrix}\end{gathered}\ ] ]    we generate a random sample of size 1000 ( similar to @xcite ) and infer mixtures using the proposed heuristic .
below are diagrams which portray the splitting of the individual components .
we show a selection of operations which result in improved mixtures .