problems have been well studied and many algorithms have been developed and successfully applied in a vast array of applications @xcite .
a generalization of the problem to multiple datasets , termed , has been introduced recently @xcite .
the recent interest in is motivated by various application domains such as when analyzing multisubject datasets in biomedical studies using or data @xcite or when solving the convolutive problem in the frequency domain using multiple frequency bins @xcite .
interestingly , several algorithms developed prior to the development of the concept are capable of achieving @xcite .
thus , a much larger set of applications than the examples above are well treated using the formulation .
one particular formulation of has been termed .
the formulation of is an extension of the ( linear , instantaneous ) model .
assumes a source within one dataset is dependent on at most one source in another dataset while sources within a dataset are mutually independent ( as in ) .
thus , reduces to performing individually on each dataset when sources possess no dependence across datasets .
of particular interest here is to determine the conditions when is identifiable .
for a real - valued single dataset problem , independent sources can be ` blindly ' identified up to a permutation and scaling ambiguity as long as no two sources are gaussian with proportional sample - to - sample correlation matrices ( * ? ? ?
* chapter 4 ) .
the framework has been shown to possess an additional type of diversity which can be exploited for identifying sources that can not be identified by , @xcite .    in this paper ,
a general framework for is presented . by ` general ' we mean an formulation that accounts for dependency between samples , i.e. , when the samples are not . prior to introducing this formulation in section iv
, we give a review of existing algorithms in section ii and define our mathematical conventions and notations in section iii .
naturally , can be achieved by maximizing the likelihood function , which is shown in section v to be the same in practice as minimizing the entropy rate ( subject to a regularity term ) .
the likelihood function has an associated of a form that we describe in section vi .
the is used in deriving the identification conditions and source separation performance bounds in sections vii and viii , respectively .
the identification conditions and performance bounds are generalizations of the results for ( of a single dataset ) .
the case when samples are is shown to have a performance bound that can be expressed compactly for the very large class of multivariate elliptical distributions . in section ix ,
the performance bounds are compared to the performance achieved by two previously published algorithms for . in the last section ,
we discuss directions for future work .
as mentioned previously , the origins of algorithms that can be used for date back to pre- times . in fact , classical @xcite achieves for linearly dependent sources in analysis of two datasets . the formulation of can be shown to serve as a basis for all algorithms reviewed here .
this is because can be derived from two different , but related principles ; maximum likelihood and eigenanalysis ( diagonalization ) . here
, we choose to separate the approaches into three classes for our review based on the source diversity exploited to achieve .
it will be shown that each type of diversity can be utilized
independent of the other two  to achieve .
the first class is applicable to problems in which the sources are assumed to have linear dependence across datasets , but are linearly independent within datasets .
the earliest approaches to extending beyond two datasets are summarized in @xcite and has been termed in @xcite .
the approaches within use cost functions based on second - order statistics that result in solutions that can be widely applied .
another approach to for linearly dependent sources can be derived using equivalently maximum likelihood or minimization of mutual information and results in @xcite .
since can be achieved using generalized eigenvalue decomposition , it can also be posed as a diagonalization problem , which can be readily extended to achieve using ` generalized joint diagonalization ' @xcite . for of linearly dependent sources
the covariance and cross - covariance matrices among the estimated sources in each dataset can be diagonalized as in @xcite .
when the sources possess nonlinear dependence across the datasets then higher - order statistics should be utilized either explicitly or implicitly .
the extension of to nonlinear dependence measures for two datasets dates back to at least 1976 @xcite .
extensions to multiple datasets is given in @xcite .
these early works are summarized in @xcite .
another extension for nonlinear of two datasets uses nonparametric univariate and bivariate density estimators in order to maximize the mutual information between two canonical correlation variates @xcite .
kernels have also been used to transform the random vectors into a ` feature - space ' where linear is then applied @xcite .
a different type of transformation is proposed in @xcite . here
measure transform functions are specified for transforming joint probability measures to identify nonlinearly dependent sources . to use either the kernel or measure transform approaches
, one must determine the appropriate transform and transform parameters to achieve for the problem at hand .
also provides a framework for exploiting nonlinear dependencies .
, as first introduced in @xcite and in the similar work of @xcite , extends to multiple datasets so as to solve the permutation ambiguity problem associated with frequency domain @xcite .
the nonlinear dependencies can be accounted for within the framework by considering non - gaussian sources .
for example , in @xcite , a nonlinear score function consistent with the second - order uncorrelated multivariate laplacian distribution is used .    as is the case for linear dependence , diagonalization methods for of nonlinearly dependent sources can be utilized . specifically , demixing matrices that diagonalize the higher - order statistics ( i.e. , cumulants of order higher than two ) associated with the estimated sources are found @xcite .      naturally for , as for
, algorithms can be developed to exploit sample - to - sample dependence .
a _ generalization _ of joint diagonalization provides such a solution by sampling the vector autocorrelation function at different time lags and finding demixing matrices which minimize correlation between the sources for all time lags , see , e.g. , @xcite .
for this paper , the domains are restricted to the sets of real ( @xmath0 ) and nonnegative natural ( @xmath1 ) numbers .
matrices and vectors from each domain are indicated by @xmath2 , @xmath3 , @xmath4 , and @xmath5 , respectively .
scalar , ( column ) vector , and matrix quantities are denoted as lower - case light face , lower - case bold face , and upper - case bold face , respectively .
the @xmath6th element of a vector @xmath7 , @xmath8_{m}}}$ ] , and an element in the @xmath6th row and @xmath9th column of a matrix @xmath10 , @xmath11_{m , n}}}$ ] , are often denoted @xmath12 and @xmath13 , respectively .
the kronecker delta , @xmath14 , is one when @xmath15 and zero otherwise .
the standard basis vector , @xmath16 , is the the @xmath9th column of identity matrix , @xmath17 .
the @xmath18 and @xmath19 denote matrices ( or vectors ) with all entries of zeros and ones , respectively , where the dimensions of the matrices are either known from the context or indicated by an additional subscript .
the superscript @xmath20 denotes the matrix transpose .
the element - wise ( hadamard ) product , element - wise division , and kronecker products are denoted by @xmath21 , @xmath22 , and @xmath23 , respectively .
we use @xmath24 , where @xmath25 , to compactly denote the the stacking of the columns of @xmath26 .
additionally , if a subset of the rows in @xmath27 are listed in the vector @xmath28^{\ensuremath{\mathsf{t}}}\in { \ensuremath{\mathbb{n}}}^{d}$ ] , where @xmath29 with a corresponding indexing matrix @xmath30 } = \left[{\ensuremath{\mathbf{e}}}_{\alpha_1 } , \ldots , { \ensuremath{\mathbf{e}}}_{\alpha_{d}}\right]^{\ensuremath{\mathsf{t}}}\in { \ensuremath{\mathbb{r}}}^{d \times m}$ ] , then @xmath30 } { \ensuremath{\mathbf{a}}}$ ] selects the subset of rows in @xmath27 indicated by @xmath31 . for compactness
, we use @xmath32 } { \ensuremath{\mathbf{a}}}\right)}}$ ] .
the complementing subset of @xmath33 is indicated by @xmath34 .
a diagonal matrix with entries given by @xmath35 is denoted by @xmath36 .
the square matrix , @xmath27 , has diagonal entries , @xmath37 , a trace , @xmath38_{n}}}$ ] , and a determinant , @xmath39 .
we indicate @xmath40 is positive definite using @xmath41 and positive semidefinite with @xmath42 .
the operator @xmath43 denotes the magnitude .    for a matrix @xmath10 with block structure ,
the matrix @xmath44 is the @xmath6th row and @xmath9th column in the block representation of the matrix @xmath10 using @xmath45 row partitions and @xmath46 column partitions .
the special block diagonal matrix is necessarily a square matrix ( implying @xmath47 ) that has off - diagonal partitions being zero , i.e. , @xmath48 for @xmath49 , and is denoted with the _ direct sum _
notation , @xmath50 , @xcite .
the common functions of random variables such as the expectation operator , entropy , and mutual information are denoted using @xmath51 , @xmath52 , and @xmath53 , respectively .
a random vector @xmath54 following the normal distribution with mean @xmath55 and covariance matrix @xmath56 is denoted @xmath57 .
we use @xmath58 to denote that a random vector @xmath54 is independent of @xmath59 .
we use standard elementary functions such as @xmath60 , @xmath61 , @xmath62 for the natural logarithm , the anti - logarithm , and the complete gamma function .
we begin by formulating the particular framework of interest , namely , in a more general manner than previously done @xcite .
the generalization allows analysis of when the samples are not , or alternatively when sample dependence is taken into account .
there are @xmath63 datasets , each containing @xmath64 samples , formed from the linear mixture of @xmath46 independent sources , @xmath65 } = { \ensuremath{\mathbf{a}}}^{[k]}{\ensuremath{\mathbf{s}}}^{[k ] } \in { \ensuremath{\mathbb{r}}}^{n \times v } ,   1\le k\le k.\end{aligned}\ ] ] the entry in @xmath9th row and @xmath66th column of @xmath67}$ ] is @xmath68}\left(v\right)$ ] , the @xmath9th row of @xmath67}$ ] is denoted with the column vector @xmath69 } = \left[s^{[k]}_n\left(1\right ) , \ldots , s^{[k]}_n\left(v\right)\right]^{\ensuremath{\mathsf{t}}}\in { \ensuremath{\mathbb{r}}}^{v}$ ] , and the @xmath66th column of @xmath67}$ ] is denoted by the column vector @xmath70}\left(v\right ) = \left[s^{[k]}_1\left(v\right ) , \ldots , s^{[k]}_n\left(v\right)\right]^{\ensuremath{\mathsf{t}}}\in { \ensuremath{\mathbb{r}}}^{n}$ ] .
the source matrices in each dataset can be concatenated to form @xmath71}\right)^{\ensuremath{\mathsf{t } } } , \ldots , \left({\ensuremath{\mathbf{s}}}^{[k]}\right)^{\ensuremath{\mathsf{t}}}\right]^{\ensuremath{\mathsf{t}}}\in { \ensuremath{\mathbb{r}}}^{nk \times v}$ ] . using this notation
, we can denote the data model with a single equation , namely @xmath72 , where @xmath73}$ ] .
the invertible mixing matrices , @xmath74 } \in { \ensuremath{\mathbb{r}}}^{n\times n}$ ] , and the sources @xmath75 are unknown real - valued quantities to be estimated .
the @xmath9th , @xmath76 } , \ldots , { \ensuremath{\mathbf{s}}}_{n}^{[k]}\right]^{\ensuremath{\mathsf{t}}}\in { \ensuremath{\mathbb{r}}}^{k \times v}$ ] , is independent of all other .
then the of the concatenated source vector , @xmath75 , can be written as @xmath77 .
the solution finds @xmath63 demixing matrices and the corresponding source estimates for each dataset , with the @xmath78th ones denoted as @xmath79}$ ] and @xmath80 } \triangleq { \ensuremath{\mathbf{w}}}^{[k ] } { \ensuremath{\mathbf{x}}}^{[k]}$ ] , respectively . the estimate of the @xmath9th component from the @xmath66th sample of the @xmath78th dataset is given by @xmath81}\left(v\right ) = \left({\ensuremath{\mathbf{w}}}_{n}^{[k]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{x}}}^{[k]}\left(v\right ) = \sum_{l=1}^{n}w_{n , l}^{[k]}x_{l}^{[k]}\left(v\right)$ ] , where @xmath82}\right)^{{\ensuremath{\mathsf{t}}}}$ ] is the @xmath9th row of @xmath79}$ ] .
furthermore , it is assumed that the mixing matrices possess no known relationship .
just as in , the objective function can be specified to be the maximization of the natural logarithm of the likelihood .
since @xmath27 is block diagonal , the estimate of the @xmath83}$ ] is block diagonal and thus we choose in the sequel to use @xmath84 , i.e. , a three - dimensional ` matrix ' , to denote the set of parameters to be estimated .
we then have that @xmath85}\right| } \label{eq : jbss_likelihood } , \end{aligned}\ ] ] where @xmath86 is the model for the distribution characterizing the multivariate source @xmath87 .
note that if @xmath72 , then @xmath88 , which implies @xmath89 .
if we consider the case when @xmath90 , then we can define the @xmath91 as a random vector process and recall the definition of entropy rate ( * ? ? ?
* eq 4.10 ) so that @xmath92 by normalizing the likelihood objective function by @xmath64 and considering the limit , @xmath93}\right| } \nonumber \\ & = \sum_{n=1}^n \left(\sum_{k=1}^k { \ensuremath{\mathcal{h}_r\left\{y_n^{[k]}\right\ } } } - { \ensuremath{\mathcal{i}_r\left\{{\ensuremath{\mathbf{y}}}_n\right\ } } } \right ) -\sum_{k=1}^k \log{\left| \det { \ensuremath{\mathbf{w}}}^{\left[k\right]}\right| } .
\label{eq : iva_entropyrate}\end{aligned}\ ] ] we can observe that minimizes the entropy rate of the estimated ( subject to the regularization term ) . this representation explains that the objective function will equally weight the minimization of the source entropy rates and the maximization of the across dataset dependence measure provided by the mutual information rate of @xmath94 .
it is also clear that the mutual information rate portion of the objective function is responsible for resolving the permutation ambiguity across multiple datasets , since without the mutual information rate of the the objective function would be identical to using on each of the @xmath63 datasets
. this representation will be useful in our identifiability discussion in section [ sec : ivaidentification ] .    in the sequel
, we will use the multivariate score function @xmath95 and @xmath96}_n = { \ensuremath{\boldsymbol{\phi}}}_n^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{e}}}_k$ ] .
here we derive the of @xmath97 .
the @xmath98 parameters result in @xmath99 dimension with the entry associated with @xmath100}_{m_1,n_1}$ ] and @xmath101}_{m_2,n_2}$ ] denoted by and computed as : @xmath102^{k_1,m_1,n_1}_{k_2,m_2,n_2 } \triangleq   { \ensuremath{e\left\ {   \frac{\partial \mathcal{l}\left({\ensuremath{\boldsymbol{\mathcal{w}}}}\right)}{\partial w^{[k_1]}_{m_1,n_1 } } \frac{\partial \mathcal{l}\left({\ensuremath{\boldsymbol{\mathcal{w}}}}\right)}{\partial w^{[k_2]}_{m_2,n_2 } } \right\ } } } .
\label{eq : fim_w}\end{aligned}\ ] ]    for the purposes of determining identifiability and the performance bound , we need only consider the locally around a solution , i.e. , @xmath103 , where @xmath104 and @xmath105 are  freely " chosen as to alleviate all scale and permutation ambiguities . in general
, this leads to a complex expression that depends on @xmath27 ; fortunately this complexity is unnecessary . due to the invariance of the on @xmath106 the mixing matrix @xmath107}$ ] ,
we need only consider @xmath108 , i.e. , the of @xmath109 depends only on the statistics of the sources , @xcite .
thus the matrix of interest is @xmath110^{k_1,m_1,n_1}_{k_2,m_2,n_2 }   & \triangleq \left .
\left[{\ensuremath{\mathbf{f}}}\left({\ensuremath{\boldsymbol{\mathcal{w}}}}\right)\right]^{k_1,m_1,n_1}_{k_2,m_2,n_2 }   \right|_{{\ensuremath{\mathbf{a } } } = { \ensuremath{\mathbf{i}}},{\ensuremath{\mathbf{w}}}={\ensuremath{\mathbf{i } } } } .\end{aligned}\ ] ]    it will prove useful to define @xmath111 } \triangleq \frac{1}{v } { \ensuremath{e\left\{\left({\ensuremath{\boldsymbol{\phi}}}_{m}^{[k_1]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}^{[k_1]}_{n } \left({\ensuremath{\mathbf{s}}}^{[k_2]}_{n}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\boldsymbol{\phi}}}_{m}^{[k_2]}\right\ } } }   , \ : 1 \leq m , n \leq n$ ] , to describe the form of the block diagonal compactly . in appendix
[ sec : derivefim ] , we show that the first @xmath46 block entries of the are given by @xmath112 and the remaining block entries are defined for @xmath113 as @xmath114\right\ } } } = v \left[\begin{array}{cc } { \ensuremath{\boldsymbol{\mathcal{k}}}}_{m , n } & { \ensuremath{\mathbf{i}}}_k \\ { \ensuremath{\mathbf{i}}}_k        & { \ensuremath{\boldsymbol{\mathcal{k}}}}_{n , m } \end{array}\right ] \label{eq : ivafim_mn } ,   \end{aligned}\ ] ] where the @xmath115 entry of @xmath116 is @xmath117 } { \ensuremath{\mathbf{r}}}_{n}^{[k_1,k_2]}\right)}}$ ] when @xmath118 , @xmath119 } \triangleq { \ensuremath{e\left\{{\ensuremath{\mathbf{s}}}_n^{[k_1 ] } \left({\ensuremath{\mathbf{s}}}_n^{[k_2]}\right)^{\ensuremath{\mathsf{t}}}\right\ } } } \in { \ensuremath{\mathbb{r}}}^{v \times v}$ ] , and @xmath120 }   \triangleq   { \ensuremath{e\left\{{\ensuremath{\boldsymbol{\phi}}}^{[k_1]}_n \left({\ensuremath{\boldsymbol{\phi}}}^{[k_2]}_n\right)^{\ensuremath{\mathsf{t}}}\right\ } } } \in { \ensuremath{\mathbb{r}}}^{v \times v}$ ] .    the form of the is a multivariate extension of the single dataset forms given in @xcite .
the has a form that is a block matrix version of the single dataset result , e.g. , see fig .
[ fig : fig_fim_n3 ] and compare to the similar form given in @xcite for complex - valued .
the @xmath121 blocks with ones in the off - diagonal elements and pair - wise cross terms in the two diagonal elements of the are here replaced with @xmath121 block matrices with identity matrices in the off - diagonal blocks and the cross terms in the two diagonal block matrices , i.e. , @xmath122 .
the identification of sources in ( real - valued ) is possible so long as no two sources are gaussian with proportional covariance matrices ( * ? ? ?
* chapter 4 ) .
when sources are said to be identifiable for , this means that the sources can be recovered up to a scale factor and arbitrary ordering , i.e. , the true mixing matrix @xmath123 can be identified upto @xmath124 , where @xmath125 is any nonsingular diagonal matrix and @xmath126 is any permutation matrix .    since the the model structure of is a generalization of the model structure for , we expect a generalization of the identification conditions for .
intuitively , the identification conditions for are related to the dependence of the sources across the datasets .
more specifically , when sources possess dependence across datasets we expect that these estimated sources can be ` aligned'this is the original motivation of @xcite . however , if there are sources for which no alignment exhibits dependence , then under the identification conditions sources can be separated but not necessarily aligned
. that is , without dependence across datasets the estimated sources of would be no different than using on each dataset individually since there is no dependency to exploit .
the identification conditions , which we present in this section , capture both cases , i.e. , when there is or is not dependence between sources across datasets .    to discuss identifiability of
, we need to provide a notation that allows us to indicate a particular subset of rows in an . for this section ,
we let @xmath127^{\ensuremath{\mathsf{t}}}\in { \ensuremath{\mathbb{n}}}^{k_\alpha}$ ] , where @xmath128 .
the complementing subset of @xmath33 in @xmath129 is indicated by @xmath130 .
the identification conditions use the following definition :    a source , @xmath131 , has an _
@xmath33-gaussian _ component when @xmath132 , and @xmath133 , where @xmath134 is nonsingular .
the @xmath33-gaussian definition is used to identify that there exist a subset of rows in an that is independent of the other rows in the same and that the given subset follows a multivariate gaussian distribution .
the theorem stating the identification conditions and its proof follow .
[ thm : iva_id ] the sources can not be identified @xmath135 and @xmath136 such that @xmath137 and @xmath87 have _ @xmath33-gaussian _ components for which @xmath138 , where @xmath139 is any full rank diagonal matrix .
given the , , , since @xmath122 is a covariance matrix , it must be positive semidefinite and is singular @xmath140 , where @xmath141 denotes the sample space of the random matrix @xmath142 .
it is convenient to rewrite the following : @xmath143 where @xmath144 and @xmath145 denote the @xmath66th columns of @xmath87 and @xmath146 , respectively .
hence , the following statements are all equivalent conditional on @xmath147 @xmath148}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_n\right ) } } - { \ensuremath{\mathrm{vec}^{\ensuremath{\mathsf{t}}}_{{\ensuremath{\boldsymbol{\beta}}}}\left({\ensuremath{\boldsymbol{\phi}}}_n\right ) } } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\beta } } } \right]}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\beta}}}}\left({\ensuremath{\mathbf{s}}}_m\right ) } } \label{eq : jbssidproof - da } \\ & \leftrightarrow & 0 & = { \ensuremath{\mathrm{vec}^{\ensuremath{\mathsf{t}}}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\boldsymbol{\phi}}}_m\right ) } } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{a}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_n\right ) } } - { \ensuremath{\mathrm{vec}^{\ensuremath{\mathsf{t}}}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\boldsymbol{\phi}}}_n\right ) } } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_m\right ) } } \label{eq : jbssidproof - d } \\ & \leftrightarrow & 0 & = { \ensuremath{\mathrm{vec}^{\ensuremath{\mathsf{t}}}_{{\ensuremath{\boldsymbol{\alpha } } } } \left({\ensuremath{\mathbf{s}}}_m\right ) } } { \ensuremath{\mathbf{r}}}_{m,\alpha}^{-1 } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{a}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_n\right ) } } - { \ensuremath{\mathrm{vec}^{\ensuremath{\mathsf{t}}}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_n\right ) } } { \ensuremath{\mathbf{r}}}_{n,\alpha}^{-1 } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_m\right ) } }   \label{eq : jbssidproof - e } \\ & \leftrightarrow & 0 & =   { \ensuremath{\mathbf{r}}}_{m,\alpha}^{-1 } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{a}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right )   -   \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) { \ensuremath{\mathbf{r}}}_{n,\alpha}^{-1 } \label{eq : jbssidproof - f } \\ & \leftrightarrow &   { \ensuremath{\mathbf{r}}}_{m,\alpha } & = \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{a}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right )   { \ensuremath{\mathbf{r}}}_{n,\alpha } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}^{-1}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) \label{eq : jbssidproof - g } \\ & \leftrightarrow &   { \ensuremath{\mathbf{r}}}_{m,\alpha } & =   \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}\right )   { \ensuremath{\mathbf{r}}}_{n,\alpha } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}\right ) \label{eq : jbssidproof - h } , \end{aligned}\ ] ] where @xmath149 } \triangleq { \ensuremath{\mathrm{diag}\left({\ensuremath{\mathbf{a}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]\right)}}$ ] , @xmath150 } \triangleq { \ensuremath{\mathrm{diag}\left({\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]\right)}}$ ] , @xmath151 , and @xmath152 .
it is straightforward to observe that , , , and are equivalent expressions . from the relationship @xmath153 , the expression in holds only when @xmath154 , i.e. , the zero entries of @xmath155 and @xmath156 are at the same locations .
see lemma [ lem : jbss ] below to explain .
since must hold for all possible values of @xmath157 and @xmath158 , must hold .
equation is equivalent since all entries of @xmath159 $ ] are nonzero by .
lastly , since @xmath160 is symmetric we must have that either @xmath161 is diagonal or @xmath149 } =   \left({\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right)^{-1}$ ] . in either case holds .
[ lem : jbss ] for @xmath118 , @xmath162}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_n\right ) } }   & = { \ensuremath{\mathrm{vec}^{\ensuremath{\mathsf{t}}}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\boldsymbol{\phi}}}_n\right ) } } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_m\right ) } } \label{eq : lem_jbss_1}\end{aligned}\ ] ] holds @xmath163}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_n\right ) } } & = { \ensuremath{\mathrm{vec}^{\ensuremath{\mathsf{t}}}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_n\right ) } } { \ensuremath{\mathbf{r}}}_{n,\alpha}^{-1 } \left({\ensuremath{\mathbf{i}}}_v { \ensuremath{\otimes}}{\ensuremath{\mathbf{d}}}_{{\ensuremath{\mathbf{b}}}\left[{\ensuremath{\boldsymbol{\alpha } } } \right]}\right ) { \ensuremath{\mathrm{vec}_{{\ensuremath{\boldsymbol{\alpha}}}}\left({\ensuremath{\mathbf{s}}}_m\right ) } }   \label{eq : lem_jbss_2}\end{aligned}\ ] ] and @xmath137 and @xmath87 each have an _
@xmath33-gaussian _ component .
@xmath164 since the left - hand side of is linear in @xmath165 we must have that @xmath166 is not a function of @xmath167 and it is necessarily linear in @xmath165 , i.e. , @xmath87 has _
@xmath33-gaussian _ component . by symmetry ,
the same can be concluded about @xmath137 .
@xmath168 if @xmath87 has _
component then @xmath169 .
it is noteworthy to mention that the identification conditions admit sources for which the distribution can be factored , i.e. , @xmath170 , where @xmath171 , @xmath172 , @xmath173 , and @xmath174 .
if , for example @xmath175 , then would produce the same identification conditions as on each dataset individually .
stated differently , identifiability of does not require the sources to possess dependence across datasets .
recalling that a prime motivation for considering the formulation is to determine when the sources can be aligned in a common way across all datasets , i.e. , under what conditions is @xmath176 } = \left({\ensuremath{\mathbf{w}}}^{[k]}\right)^{-1 } = { \ensuremath{\mathbf{a}}}^{[k ] } { \ensuremath{\mathbf{p } } } { \ensuremath{\boldsymbol{\lambda}}}^{[k]}$ ] , where @xmath177}$ ] is any full rank diagonal matrix and @xmath178 is a permutation matrix commonly shared by all datasets .
the common permutation identification condition is given in the next theorem which uses the following definition :    a source , @xmath131 , is _ @xmath33-independent _ when @xmath132 .
the @xmath33-independent definition is used to identify that there exist a subset of rows in an ( or ) that is independent of the other rows in the ( ) .
[ thm : iva_permutation ] assuming the identification conditions of theorem [ thm : iva_id ] are satisfied , i.e. , in the limit as @xmath90 so that @xmath179}\right)^{-1 } = { \ensuremath{\mathbf{a}}}^{[k ] } { \ensuremath{\mathbf{p}}}^{[k ] } { \ensuremath{\boldsymbol{\lambda}}}^{[k]}$ ] : + the permutation matrix associated with each dataset is common @xmath180 such that both @xmath181 and @xmath91 are _
the objective function given in makes it clear that any permutation matrix at most effects the @xmath182 term .
furthermore , we only need consider permutation matrices that can achieve the global minimum . the proof is by contradiction ( in both directions ) : @xmath183 }
\neq { \ensuremath{\mathbf{p}}}^{[k_2 ] } \\ & \leftrightarrow &   { \ensuremath{\mathcal{i}_r\left\{{\ensuremath{\mathbf{s}}}_m^{[{\ensuremath{\boldsymbol{\alpha } } } ] } ; { \ensuremath{\mathbf{s}}}_n^{[{\ensuremath{\boldsymbol{\alpha}}}^c]}\right\ } } } + { \ensuremath{\mathcal{i}_r\left\{{\ensuremath{\mathbf{s}}}_n^{[{\ensuremath{\boldsymbol{\alpha } } } ] } ; { \ensuremath{\mathbf{s}}}_m^{[{\ensuremath{\boldsymbol{\alpha}}}^c]}\right\ } } }   & =   { \ensuremath{\mathcal{i}_r\left\{{\ensuremath{\mathbf{s}}}_m^{[{\ensuremath{\boldsymbol{\alpha } } } ] } ; { \ensuremath{\mathbf{s}}}_m^{[{\ensuremath{\boldsymbol{\alpha}}}^c]}\right\ } } } + { \ensuremath{\mathcal{i}_r\left\{{\ensuremath{\mathbf{s}}}_n^{[{\ensuremath{\boldsymbol{\alpha } } } ] } ; { \ensuremath{\mathbf{s}}}_n^{[{\ensuremath{\boldsymbol{\alpha}}}^c]}\right\ } } }   \\ & \leftrightarrow & 0   & = { \ensuremath{\mathcal{i}_r\left\{{\ensuremath{\mathbf{s}}}_m^{[{\ensuremath{\boldsymbol{\alpha } } } ] } ; { \ensuremath{\mathbf{s}}}_m^{[{\ensuremath{\boldsymbol{\alpha}}}^c]}\right\ } } } + { \ensuremath{\mathcal{i}_r\left\{{\ensuremath{\mathbf{s}}}_n^{[{\ensuremath{\boldsymbol{\alpha } } } ] } ; { \ensuremath{\mathbf{s}}}_n^{[{\ensuremath{\boldsymbol{\alpha}}}^c]}\right\ } } }   \\ & \leftrightarrow &   { \ensuremath{\mathbf{s}}}_m \text { and } { \ensuremath{\mathbf{s}}}_n & \text { are $ { \ensuremath{\boldsymbol{\alpha}}}$-independent}\end{aligned}\ ] ] we have used the fact that @xmath184 with equality @xmath185 , which implies by the assumption of that @xmath186 } ; { \ensuremath{\mathbf{s}}}_j^{[{\ensuremath{\boldsymbol{\alpha}}}_2]}\right\ } } } = 0 \ \forall
i \neq j , { \ensuremath{\boldsymbol{\alpha}}}_1 , \ { \ensuremath{\boldsymbol{\alpha}}}_2 $ ] , where @xmath187 and @xmath188 are any indexing sets .
thus , theorem [ thm : iva_permutation ] provides an additional restriction on the sources ( in a pairwise manner ) which is required when the estimated dependent sources across all datasets are to be ` aligned ' .
it is now insightful to consider important special cases of with regard to the identification conditions .
we begin by considering the case when the @xmath64 samples are .
this is equivalent to having @xmath189 , which implies that the identification conditions can be derived as a special case of theorem [ thm : iva_id ] .
[ thm : ivaiid_id ] the sources can not be identified @xmath135 and @xmath136 such that @xmath181 and @xmath91 have _ @xmath33-gaussian _ components and @xmath190 , where @xmath191 is any full rank diagonal matrix .
another special case of interest is when @xmath192 , yielding the same formulation as assuming sample - to - sample dependence , i.e. , not samples , the most general form for real - valued .
[ thm : ica_id ] the sources can not be identified @xmath136 such that @xmath193 and @xmath194 are _ gaussian _ and @xmath195 , where @xmath196 .
it can be verified that the identification conditions of theorem [ thm : ica_id ] are consistent with the results found in ( * ? ? ?
* chapter 4 ) and @xcite .
another special case of interest is when @xmath192 , and assuming samples .
[ thm : ica_id_iid ] the sources can not be identified @xmath136 such that @xmath197 and @xmath198 are gaussian .
the claim of theorem [ thm : ica_id_iid ] , originally given in @xcite , states the well known result for that at most one source can be gaussian for identification of all sources .
algorithms based on the assumption using higher - order statistics have been the most widely exploited type of diversity in the derivation of algorithms .
additional _ diversity _ can extend the and identification conditions .
an example is when data is complex - valued , a case we do not consider in this paper .
the associated with the parameter vector @xmath199 is the inverse of the , i.e. , @xmath200 , where @xmath201 is an estimator for @xmath199 . due to the block diagonal structure of we
have that the inverse ( if it exists , see identifiability discussion in section [ sec : ivaidentification ] ) of the portion of the associated with the @xmath6th and @xmath9th source denoted by @xmath122 in is @xmath202 . \ ] ]    it yields the following on the estimates of the demixing matrix quantities , @xmath203}\right\ } } } & \geq \frac{1}{v } \mathbf{e}_k^{\ensuremath{\mathsf{t}}}\left({\ensuremath{\boldsymbol{\mathcal{k}}}}_{m , n } - { \ensuremath{\boldsymbol{\mathcal{k}}}}_{n , m}^{-1}\right)^{-1 } \mathbf{e}_k , \
1 \leq m \neq n \leq n.\end{aligned}\ ] ]    for this formulation , the definition of the is the same as in @xcite , namely : @xmath204 } & \triangleq { \ensuremath{e\left\{\left(g_{m , n}^{[k]}\right)^2\right\ } } } \frac{{\ensuremath{e\left\{{\left| { \ensuremath{\mathbf{s}}}_n^{[k]}\right|}^2\right\}}}}{{\ensuremath{e\left\{{\left| { \ensuremath{\mathbf{s}}}_m^{[k]}\right|}^2\right\ } } } } , \ 1 \leq
m \neq n \leq n,\end{aligned}\ ] ] where @xmath205 } = { \ensuremath{\mathbf{e}}}_m^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{g}}}^{[k ] } { \ensuremath{\mathbf{e}}}_n$ ] and @xmath206 } \triangleq { \ensuremath{\mathbf{w}}}^{[k ] } { \ensuremath{\mathbf{a}}}^{[k]}$ ] is called the @xmath78th global demixing - mixing matrix .
the for is then : @xmath204 } \geq \frac{1}{v } \mathbf{e}_k^{\ensuremath{\mathsf{t}}}\left({\ensuremath{\boldsymbol{\mathcal{k}}}}_{m , n } - { \ensuremath{\boldsymbol{\mathcal{k}}}}_{n , m}^{-1}\right)^{-1 } \mathbf{e}_k \frac{{\ensuremath{e\left\{{\left| { \ensuremath{\mathbf{s}}}_n^{[k]}\right|}^2\right\}}}}{{\ensuremath{e\left\{{\left| { \ensuremath{\mathbf{s}}}_m^{[k]}\right|}^2\right\}}}}. \ ] ]    since the _ sources _ are ( potentially ) multivariate in the formulation , it makes sense to define the according to @xmath207 } , \
m \neq n \leq n.\end{aligned}\ ] ]    after some simple manipulation , the following compact form for the results : @xmath208 where @xmath209 . in what follows , for notational simplicity and without loss of generality
, we assume the sources have equal energy within each dataset , i.e. , @xmath210 .    when the samples are , then the simplifies further if we note that : @xmath211 } = { \ensuremath{e\left\{{\ensuremath{\mathbf{s}}}_n^{[k_1 ] } \left({\ensuremath{\mathbf{s}}}_n^{[k_2]}\right)^{\ensuremath{\mathsf{t}}}\right\ } } }   =   \sigma^{[k_1,k_2]}_n { \ensuremath{\mathbf{i}}}_{v},\end{aligned}\ ] ] @xmath212 }     =     { \ensuremath{e\left\{{\ensuremath{\boldsymbol{\phi}}}^{[k_1]}_m \left({\ensuremath{\boldsymbol{\phi}}}^{[k_2]}_m\right)^{\ensuremath{\mathsf{t}}}\right\ } } } =   \gamma^{[k_1,k_2]}_m { \ensuremath{\mathbf{i}}}_{v},\end{aligned}\ ] ] and for @xmath213 , @xmath214 }   = \frac{1}{v } { \ensuremath{\mathrm{tr}\left({\ensuremath{\boldsymbol{\gamma}}}_{m}^{[k_2,k_1 ] } { \ensuremath{\mathbf{r}}}_{n}^{[k_1,k_2]}\right ) } } =   \gamma^{[k_1,k_2]}_m \sigma^{[k_1,k_2]}_n   , \end{aligned}\ ] ] where @xmath215}_n \triangleq { \ensuremath{e\left\{s_n^{[k_1]}\left(v\right ) s_n^{[k_2]}\left(v\right)\right\ } } } \in { \ensuremath{\mathbb{r}}}$ ] and @xmath216}_m \triangleq { \ensuremath{e\left\{\phi^{[k_1]}_{m}\left(v\right ) \phi^{[k_2]}_{m}\left(v\right)\right\ } } } \in { \ensuremath{\mathbb{r}}}$ ] are not dependent on @xmath66 due to the assumption .    for the discussion we simplify by replacing the notation with notation , i.e.
, we define the , @xmath91 , as a random vector with @xmath64 realizations denoted by @xmath217 .
in addition , the multivariate score function is denoted by @xmath218 . for now , let @xmath219 and @xmath220 , from which we observe that @xmath221 .
the above gives the following on the estimates of the demixing matrix entries when the samples are , @xmath222    the relationship between @xmath223 and @xmath224 given in the following lemma is the multivariate extension of the result given by ( * ? ? ?
* lemma 1b of appendix b ) , which has also been given in ( * ? ? ?
* chapter 4 ) .
[ lem : gammavsinvcov ] @xmath225 , with equality @xmath226 , i.e. , @xmath227 follows the gaussian distribution .
the proof applies the extension of the cauchy - schwarz inequality for covariance matrices as given in @xcite .
specifically , @xmath228 , with equality @xmath229 . by noting that @xmath230 we arrive at the assertion .    from this lemma , we see that a measure of non - gaussianity ( or higher - order statistics ) is captured by the ` difference ' between @xmath223 and @xmath231 .
next , we show for elliptical distributions  a broad class of source distributions  how this non - gaussianity measure can be captured by a scalar quantity .
the ( assuming it exists ) for a zero - mean random vector following the elliptical distribution is @xmath232 where @xmath233 is the positive definite matrix frequently termed the dispersion matrix , @xmath234 is some nonnegative function , and @xmath235 denotes the constant that makes integrate to one .
if the covariance matrix , @xmath236 , exists , then for any elliptical distribution it is a scalar multiple of the dispersion matrix , i.e. , @xmath237 , where @xmath238 .
then the score function , @xmath239 , where @xmath240 .    for elliptical distributions ( see appendix [ sec : ellipticalgammamx ] ) , @xmath241 , where @xmath242 . by application of lemma [ lem : gammavsinvcov ]
this implies that @xmath243 with equality gaussian . ] .
therefore , the for with elliptical sources is @xmath244 for this performance bound we provide the following theorem .
[ thm : isrbound ] if two follow distributions from the elliptical family with covariance matrices , @xmath245 and @xmath246 , then @xmath247 is less than or equal to the @xmath247 associated with gaussian having the same covariance matrices .
see @xcite for proof that @xmath248 . for elliptically distributed sources , via lemma [ lem : gammavsinvcov ] , we have that @xmath249 and @xmath250 , thus @xmath251 and since @xmath252 , it implies @xmath253 , and thus @xmath254 .
a special case , which arrives at a form directly analogous to the form , occurs when @xmath255 : @xmath256 this expression clearly shows how for second - order uncorrelated elliptical sources , the ` degree ' of non - gaussianity as expressed by @xmath257 , directly determines the source separation performance .
in fact , as shown in the following theorem , the same statement holds for second - order correlated elliptical sources .
[ thm : isrvskappa ] if three follow distributions from the elliptical family with covariance matrices , @xmath258 , and @xmath246 , and @xmath259 then @xmath260 .    for elliptically distributed sources , via lemma
[ lem : gammavsinvcov ] , we have that @xmath261 and @xmath250 , thus @xmath262 and since @xmath252 implies @xmath253 , and thus @xmath254 .      another special case , which is of particular interest , is when there is only one dataset , i.e. , @xmath192 .
for this case , the expressions above further simplify to the more extensively studied performance bounds @xcite .
if @xmath192 , we can replace the notation with source component notation , i.e. , let @xmath263 be the random vector and the multivariate score function be denoted by @xmath264 . then , for this section we have @xmath265 and @xmath266 , from which we observe that for @xmath118 , @xmath267 . also , @xmath268
.\end{aligned}\ ] ] two particular subcases in are of interest .
the first case is when the samples are with unit variance , for which @xmath269 , @xmath270 , and @xmath271 , where @xmath272 .
these simplifications give the same results as in ( * ? ? ?
38 ) and ( * ? ? ?
2 ) , namely : @xmath273    the second subcase of is for sources with gaussian sample - to - sample dependence , i.e. , @xmath274
. then we have that @xmath275 and @xmath276 , which corresponds to ( * ? ? ?
in this section , we compare the performance of several algorithms versus the given in section [ sec : crlb ] .      for our first set of experiments , we consider sources following the distribution , an elliptical distribution with @xmath277 and normalization constant @xmath278 , where @xmath279 is termed the shape parameter .
this distribution possesses a score function which includes the score functions used in both @xcite and @xcite as special cases . in this section
, we consider , where the algorithm was presented in @xcite , using simulated datasets with samples from the family .
the performance of is compared with the derived in section [ sec : crlb ] .    for this experiment , there are @xmath280 of dimension @xmath281 .
all the sources use the same shape parameter , @xmath282 . the covariance matrix associated with each source
is randomly picked for the experiment , yet fixed for all trials in the experiment .
the @xmath78th entry of each scv is used as a latent source for the @xmath78th dataset .
entries of the random mixing matrices , @xmath74}$ ] , are from the standard normal distribution and are randomly selected for each trial .
we compute the theoretical for and compare this value with the achieved using with the correct shape parameter for each source .
we then compute the total theoretical normalized , defined as , @xmath283 we compare this theoretical with the average computed from @xmath284 independent trials of the algorithm as we vary the number of samples per dataset , @xmath64 .
due to the presence of local minima in the objective function for non - gaussian sources @xcite , the algorithm may converge to local minima . at local minima ,
the sources are separated within a dataset but the scvs are not successfully identified , i.e. , the permutation ambiguity is unresolved .
we first compare the for the with the mean of the achieved over _ successful _ trials .
a trial is deemed successful if the location of the maximum absolute entry in each row of @xmath206 } = { \ensuremath{\mathbf{w}}}^{[k ] } { \ensuremath{\mathbf{a}}}^{[k]}$ ] is unique within each dataset and colocated across the datasets ( the former indicates sources are separated within each dataset and the latter indicates if the permutation ambiguity is resolved ) . the fraction of trials which are successful increases as @xmath282 decreases and/or as the sample size per dataset increases .
the lowest success rate was 98% , when @xmath285 and @xmath286 .
for all other settings the success rate was greater than 99.5% . from fig .
[ fig : fig_mpevsshape_corr_isr ] , the performance of the algorithm approaches the as the sample size per dataset increases .
we also show in fig .
[ fig : fig_mpevsshape_corr_2shapes_isr]for the same experiment described above  the performance of the when the algorithm selects between one of two shape parameters ( @xmath287 ) according to which shape parameter provides the lowest cost .    , and
thus does not use exact knowledge of the shape parameter .
all results are compared with the .
, width=355 ]    in another experiment , we use the same parameters as before except now the each have identity covariance matrices . for this experiment , there are nonidentifiable conditions as @xmath288 , thus we compare the for the with the median rather than the mean . from fig .  [
fig : mpe_crlb_vs_shape_uncorr_varyt ] , the performance of the algorithm approaches the as the sample size per dataset increases .
, varies is compared with the median of all @xmath284 trials for different numbers of samples , @xmath64.,width=355 ]    in both fig .
[ fig : fig_mpevsshape_corr_isr ] and fig .
[ fig : mpe_crlb_vs_shape_uncorr_varyt ] , the follows the behavior predicted by theorems [ thm : ivaiid_id ] , [ thm : isrbound ] , and [ thm : isrvskappa ] .
namely , the is infinite when sources are gaussian and @xmath289 for all sources ; the maximum occurs when sources are gaussian ( @xmath290 ) ; and as @xmath282 moves ` away ' from one the non - gaussianity measure @xmath257 increases , which yields better source separation , i.e. , lower .      in this section ,
we consider the effect of sample dependency . to the best of our knowledge , there is only one algorithm in the framework that accounts for sample - to - sample dependence , namely as given in @xcite .
the performance of is compared with the derived in section [ sec : crlb ] .
all the sources are a vector moving average of gaussian samples , i.e. , @xmath291 where @xmath292 and @xmath293_{k_1,k_2 } } } \sim { \ensuremath{\mathcal{n}\left(0,1\right)}}$ ] . for this experiment , there are @xmath280 sources for @xmath294 datasets , each with @xmath295 samples and @xmath296
. entries of the random mixing matrices , @xmath74}$ ] , are from the standard normal distribution and are randomly selected for each trial .
we compute the theoretical for assuming the data was generated with @xmath297 . since @xmath296 for the data , the performance bound is shown to decrease until the lag is 3 .
the performance bound for @xmath296 is shown for lags greater than 3 .
we compare the performance bounds with the average over @xmath298 independent trials of the achieved using with various lags . due to estimating orthogonal demixing matrices there exists a noticeable difference between the for and the observed .    ) . the is shown assuming at most lag = 3.,height=259 ]
the use of for the separation of multiple datasets concurrently has been a more recent development within the general literature .
a variety of algorithms have been developed that are essentially the multivariate extensions of algorithms which take into account the dependence of sources between datasets in a variety of ways .
there are three principal reasons for using these algorithms ( versus just using individually on each dataset ) .
first , to increase the set of sources which can be identified .
second , to automatically ` align ' dependent sources .
third , to maximize the achievable source separation . in this work ,
we have given the larger set of sources which can be identified by , proven when the estimated sources can be ` aligned ' , and provided the bound on achievable source separation using .
these results are achieved for an that accounts for linear and nonlinear dependence of sources across datasets , non - gaussianity , and sample - to - sample dependence .
it is clear that bridges the gap between and
.    it will be interesting for future work to consider the additional diversity of complex - valued sources which are improper or noncircular .
additionally , our work will be useful for assessing the performance of future algorithms which account for sample dependency in an framework .
here we derive the of @xmath97 .
the @xmath98 parameters result in @xmath99 dimension with the entry associated with @xmath100}_{m_1,n_1}$ ] and @xmath101}_{m_2,n_2}$ ] given by .    for the computations to follow it is useful to observe that , @xmath299}\right|}}{\partial w^{[k]}_{m , n } } & = \delta_{l , k } { \ensuremath{\mathbf{e}}}_m^{\ensuremath{\mathsf{t}}}\left({\ensuremath{\mathbf{w}}}^{[k]}\right)^{-{\ensuremath{\mathsf{t } } } } { \ensuremath{\mathbf{e}}}_n\end{aligned}\ ] ]    @xmath300}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{w}}}_m^{[1 ] } , \ldots , \left({\ensuremath{\mathbf{x}}}^{[k]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{w}}}_m^{[k ] } \right ] \in { \ensuremath{\mathbb{r}}}^{k \times v},\end{aligned}\ ] ]    @xmath301}_{m , n } } & =   \delta_{l , m}{\ensuremath{\mathrm{diag}\left({\ensuremath{\mathbf{e}}}_k\right)}}{\ensuremath{\mathbf{x}}}_n \in { \ensuremath{\mathbb{r}}}^{k \times v},\end{aligned}\ ] ]    and @xmath302}_{m , n } }    &   = { \ensuremath{\mathrm{tr}\left ( \frac{\partial \log \left(p_{m}\left({\ensuremath{\mathbf{y}}}_m\right)\right ) } { \partial { \ensuremath{\mathbf{y}}}_m^{\ensuremath{\mathsf{t } } } }   \frac{\partial{\ensuremath{\mathbf{y}}}_m}{\partial w^{[k]}_{m , n } } \right ) } } \label{eq : partiallogp_dw_step1 } \\ &   = -{\ensuremath{\mathrm{tr}\left({\ensuremath{\boldsymbol{\phi}}}_m^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathrm{diag}\left({\ensuremath{\mathbf{e}}}_k\right)}}{\ensuremath{\mathbf{x}}}_n\right ) } } \\ &   = -\left ( { \ensuremath{\boldsymbol{\phi}}}^{[k]}_m \right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{x}}}^{[k]}_n   .\end{aligned}\ ] ] note that is due to applying the chain rule given in ( * ? ? ?
thus the gradient of the likelihood function in is @xmath303}_{m , n } } = -\left({\ensuremath{\boldsymbol{\phi}}}_m^{[k]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{x}}}^{[k]}_n + v w_{n , m}^{-[k]},\end{aligned}\ ] ] where @xmath304}_{m , n}$ ] is the entry in the @xmath6th row and @xmath9th column @xmath179}\right)^{-1}$ ] .
letting @xmath305 we have the of interest with entries given by @xmath306^{k_1,m_1,n_1}_{k_2,m_2,n_2 }    \triangleq & \left .
\left[{\ensuremath{\mathbf{f}}}\left({\ensuremath{\boldsymbol{\mathcal{w}}}}\right)\right]^{k_1,m_1,n_1}_{k_2,m_2,n_2 }   \right|_{{\ensuremath{\mathbf{a } } } = { \ensuremath{\mathbf{i } } } , { \ensuremath{\mathbf{w}}}={\ensuremath{\mathbf{i } } } } \\   = & { \ensuremath{e\left\{\left(({\ensuremath{\boldsymbol{\phi}}}_{m_1}^{[k_1]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}^{[k_1]}_{n_1 } \left({\ensuremath{\mathbf{s}}}^{[k_2]}_{n_2}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\boldsymbol{\phi}}}_{m_2}^{[k_2]}\right\ } } } + v^2 \delta_{m_1,n_1 } \delta_{m_2,n_2 } \\ & - v { \ensuremath{e\left\{\left({\ensuremath{\boldsymbol{\phi}}}_{m_2}^{[k_2]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}^{[k_2]}_{n_2}\right\ } } } \delta_{m_1,n_1 } \\ & - v { \ensuremath{e\left\{\left({\ensuremath{\boldsymbol{\phi}}}_{m_1}^{[k_1]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}^{[k_1]}_{n_1}\right\ } } } \delta_{m_2,n_2 } \\ = &   { \ensuremath{e\left\{\left({\ensuremath{\boldsymbol{\phi}}}_{m_1}^{[k_1]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}^{[k_1]}_{n_1 } \left({\ensuremath{\mathbf{s}}}^{[k_2]}_{n_2}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\boldsymbol{\phi}}}_{m_2}^{[k_2]}\right\ } } } - v^2 \delta_{m_1,n_1 } \delta_{m_2,n_2 } , \end{split}\end{aligned}\ ] ] where the following expression holds , @xmath307}_{n } \left({\ensuremath{\boldsymbol{\phi}}}_{m}^{[k_2]}\right)^{\ensuremath{\mathsf{t}}}\right\ } } } = \delta_{k_1,k_2 } \delta_{m , n } { \ensuremath{\mathbf{i}}}_v$ ] , see @xcite . since , by assumption , both @xmath308}\right\ } } } = { \ensuremath{\mathbf{0}}}$ ] and @xmath309}\right\ } } } = { \ensuremath{\mathbf{0}}}$ ] , then it is true that @xmath310^{k_1,m_1,n_1}_{k_2,m_2,n_2}=0 $ ] when one of the entries in @xmath311 is unique .
it is also zero when @xmath312 , i.e. , @xmath313}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}^{[k_1]}_{m_1}\right\ } } } { \ensuremath{e\left\{\left({\ensuremath{\mathbf{s}}}^{[k_2]}_{m_2}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\boldsymbol{\phi}}}_{m_2}^{[k_2]}\right\ } } } - v^2 = v^2-v^2 = 0 $ ] .
thus , there are only three nonzero cases to consider : @xmath110^{k_1,m_1,n_1}_{k_2,m_2,n_2 } & = \begin{cases } v \left(\mathcal{k}_{m_1,m_1}^{[k_1,k_2 ] }   - v\right ) & m_1 = n_2 = m_2 = n_1 \\ v \mathcal{k}_{m_1,n_1}^{[k_1,k_2 ] }    & m_1=m_2 \neq n_1 = n_2 \\ v \delta_{k_1,k_2 }   & m_1 = n_2 \neq m_2 = n_1 \\ 0 & \textrm{otherwise } , \end{cases}\end{aligned}\ ] ] where @xmath111 } \triangleq \frac{1}{v } { \ensuremath{e\left\{\left({\ensuremath{\boldsymbol{\phi}}}_{m}^{[k_1]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}^{[k_1]}_{n } \left({\ensuremath{\mathbf{s}}}^{[k_2]}_{n}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\boldsymbol{\phi}}}_{m}^{[k_2]}\right\ } } } = \frac{1}{v } { \ensuremath{\mathrm{tr}\left({\ensuremath{e\left\ { { \ensuremath{\boldsymbol{\phi}}}_{m}^{[k_2 ] } \left({\ensuremath{\boldsymbol{\phi}}}_{m}^{[k_1]}\right)^{\ensuremath{\mathsf{t}}}{\ensuremath{\mathbf{s}}}_{n}^{[k_1 ] } \left({\ensuremath{\mathbf{s}}}^{[k_2]}_{n}\right)^{\ensuremath{\mathsf{t}}}\right\}}}\right)}}$ ] is the @xmath115 entry of @xmath314 .
the form of this matrix ( e.g. , see fig .  [
fig : fig_fim_n3 ] ) is the block - matrix extension of that for the single dataset given in @xcite .
all entries are of @xmath315 matrices and we use @xmath316 denote zero blocks .
the entries of associated with @xmath317 , @xmath318 , and @xmath319 are indicated by blue , green , and red , respectively.,width=336 ]    there exists a permuted in which there are @xmath320 nonzero matrices along the diagonal , i.e. , @xmath321 .
\label{eq : app : ivafim}\end{aligned}\ ] ] the submatrices are given by @xmath322 and @xmath114\right\ } } } = v \left[\begin{array}{cc } { \ensuremath{\boldsymbol{\mathcal{k}}}}_{m , n } & { \ensuremath{\mathbf{i}}}_k \\ { \ensuremath{\mathbf{i}}}_k        & { \ensuremath{\boldsymbol{\mathcal{k}}}}_{n , m } \end{array}\right ]   \label{eq : app : ivafim_mn},\end{aligned}\ ] ] where @xmath323 and @xmath324 .
it is also useful to note that for @xmath325 we have @xmath326 }   = \frac{1}{v } { \ensuremath{\mathrm{tr}\left({\ensuremath{\boldsymbol{\gamma}}}_{m}^{[k_2,k_1 ] } { \ensuremath{\mathbf{r}}}_{n}^{[k_1,k_2]}\right ) } } $ ] , where @xmath119 } \triangleq { \ensuremath{e\left\{{\ensuremath{\mathbf{s}}}_n^{[k_1 ] } \left({\ensuremath{\mathbf{s}}}_n^{[k_2]}\right)^{\ensuremath{\mathsf{t}}}\right\ } } } \in { \ensuremath{\mathbb{r}}}^{v \times v}$ ] and @xmath120 }   \triangleq   { \ensuremath{e\left\{{\ensuremath{\boldsymbol{\phi}}}^{[k_1]}_n \left({\ensuremath{\boldsymbol{\phi}}}^{[k_2]}_n\right)^{\ensuremath{\mathsf{t}}}\right\ } } }   \in { \ensuremath{\mathbb{r}}}^{v \times v}$ ] .
in this appendix , we show that the score function covariance matrix , @xmath327 , is a scalar multiple of the inverse of the covariance matrix for all elliptical distributions defined by .
we begin by letting @xmath328 so that @xmath329 , which results in @xmath330 .
to compute the expectation requires the following multivariate integral to be evaluated : @xmath331 we use a transformation of variables utilized for similar problems in @xcite , namely , @xmath332 where @xmath333 , @xmath334 , @xmath335 , @xmath336 . by noting that @xmath337 and the jacobian of the transformation from @xmath338 to @xmath339^{\ensuremath{\mathsf{t}}}$ ] is @xmath340 , we have @xmath341 .
there are two cases , @xmath342 and @xmath343 , required to evaluate .
let us consider the former first , @xmath344 where we have made use of @xmath345 / \gamma\left[\left(n+2\right)/2\right]$ ] when @xmath346 .
now for the off - diagonal terms , e.g. , when @xmath347 , @xmath348 , where we have used the following @xmath349 when @xmath350 .
the result holds for the more general case when @xmath351 and @xmath352 and we arrive at the final expression of @xmath353 , where @xmath354 .
m.  anderson , t.  adali , and x .-
li , `` joint blind source separation of multivariate gaussian sources : algorithms and performance analysis , '' _ ieee trans .
signal process .
_ , vol .  60 , no .  4 , pp . 16721683 , apr
2012 .
m.  anderson , x .-
li , and t.  adali , `` nonorthogonal independent vector analysis using multivariate gaussian model , '' in _ latent variable analysis and signal separation _ , ser .
lecture notes in computer science.1em plus 0.5em minus 0.4emspringer berlin / heidelberg , 2010 , vol . 6365 , pp .
354361 .
j.  va , m.  anderson , x .-
li , and t.  adali , `` a maximum likelihood approach for independent vector analysis of gaussian data sets , '' in _ ieee international workshop on machine learning for signal processing ( mlsp 2011 ) _ , beijing , china , sep
. 2011 .
li , m.  anderson , and t.  adali , `` second and higher - order correlation analysis of multiset multidimensional variables by joint diagonalization , '' in _ latent variable analysis and signal separation _ , ser .
lecture notes in computer science.1em plus 0.5em minus 0.4emspringer berlin / heidelberg , 2010 , vol . 6365 , pp .
197204 .
t.  melzer , m.  reiter , and h.  bischof , `` , '' in _ _ , ser .
lecture notes in computer science , g.  dorffner , h.  bischof , and k.  hornik , eds.1em plus 0.5em minus 0.4emspringer berlin heidelberg , 2001 , vol . 2130 , pp .
353360 .
t.  kim , t.  eltoft , and t .- w .
lee , `` independent vector analysis : an extension of ica to multivariate components , '' in _ independent component analysis and blind signal separation _ ,
lecture notes in computer science.1em plus 0.5em minus 0.4emspringer berlin / heidelberg , 2006 , vol . 3889 , pp .
165172 .
a.  hiroe , `` solution of permutation problem in frequency domain ica , using multivariate probability density functions , '' in _ independent component analysis and blind signal separation _ , ser .
lecture notes in computer science , j.  rosca , d.  erdogmus , j.  c. prncipe , and s.  haykin , eds.1em plus 0.5em minus 0.4emspringer berlin heidelberg , 2006 , vol . 3889 , pp .
601608 .
t.  kim , h.  t. attias , s .- y .
lee , and t .- w .
lee , `` blind source separation exploiting higher - order frequency dependencies , '' _ ieee trans .
audio speech lang .
_ , vol .
15 , no .  1 ,
7079 , jan . 2007 .
p.  tichavsk , z.  koldovsk , and e.  oja , `` performance analysis of the fastica algorithm and cramr - rao bounds for linear independent component analysis , '' _ ieee trans .
signal process .
_ , vol .  54 , no .  4 , pp .
11891203 , apr .
2006 .
m.  anderson , g .- s .
fu , r.  phlypo , and t.  adali , `` independent vector analysis , the kotz distribution , and performance bounds , '' in _ proc .
, speech signal process .
( icassp ) _
, 2013 , accepted .