a natural step to extend most of the existing many - body single - particle self - consistent methods is to include the full momentum and energy dependence of the vertex corrections .
historically , the self - consistent approach for vertex corrections was first considered by landau , abrikosov and khalatnikov in the context of the high energy behavior of quantum electrodynamics.@xcite the original goal was to develop a non - perturbative method which encodes the information in terms of a system of closed integral equations .
the parquet equations , in principle , provide a framework for self - consistent determination of the self - energy and the vertex corrections .
they were proposed for both boson - boson scattering and fermion - fermion scattering during the 1950s.@xcite methods similar to the parquet equations were first introduced in the context of many - body theory by de dominics and martin.@xcite one of the early practical applications was on the x - ray absorption and emission problem by roulet , gavoret , and nozires.@xcite since then , various problems have been studied by the parquet summation approach , most notably , the fermi liquid in a strong magnetic field,@xcite the disordered electron gas in a strong transverse magnetic field,@xcite the anderson impurity model,@xcite random potential problems,@xcite the hubbard model,@xcite
helium-4,@xcite helium-3,@xcite local moment formation,@xcite the vortex liquid model,@xcite the matrix models,@xcite and nuclear structure calculations.@xcite while these applications of parquet formulation provide a lot of important insights , most of the calculations are based on various approximated forms of the parquet equations .    it is obvious that going from a one particle to a two - particle self - consistent calculation represents a significant increase in the computational effort , as each two - particle vertex contains three independent momentum and frequency indices . from the point of view of practical calculation , the number of elements for each index is around a few thousands .
therefore , the number of elements for the vertices are around tens of millions to a few billions .
moreover , all the information is encoded in integral equations with complicated structure , in which simplification does not seem to be immediately possible .
indeed , in the past , the most successful application using the full parquet equations was largely limited to the single anderson impurity model .
@xcite with recent advances in computational infrastructure where peta - scale performance has become available , calculations for lattice models , such as the hubbard model are now feasible .
for example , the solution of the parquet equations for a @xmath0 hubbard cluster with on - site coupling @xmath1 and temperature @xmath2 was recently obtained.@xcite    however , limitations on computer performance and storage are apparently not the sole obstacles for obtaining the solution of the parquet equations . another major barrier is the stability of the solvers .
the simple iteration method , which is widely adapted for the dynamical mean field method , often fails to provide a stable solution for the parquet equations . in most cases ,
a damping scheme has to be employed .
even with the damping scheme , when the temperature is low or the coupling is large , finding a stable solution still seems to be rather difficult .
@xcite    given the large number of variables and the complexity of the parquet equations , instabilities in their solution may not be unexpected .
methods based on the local gradient are not likely to be suitable as the hessian can not be readily calculated .
most of the non - linear solvers only have local convergence properties
. this may not pose a problem if we have a reasonable guess which is close enough to the true solution .
unfortunately , it is not easy to obtain a good initial guess for the parquet equations .
methods that in principle allow  global " convergence , for example , the homotopy method or continuous loading method , have been proposed as possible ways to improve the calculations.@xcite while these tend to improve convergence , many steps are required for the solution to move along the homotopy path . moreover
, practical experience seems to suggest that convergence may still not be achieved when the temperature is low or the coupling is strong .
it is clear that a better solver is definitely required for the practical application of the parquet method within the context of the strongly correlated systems .
one of the most prominent differences between the parquet formulation and most of the other approximation schemes such as rpa,@xcite self - consistent spin fluctuations approach,@xcite and fluctuation exchange approach@xcite is that the so - called crossing symmetry is obeyed by construction of the parquet equations .
the crossing symmetry@xcite implies that a vertex in one channel can also produce the vertex in all other channels by pulling or crossing the vertex legs and multiplying by appropriate constants .
it also implies the pauli exclusion principle is automatically satisfied .
however , in the course of the iteration process , as long as the exact solution of the parquet equations is not obtained , the crossing symmetry is violated . the main point in the present paper
is to highlight that the crossing symmetry is crucial for obtaining a stable solution .
we devise a modified iteration scheme which can obtain a stable solution for the parquet equations at lower temperature and stronger coupling than that from the previous schemes.@xcite this is achieved primarily by restoring the crossing symmetry at each step of the iteration .
it is important to notice that because of the large number of vertex functions , for production runs , massively parallel machines are absolutely necessary .
since the vertex functions in different channels are mixed in the parquet formulation , an efficient scheme to transform the vertex functions storage in different nodes is critical to improving the overall efficiency of the calculations .
some of the computational details have been explained in the previous publication.@xcite we have further improved the scheme which allows us to hide the communication latency across different nodes behind the local calculations within the nodes , which effectively further speeds up the calculations .
the model used for testing the computational scheme for solving the parquet equations is the standard hubbard model at half - filling .
the hamiltonian is @xmath3 where @xmath4 is the on - site repulsion and @xmath5 is the hopping matrix which establishes a unit of energy .
the paper is organized as follows . in section
ii , we reproduce the parquet equation which also serves to fix the notation . in section iii , we describe the iteration scheme for solving the parquet equations . in section iv
, we discuss the violation of crossing symmetry . in section
v , we present a modified iteration scheme which explicitly restores the crossing symmetry .
we also discuss the limitation of the modified scheme and the possible directions for further developments . in section
vi , we present the leading eigenvalues of the antiferromagnetic channel as a function of the temperature and coupling strength and find that the parameter region of stable solutions is greatly increased when the crossing symmetry is enforced . a brief summary is contained in section vii . in the appendix , we present a latency hiding scheme which allows substantial increase of the efficiency for solving the parquet equations .
in order for the current paper to be reasonably self - contained , we provide in this section a brief description of the parquet equations , largely following ref . .
the purpose is to highlight the structure of the equations and to define the notations that will be used in the subsequent sections .
we do not intend to provide a derivation of the formulation .
for this we refer the readers to the literature where ample discussions can be found.@xcite    standard perturbative expansions attempt to describe all the scattering processes , at the lowest orders , as single- or two - particle feynman diagrams . in the single - particle formulation
the self - energy describes the many - body processes that renormalize the motion of a particle in the interacting background of all the other particles . in the two - particle context ,
one is able to probe the interactions between particles using the so - called vertex functions , which are matrices describing the two particle scattering processes .
for example , the reducible ( full ) two - particle vertex @xmath6 describes the amplitude of a particle - hole pair scattered from its initial state @xmath7 into the final state @xmath8 .
here , @xmath9 represents a set of indices which combines the momentum @xmath10 , the matsubara frequency @xmath11 and , if needed , the spin @xmath12 and band index @xmath13 .
since the total momentum and energy of the vertex are conserved , it is convenient to adapt the notation @xmath14 for the numerical implementation of the single band hubbard model .
@xcite    in general , depending on how particles or holes are involved in the scattering processes , one can define three different two - particle scattering channels .
these are the particle - hole ( p - h ) horizontal channel , the p - h vertical channel and the particle - particle ( p - p ) channel .
one can further discriminate the vertices according to their topology .
starting from the reducible vertex @xmath15 introduced above , we may define the irreducible vertex @xmath16 corresponding to the subclass of diagrams in @xmath15 that can not be separated into two parts by cutting two horizontal green function lines .
similarly , the fully irreducible vertex @xmath17 corresponds to the subclass of diagrams in @xmath16 that can not be split into two parts by cutting two green function lines in any channel .
furthermore , since we are mostly interested in models that preserve the @xmath18 spin rotation symmetry , and since this is an exact symmetry for our two - dimensional calculations at non - zero temperature , it is convenient to preserve this symmetry .
this is accomplished by decomposing the vertices in the so - called spin - diagonalized representation.@xcite in this representation , the spin degrees of freedom decompose the particle - hole channel into the density and the magnetic channels , and the particle - particle channel into the spin singlet and the spin triplet channels which we denote as @xmath19-channel , @xmath20-channel , @xmath21-channel , and @xmath22-channel respectively .
they are defined as follows ,    @xmath23    and similarly for @xmath15 and @xmath17 .
we reproduce the full set of equations for the parquet formulation in the spin diagonalized representation in the following.@xcite the schwinger - dyson equation is    @xmath24    where @xmath25 is the single - particle green function , which itself can be calculated from the self - energy using the dyson equation , @xmath26 where @xmath27 is the bare green function . here ,
the indices @xmath28 , @xmath29 and @xmath30 combine momentum @xmath31 and matsubara frequency @xmath32 , i.e.  @xmath33 .
the reducible and the irreducible vertices in a given channel are related by the bethe - salpeter equation , @xmath34 @xmath35 where @xmath36 for the density and magnetic channels and @xmath37 for the spin singlet and spin triplet channels .
the vertex ladders are defined as @xmath38 where @xmath39 is the product of two single - particle green functions .
the parquet equations in the spin diagonalized representation are    @xmath40    @xmath41    @xmath42    @xmath43    it is important to note at this point that if we substitute the irreducible vertices @xmath44 ( eqs .
[ gamma_d],[gamma_m],[gamma_s ] , and [ gamma_t ] ) into the bethe - salpeter equation ( eqs .
[ sd_bsph_eq ] and [ sd_bspp_eq ] ) the crossing symmetry in the full vertex @xmath15 is automatically satisfied regardless of the numerical values of the vertex ladders @xmath45 and @xmath46 , assuming the fully irreducible vertices , @xmath17 , are crossing symmetric .
we write all the full vertices explicitly in the following using only the vertex ladders , @xmath45 , @xmath46 , and the fully irreducible vertices , @xmath17 .
@xmath47    @xmath48    @xmath49    @xmath50    these relations allow us to restore the crossing symmetry for the full vertices without heavy computational overhead .
the prominent technical problem at hand is whether or not we can solve this set of equations efficiently without resorting to any approximated scheme .
an obvious difficulty is to handle the large number of variables . on going from the one - particle level calculation to two - particle level calculation
, the number of variables which has to be monitored grows as the third power of the linear dimension of the system .
if @xmath51 is the number of lattice sites times the number of discrete matsubara frequencies , i.e. , @xmath52 , the largest @xmath51 that can be handled is in the range @xmath53 i.e. , the number of variables can be over one billion .
one can immediately see that practical calculations for reasonably large system sizes pose a serious problem , although not insurmountable with modern computational facilities where a large number of computer nodes are accessible .
however , in addition to the large number of computations associated with solving the parquet equations , they also require a complex communication pattern between the different processes as we discussed in more detail in the appendix .
moreover , a numerical instability is not unexpected , especially when the system in the proximity of a phase transition .
the parquet formulation consists of two sets of equations .
the first set , made of the parquet equations and the bethe - salpeter equation , determines the full vertex @xmath15 and the irreducible vertex @xmath44 given the one - particle self - energy @xmath54 and the fully irreducible vertex @xmath17 as the inputs .
the second set of equations determine the one - particle quantities given the full vertex @xmath15 ; it includes the schwinger - dyson equation and the dyson equation .    since the method is iteration based , the initial guess is crucial for obtaining a converged solution . in principle , the initial guess can be approximated , for example , by second order perturbation .
however , in practice , this is not the optimal choice , especially when the self - energy from the second order perturbation is small . in this case
the green function will quickly destabilize the calculation .
this may relate to the fact that the damping from the imaginary part of the self - energy is quickly reduced .
since we are supposing that we know the fully irreducible vertex @xmath17 , a practical scheme is to choose the irreducible , @xmath44 and full vertices , @xmath15 , equal to @xmath17 , and a large value ( a few times of the bandwidth ) for the imaginary part of the self - energy .        fig .
[ algorithm_a ] is an illustration of the flow diagram of the algorithm , where the fully irreducible vertices are the initial input for the calculation .
the algorithm can be described as the following :    \1 .
set the initial conditions for the irreducible and full vertices , and the self - energy .
update the green s functions and calculate the bare susceptibility , @xmath39 , which is given by the product of two green s functions .
solve the parquet and the bethe - salpeter equations for the irreducible vertices , @xmath44 .
simple iteration is used until the convergence criteria are met for the irreducible vertices .
this completes the update of the vertices .
the next step is to use the irreducible vertices obtained from the parquet equations to construct the full vertices .
solve the bethe - salpeter equation to obtain the full vertices , @xmath15 , using the irreducible vertices from the previous step ; this is executed exactly by calling the lapack routines for the inverse of the matrices . @xcite    with the full vertices obtained , we can update the self - energy .
solve the dyson - schwinger equation to obtain the self - energy from the full vertices .
simple iteration is used until certain convergence criteria are met for the self - energy .
solve the dyson equation for the fully dressed green function from the self - energy .
this completes the iteration loop , and the procedure is repeated from step 2 , until convergence is reached for both the self - energy and the irreducible vertices .    in practice
, step 2 which attempts to obtain the irreducible vertices needs to be iterated for a few times to get a reasonable convergence , even in the case where the coupling is weak and the temperature is high .
on the other hand , step 4 which attempts to solve the self - energy from the updated full vertices is not iterated more than one time at each loop , so as to avoid instability ( we define instability here as the failure to obtain a converged solution from the iterative solver ) .
attempting to solve the self - energy at the early stage of the iteration procedure where the full vertices are not well converged can generally lead to instability .
although in the present paper , we only focus on the hubbard model , instabilities in the iteration process have also been observed in solving the parquet equations for nuclear structure calculations.@xcite    a widely used method to avoid the instability in the iterative process is to introduce a damping factor in the updates of the variables .
the updates are modified as follows .
@xmath55 with this damping scheme , the solution for the half - filled hubbard model on a @xmath56 cluster has been successfully obtained for @xmath1 and temperature , @xmath2.@xcite however , in the strong coupling regime , obtaining a stable solution still seems to be difficult , even though a rather heavy damping is employed
.    we will demonstrate the instability problem of the simple iterative process by monitoring the leading eigenvalues @xmath57 defined as @xmath58 for @xmath59 and @xmath20 ; similarly @xmath60 for @xmath61 and @xmath22 . in principle , these leading eigenvalues signal a phase transition by going through @xmath62 , expressing the divergence of the susceptibilities in the corresponding channel .    in fig .
[ nosym ] , we plot the leading eigenvalues of the density , magnetic , spin singlet , and spin triplet channels as a function of the number of iteration steps calculated with this simple iteration method ( si ) .
the calculation is done on a 2@xmath632 cluster with 32 frequencies and temperature @xmath64 ; the damping parameter is @xmath65 .
a converged solution is obtained for @xmath1 ; however , for @xmath66 and @xmath67 the iterative solutions diverge .
changing the damping or the initial self - energy does not help in obtaining a converged solution for the larger values of @xmath4 .
these are illustrative examples which show the problem of using the simple iteration method for solving the parquet equations . for weak coupling and not too low temperature ,
converged solution can be obtained . beyond
weak coupling the iteration becomes divergent .
a widely used method to alleviate the divergence in the non - linear solver is the so - called continuous loading or homotopy method .
the basis of the continuous loading method is to construct an auxiliary equation with a tuning parameter @xmath68 , so that its solution is trivial for @xmath69 but the solution of the original equation is recovered for @xmath70 .
symbolically we can write down the set of equations of the parquet formulation as @xmath71 , where @xmath72 is a large vector .
we define the auxiliary function as @xmath73 , where @xmath74 is a function with a trivial solution . in our study
we choose @xmath75 as a vector containing all the elements of @xmath76 and @xmath77 , where @xmath78 and @xmath79 are the initial guesses for the irreducible vertices and the self - energy respectively .
the iteration method is used to solve the function @xmath80 , instead of the @xmath72 .
one can readily see that the solution of @xmath81 is trivial while the solution of the @xmath72 is recovered when @xmath70 .
the idea is to find a converged solution for @xmath80 with a small enough @xmath68 where a converged solution can be readily obtained , and then gradually increase @xmath68 until it goes to @xmath62 .
therefore , a series of @xmath68 values are needed , which we denote as @xmath82 .    we plot the leading eigenvalues from the continuous loading method in fig .
[ continue_nosym ] .
the values of @xmath68 used are @xmath83 @xmath84 . @xmath85 and @xmath86
iterations are performed for @xmath87 and @xmath88 respectively . for @xmath1 , converged solution is obtained for @xmath70 .
however , for @xmath66 and @xmath67 , the iterative solution diverges .
the divergences appear before the homotopy parameter is pushed to @xmath70 .
these examples illustrate the generic behavior when solving the parquet equations by the simple iteration method beyond weak coupling .
they also illustrate that the continuous loading method may not be sufficiently robust to solve the problem .
the damping factor @xmath89 used in these calculations is @xmath90 which we believe is a fairly small value , although it may still not be sufficient .
a rule of thumb for choosing the damping parameter is that the damping parameter should be close to the value of the inverse of the residual between two consecutive iterations ; unfortunately , with the huge number of variables , this choice will result in a very small step and may not be a practical option.@xcite
the exact solution of the parquet equations automatically satisfies the crossing symmetry .
it is one of the most important differences between the parquet formulation and most other perturbative methods .
however , within the iteration scheme presented in the last section , the crossing symmetry is not fulfilled unless the iteration converges to an exact solution . from the above section
, we clearly find that the iteration method , even with the help of the continuous loading scheme , is not robust enough to obtain a converged solution beyond weak coupling .
it is desirable to quantify the violation of crossing symmetry .
the following six equalities are the consequence of the crossing symmetry ( see fig .
[ crossing_symmetries ] for a diagrammatic representation of these crossing symmetry ( cs ) operations ) .
@xcite @xmath91(p+p^{'}+q)_{-p^{'},-p } \equiv r_{1 } ,   \\
l_{2 } \equiv f_{s}(q)_{p , p^ { ' } } =   \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\ ; \\ \nonumber [ -(3/2 ) f_{m}+(1/2)f_{d}](p+p^{'}+q)_{-p^{'},-p } \equiv r_{2 } ,   \\ l_{3 } \equiv f_{m}(q)_{p , p^ { ' } } =   \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\ ; \\ \nonumber [ ( 1/2)f_{t}-(1/2)f_{t}](p+p^{'}+q)_{-p - q ,-
p } \equiv r_{3 } , \;\\
l_{4 } \equiv f_{d}(q)_{p , p^ { ' } } =   \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\ ; \\ \nonumber [ ( 3/2)f_{t}+(1/2)f_{t}](p+p^{'}+q)_{-p - q ,-
p } \equiv r_{4 } , \;\\
l_{5 } \equiv f_{m}(q)_{p , p^ { ' } } =   \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\ ; \\ \nonumber [ ( 1/2)f_{m}-(1/2)f_{d}](p^{'}-p)_{p , p+q } \equiv r_{5 } , \;\;\;\;\;\;\;\;\;\;\ ; \\ l_{6 } \equiv f_{m}(q)_{p , p^ { ' } } =   \label{crossing_equalities6 } \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\ ; \\ \nonumber [ -(3/2)f_{m}+(1/2)f_{d}](p^{'}-p)_{p , p+q } \equiv r_{6}. \;\;\;\;\;\;\;\ ; \nonumber \end{aligned}\ ] ]    in fig .
[ nosym_crossing ] we plot the violation of crossing symmetry versus the number of iterations .
it can be seen clearly that the crossing symmetry can not be perfectly restored , even for the case of @xmath92 , the measures of violation of crossing show oscillatory decreasing behavior , and the rate of decrease is quite slow even though the leading eigenvalues seem to be well converged . obviously , for the cases of @xmath66 and @xmath67 the crossing symmetry is severely violated and at the verge of the divergence there are sharp increases in the crossing symmetry violation .
this may suggest that if the crossing symmetry can be restored , the divergence may be avoided beyond weak coupling .    in fig .
[ continue_nosym_crossing ] , we plot the crossing symmetry violation as a function of iteration steps with the continuous loading method and the same parameters as in fig .
[ continue_nosym ] .
it is important to note that the homotopy function @xmath93 does not respect the crossing symmetry except at @xmath70 .
therefore , although the solution is converged , as long as @xmath94 , the crossing symmetry is violated .
the data for @xmath95 , and @xmath67 are shown in the upper , the middle , and the lower panels respectively .
the data for @xmath1 show a peak at the beginning of the iteration procedure when @xmath68 is increased and gradually converges to a finite value . for @xmath68 close to @xmath62 , the data show a similar oscillatory behavior as that from the simple iteration method .
similar behaviors are also observed for @xmath66 and @xmath96 , however , the iterations fail to converge for @xmath68 close to @xmath62 ; the crossing symmetry is strongly violated .
similar to that for the simple iteration method , at the verge of the divergence , there are sharp increases of the violation of crossing symmetry .
cluster at temperature @xmath64 .
the initial condition for the self - energy is set at @xmath97 , and that for the irreducible vertex is set at the bare hubbard coupling .
the damping factor @xmath89 is set at @xmath90 .
the solution is well converged for @xmath1 .
however , divergence occurs for @xmath66 at the @xmath98th iteration step . for @xmath96
, divergence occurs at the @xmath99th iteration step .
[ nosym],scaledwidth=26.0% ]     cluster at temperature @xmath64 .
the initial condition for the self - energy is set at @xmath97 , and that for the irreducible vertex is set at the bare hubbard coupling .
the damping factor @xmath89 is set at @xmath90 .
the solution is well converged for @xmath1 .
however , divergence occurs for @xmath66 at the @xmath98th iteration step . for @xmath96
, divergence occurs at the @xmath99th iteration step .
[ nosym],scaledwidth=26.0% ]     cluster at temperature @xmath64 .
the initial condition for the self - energy is set at @xmath97 , and that for the irreducible vertex is set at the bare hubbard coupling .
the damping factor @xmath89 is set at @xmath90 .
the solution is well converged for @xmath1 .
however , divergence occurs for @xmath66 at the @xmath98th iteration step . for @xmath96
, divergence occurs at the @xmath99th iteration step .
[ nosym],scaledwidth=26.0% ]     and use them to define the auxiliary function as @xmath73 , where @xmath74 is a function with a trivial solution .
we choose @xmath74 as a vector containing all the elements of @xmath76 and @xmath77 .
the iteration method is used to solve the function @xmath80 , instead of the @xmath72 .
the solution of the @xmath72 is recovered when @xmath70 .
a series of @xmath68 values are needed , which we denote as @xmath82 .
the function @xmath100 is solved by the simple iteration method with the initial conditions given by the converged solution of the function @xmath101 . for @xmath1
, we can push the value of @xmath68 to @xmath62 to obtain the converged solution for @xmath102 , and the solution of the parquet equations is recovered .
however , the iteration procedure diverges for the cases of @xmath66 and @xmath96 ; they diverge at @xmath103 and @xmath104 respectively .
these examples show that for the cases where simple iteration method diverges , the continuous loading method may not eliminate the divergence , even though the value of @xmath68 is pushed fairy close to @xmath62.[continue_nosym],scaledwidth=26.0% ]     and use them to define the auxiliary function as @xmath73 , where @xmath74 is a function with a trivial solution .
we choose @xmath74 as a vector containing all the elements of @xmath76 and @xmath77 .
the iteration method is used to solve the function @xmath80 , instead of the @xmath72 .
the solution of the @xmath72 is recovered when @xmath70 .
a series of @xmath68 values are needed , which we denote as @xmath82 .
the function @xmath100 is solved by the simple iteration method with the initial conditions given by the converged solution of the function @xmath101 . for @xmath1
, we can push the value of @xmath68 to @xmath62 to obtain the converged solution for @xmath102 , and the solution of the parquet equations is recovered .
however , the iteration procedure diverges for the cases of @xmath66 and @xmath96 ; they diverge at @xmath103 and @xmath104 respectively .
these examples show that for the cases where simple iteration method diverges , the continuous loading method may not eliminate the divergence , even though the value of @xmath68 is pushed fairy close to @xmath62.[continue_nosym],scaledwidth=26.0% ]     and use them to define the auxiliary function as @xmath73 , where @xmath74 is a function with a trivial solution .
we choose @xmath74 as a vector containing all the elements of @xmath76 and @xmath77 .
the iteration method is used to solve the function @xmath80 , instead of the @xmath72 .
the solution of the @xmath72 is recovered when @xmath70 .
a series of @xmath68 values are needed , which we denote as @xmath82 .
the function @xmath100 is solved by the simple iteration method with the initial conditions given by the converged solution of the function @xmath101 . for @xmath1
, we can push the value of @xmath68 to @xmath62 to obtain the converged solution for @xmath102 , and the solution of the parquet equations is recovered .
however , the iteration procedure diverges for the cases of @xmath66 and @xmath96 ; they diverge at @xmath103 and @xmath104 respectively .
these examples show that for the cases where simple iteration method diverges , the continuous loading method may not eliminate the divergence , even though the value of @xmath68 is pushed fairy close to @xmath62.[continue_nosym],scaledwidth=26.0% ]    , versus the number of iterations for the simple iteration ( si ) method with the same parameters as fig .
[ nosym ] .
the six measures of crossing symmetry violation are defined as @xmath105 and @xmath106 are defined respectively as the left hand side and the right hand side of the eqs .
[ crossing_equalities1 ]  [ crossing_equalities6 ] .
the data for @xmath1 shows an oscillatory but decreasing trend .
this is expected for the case where the iteration provides a well converged solution .
one should note that although the leading eigenvalues seem to be converged , the crossing symmetry is not perfectly constructed from the iteration . for @xmath66 and @xmath96 ,
the iteration fails to provide converged solutions , and the crossing symmetry is strongly violated .
in particular , at the verge of the divergence , there is a sharp increase of the violation of crossing symmetry .
[ nosym_crossing],scaledwidth=26.0% ]    , versus the number of iterations for the simple iteration ( si ) method with the same parameters as fig .  [ nosym ] .
the six measures of crossing symmetry violation are defined as @xmath105 and @xmath106 are defined respectively as the left hand side and the right hand side of the eqs .
[ crossing_equalities1 ]  [ crossing_equalities6 ] .
the data for @xmath1 shows an oscillatory but decreasing trend .
this is expected for the case where the iteration provides a well converged solution .
one should note that although the leading eigenvalues seem to be converged , the crossing symmetry is not perfectly constructed from the iteration .
for @xmath66 and @xmath96 , the iteration fails to provide converged solutions , and the crossing symmetry is strongly violated . in particular , at the verge of the divergence , there is a sharp increase of the violation of crossing symmetry .
[ nosym_crossing],scaledwidth=26.0% ]    , versus the number of iterations for the simple iteration ( si ) method with the same parameters as fig .
[ nosym ] .
the six measures of crossing symmetry violation are defined as @xmath105 and @xmath106 are defined respectively as the left hand side and the right hand side of the eqs .
[ crossing_equalities1 ]  [ crossing_equalities6 ] .
the data for @xmath1 shows an oscillatory but decreasing trend .
this is expected for the case where the iteration provides a well converged solution .
one should note that although the leading eigenvalues seem to be converged , the crossing symmetry is not perfectly constructed from the iteration . for @xmath66 and @xmath96
, the iteration fails to provide converged solutions , and the crossing symmetry is strongly violated . in particular , at the verge of the divergence , there is a sharp increase of the violation of crossing symmetry .
[ nosym_crossing],scaledwidth=26.0% ]    , versus the number of iterations for the continuous loading ( cl ) method with the same parameters and the same definition of @xmath107 as in fig .
[ nosym_crossing ] .
the homotopy function @xmath93 does not respect the crossing symmetry except at @xmath70 .
therefore , although the solution may be converged for some values of @xmath68 , as long as @xmath94 , the violation of crossing symmetry is non - zero . for @xmath1 the crossing symmetry violations peak near the beginning of the iteration procedure where @xmath68 is small and gradually converge to a finite value when @xmath70 .
similar behaviors are also observed for @xmath66 and @xmath96 ; however , since the iteration fails to converge for @xmath68 close to @xmath62 , the crossing symmetry is strongly violated .
similar to that observed in the simple iteration method , at the verge of the divergence , there is a sharp increase of the violation of crossing symmetry .
[ continue_nosym_crossing],scaledwidth=26.0% ]    , versus the number of iterations for the continuous loading ( cl ) method with the same parameters and the same definition of @xmath107 as in fig .
[ nosym_crossing ] . the homotopy function @xmath93 does not respect the crossing symmetry except at @xmath70 .
therefore , although the solution may be converged for some values of @xmath68 , as long as @xmath94 , the violation of crossing symmetry is non - zero . for @xmath1 the crossing symmetry violations peak near the beginning of the iteration procedure where @xmath68 is small and gradually converge to a finite value when @xmath70 .
similar behaviors are also observed for @xmath66 and @xmath96 ; however , since the iteration fails to converge for @xmath68 close to @xmath62 , the crossing symmetry is strongly violated .
similar to that observed in the simple iteration method , at the verge of the divergence , there is a sharp increase of the violation of crossing symmetry .
[ continue_nosym_crossing],scaledwidth=26.0% ]    , versus the number of iterations for the continuous loading ( cl ) method with the same parameters and the same definition of @xmath107 as in fig .
[ nosym_crossing ] .
the homotopy function @xmath93 does not respect the crossing symmetry except at @xmath70 .
therefore , although the solution may be converged for some values of @xmath68 , as long as @xmath94 , the violation of crossing symmetry is non - zero . for @xmath1 the crossing symmetry violations peak near the beginning of the iteration procedure where @xmath68 is small and gradually converge to a finite value when @xmath70 .
similar behaviors are also observed for @xmath66 and @xmath96 ; however , since the iteration fails to converge for @xmath68 close to @xmath62 , the crossing symmetry is strongly violated .
similar to that observed in the simple iteration method , at the verge of the divergence , there is a sharp increase of the violation of crossing symmetry .
[ continue_nosym_crossing],scaledwidth=26.0% ]
although it does not seem to be easy to analyze all the causes of the instability in the iteration , based on the discussion in the above section , our conjecture is that one of the possible reasons for the instability is that certain symmetries are violated in the course of the iteration process . a possible strategy to improve the iteration scheme is to impose those symmetries explicitly into the iteration process , so that at each step of the iteration these symmetries are not violated .
the full vertices @xmath15 obtained by the solution of the bethe - salpeter equation can not guarantee the crossing symmetry , unless the exact solution is attained .
therefore , so as to preserve the crossing symmetry , the simplest method is to use the full vertices obtained by solving the bethe - salpeter equation ( that is the full vertices obtained from the step 3 of the algorithm presented in the section iii ) , and feed them back into the parquet equation to reconstruct the crossing symmetric full vertices .
[ algorithm_b ] illustrates the flow diagram for solving the parquet equations with explicit restoration of the crossing symmetry in the full vertices .
here is the algorithm which explicitly preserves the crossing symmetry :    .
because of this explicit restoration of the crossing symmetry , in practice , the step 2 is only iterated for one time as the self - consistency is not required to generate the crossing symmetry.,scaledwidth=51.0% ]    \1 . set the initial conditions for the irreducible and full vertices , and the self - energy .
update the green s functions and calculate the bare susceptibility , @xmath39 .
solve the parquet and the bethe - salpeter equations for the irreducible vertices , @xmath44 .
simple iteration is used until the convergence criteria are met for the irreducible vertices .
since we will restore the crossing symmetry of the full vertices in the step 3b , we find that it is not necessary to attain the self - consistency for step 2 . in practice , we iterate the parquet equations for one time only .
the next step is to use the irreducible vertices obtained from the parquet equations to construct the full vertices .
solve the bethe - salpeter equation to obtain the full vertices , @xmath15 , using the irreducible vertices from the previous step .
this is executed exactly by calling the lapack routines for the inverse of the matrices .
@xcite    3b . use the new irreducible vertices obtained in step 2 and the full vertices obtained in step 3a to form the vertex ladders . construct the full vertices from eqs .
[ sd_bsph_eq ] and [ sd_bspp_eq ] using the vertex ladders .
following these steps , the crossing symmetry is restored in the full vertex @xmath15 .    with
the full vertices obtained , we can update the self - energy .
solve the dyson - schwinger equation to obtain the self - energy from the full vertices .
simple iteration is used until the convergence criteria are met for the self - energy .
solve the dyson equation for the fully dressed green function from the self - energy .
this completes the iteration loop , and the procedure is repeated from step 2 until the criteria of convergence are met for both the self - energy and the irreducible vertices .
the main difference between the current algorithm and the previous algorithm we present in section iii is in step 3 where the full vertices are constructed . in the previous algorithm the bethe - salpeter equation
is solved many times to attain convergence .
when the absolute convergence is attained , the crossing symmetry will be satisfied . in the current algorithm
, we just explicitly solve the bethe - salpeter equation in step 3a to refresh the full vertices .
once we obtain the full vertices , in step 3b , we construct the new vertex ladders and the new full vertices from the vertex ladders using the bethe - salpeter equation . by doing so , the crossing symmetry of the full vertices will be satisfied ; see eqs .
[ f_d ] , [ f_m ] , [ f_s ] , and [ f_t ] . in fig .
[ sym ] we show the leading eigenvalues using the same set of parameters used in fig .
[ nosym ] .
while the simple iteration scheme without crossing symmetry fails to converge for the case of @xmath108 and @xmath109 , it provides a converged solution when the crossing symmetry is explicitly restored .    .
the only difference is that the crossing symmetry in the full vertex , @xmath15 , is explicitly restored at each step of the iteration .
this is easily achieved by constructing the full vertex directly from eqs .
[ sd_bsph_eq ] and [ sd_bspp_eq ] .
the simple iteration scheme without crossing symmetry fails for the case of @xmath108 and @xmath109 . with the crossing symmetry explicitly restored ,
converged solutions are obtained .
[ sym],scaledwidth=26.0% ]    .
the only difference is that the crossing symmetry in the full vertex , @xmath15 , is explicitly restored at each step of the iteration .
this is easily achieved by constructing the full vertex directly from eqs .
[ sd_bsph_eq ] and [ sd_bspp_eq ] .
the simple iteration scheme without crossing symmetry fails for the case of @xmath108 and @xmath109 . with the crossing symmetry explicitly restored , converged
solutions are obtained .
[ sym],scaledwidth=26.0% ]    .
the only difference is that the crossing symmetry in the full vertex , @xmath15 , is explicitly restored at each step of the iteration .
this is easily achieved by constructing the full vertex directly from eqs .
[ sd_bsph_eq ] and [ sd_bspp_eq ] .
the simple iteration scheme without crossing symmetry fails for the case of @xmath108 and @xmath109 . with the crossing symmetry explicitly restored , converged
solutions are obtained .
[ sym],scaledwidth=26.0% ]    .
two symmetries are explicitly restored at each step of the iteration : the crossing symmetry ( cs ) in the full vertex , @xmath15 , and the time - reversal symmetry ( trs ) for the self - energy , @xmath54 , and both the irreducible vertex , @xmath44 , and the full vertex , @xmath15 , ( that is @xmath110 and similarly for @xmath44 ) .
notice there is no substantial gain in the convergent rate compared to the case with only the crossing symmetry being restored .
[ fullsym],scaledwidth=26.0% ]    .
two symmetries are explicitly restored at each step of the iteration : the crossing symmetry ( cs ) in the full vertex , @xmath15 , and the time - reversal symmetry ( trs ) for the self - energy , @xmath54 , and both the irreducible vertex , @xmath44 , and the full vertex , @xmath15 , ( that is @xmath110 and similarly for @xmath44 ) .
notice there is no substantial gain in the convergent rate compared to the case with only the crossing symmetry being restored .
[ fullsym],scaledwidth=26.0% ]    .
two symmetries are explicitly restored at each step of the iteration : the crossing symmetry ( cs ) in the full vertex , @xmath15 , and the time - reversal symmetry ( trs ) for the self - energy , @xmath54 , and both the irreducible vertex , @xmath44 , and the full vertex , @xmath15 , ( that is @xmath110 and similarly for @xmath44 ) .
notice there is no substantial gain in the convergent rate compared to the case with only the crossing symmetry being restored .
[ fullsym],scaledwidth=26.0% ]      besides imposing the crossing symmetry on the full vertices , some of the internal symmetries can also be imposed on the irreducible vertices and the self - energy without much computational overhead .
we illustrate the time - reversal symmetry in the self - energy @xmath111 and the vertices ( spatial reflection symmetry and parity invariance are assumed ) , @xmath112    as these symmetry operations do not mix vertices across different values of @xmath30 , and providing that the data is distributed with one or more @xmath30 at each node , the time reversal symmetry of the vertices can be imposed without invoking communications across different nodes .
therefore , enforcing time - reversal symmetry will only cause a very minor computational overhead .
other symmetries , such as the point group symmetry for the square lattice can be rather cumbersome .
an expensive scheme involving heavy internode communication would be required to impose the complete set of point group symmetries .
however , we may impose an important subset of the operations @xmath113 for which @xmath114 without expensive communications . in these cases
, the vertices may be symmetrized by performing the sum @xmath115 where @xmath116 is the number of elements in this subset of operations . for general @xmath30 in the cluster brillouin zone
there would be no @xmath89 such that @xmath117 apart from the identity .
however , for the points of high symmetry , @xmath117 for all @xmath89 . generally , the instabilities first occur here , so imposing the point group symmetries at these @xmath30 values should have the greatest impact .    in fig .
[ fullsym ] we show the leading eigenvalues of various channels when both crossing and time - reversal symmetries are imposed for the same set of parameters being used from the data in fig .
[ nosym ] and [ sym ] .
spatial reflection symmetry , parity invariance and spin rotation invariance are assumed as appropriate for the two - dimensional hubbard model at non - zero temperature .
we can see that there is only very marginal improvement for the convergence compared to the results without explicitly restoring the time - reversal symmetry ( see fig .
[ sym ] ) .
we also use the scheme described above to partially impose the point group symmetries . however , these symmetries resulted in no additional improvements and therefore no results are shown .
with the improved scheme proposed in this paper , we are able to explore a wider range of temperature and coupling strength for the half - filled hubbard model . in fig .
[ eigen_ref ] we show the leading eigenvalue for the most singular channel , the antiferromagnetic channel , @xmath118 , as a function of @xmath4 for a range of temperatures as low as @xmath119 .
the data points enclosed in a black square correspond to the cases where the simple iteration without any symmetry restoration provides a converged solution . for all temperatures , @xmath118 increase sharply at weak coupling ( @xmath120 ) , and they tend to saturate at strong coupling ( @xmath121 ) .
they are most sensitive to temperature at the intermediate coupling ( @xmath122 ) .
we emphasize that convergency is not possible without the improved scheme , unless a large number of iterations are used to attain the crossing symmetry .
we present improvements of numerical implementations for solving the parquet equations for the hubbard model .
the main strategy is to enforce the symmetries in the iteration process .
the most prominent advantage of the parquet formulation , compared to most of the other approaches , is that the crossing symmetry is exactly fulfilled . however , in general , it is true only if the exact solution is found . with the simple iteration method , the crossing symmetry is strongly violated prior an instability , suggesting that the instability is due to these symmetry violations .
the continuous loading or homotopy method does not improve convergence significantly beyond the simple iteration method .
we note that the solutions of the continuous loading function do not preserve crossing symmetry .
this may partly explain why the continuous loading method does not provide significant improvement over simple iteration .
we present a simple method to enforce the crossing symmetry at each step of the iteration which does not substantially increase the computational cost .
the addition of these symmetry constraints can greatly improve the stability of the calculation , so that a wider range of parameters can be explored by the parquet formulation . along this line of thought
, one can expect that the stability may be further improved if other symmetries are also imposed , the obvious ones being time - reversal and point group .
however , these additional symmetries did not improve the stability significantly beyond that obtained with crossing symmetry alone .
the parquet formulation still remains as one of the best approaches for calculating the two - particle vertex functions in a self - consistent manner . at present , solving the parquet equations for a large lattice size is still a very challenging task ; however , with the continuous advances of computational facilities , it should become more feasible in the foreseeable future . a promising direction , which allows immediate application of the parquet formulation , is to incorporate it as part of the multi - scale many - body approach.@xcite    we would like to acknowledge very useful discussions with karen tomko , and we thank peter reis for his careful reading of the manuscript .
this work was supported in part by the doe scidac grant de - fc02 - 06er25792 ( kmt , hf , syz , and mj ) and the u.s . national science foundation la - sigma grant eps-1003897 ( jr , jm , and mj ) .
supercomputer support was provided by the nsf teragrid under grant number tg - dmr100007 .
this research also used resources of the national center for computational sciences at oak ridge national laboratory , which is supported by the office of science of the u.s .
department of energy under contract no .
de - ac05 - 00or22725 .
this appendix describes a highly effective implementation of the symmetry - enforcing variant of the parquet formulation described earlier in the paper .
the communication bottleneck in this implementation is the expensive tensor rotations required to rotate the vertex ladders ( eqs .
[ eq : phi_ladder ] and [ eq : psi_ladder ] ) between the forms used in the bethe - salpeter equation , eqs .
[ sd_bsph_eq ] and [ sd_bspp_eq ] , to those used in the parquet equations , eqs .
[ gamma_d ]  [ gamma_t ] .
if we distribute the equations between the processes executing on the compute nodes of a parallel machine using the transfer momenta @xmath30 , the tensor rotations are done with an expensive all - to - all communication among those processes , in which every node needs to communicate with all the other nodes .
the mpi implementation of the all - to - all communication@xcite is a collective operation that is blocking , i.e. , each process has to wait until the message has been sent out .
the key aspect of our implementation is the decomposition of the required communication so that non - blocking communication primitives can be effectively utilized .
the non - blocking communication enables latency hiding by overlapping computations and communications .
four different forms of tensor rotations are required : @xmath123 note that the indices in subscripts and those in parenthesis are equivalent , with the latter only distinguished by also labeling the nodes where data is distributed .
the size of the tensors is @xmath124 , where @xmath51 is the number of momentum points times the number of discrete matsubara frequencies , i.e. , @xmath52 .
all indices are in modulo arithmetic at each of the @xmath125 dimensions , where @xmath126 ( the cluster dimension ) , and the @xmath127 is for the matsubara frequency .
because it takes many iterations ( up to a few hundred for low temperatures and strong coupling ) to obtain converged solutions , the total number of tensor rotations is significant and account for a large fraction of the computational time .
we use the hybrid mpi / openmp model for the computations .
the rank three tensors are decomposed and evenly distributed into @xmath128 virtual nodes .
each virtual node consists of a few cores .
the size of a virtual node ( i.e. , the number of cores ) is less than or equal to the size of a physical node .
specifically , we slice the rank three tensors to a set of two - dimensional arrays based on the index in parenthesis , e.g. , @xmath30 and @xmath129 for the left and right sides of eq .
[ rot1 ] , respectively .
then , each two dimensional matrix is assigned to a virtual node .
since we have @xmath130 layers of two dimensional slices , the total number of virtual nodes also becomes @xmath131 . in this scenario
, every rotation requires data communications among all nodes .
the following describes the data access patterns for our implementation of the tensor rotations .
* step 1 : * this step involves no mpi communication and is done before any data is sent between nodes .
the tensor elements are locally rearranged in order to collect specific elements to be grouped and sent to designated destination nodes .
the index in parenthesis of the tensors on the right of eqs .
[ rot1][rot4 ] represents the rank of a sending node in which a sliced two dimensional matrix resides . for rotations 1 and 2 ,
rank of sending node @xmath132 is @xmath133 for rotations 3 and 4 , @xmath132 is @xmath134 using eqs .
[ rot12s ] and [ rot34s ] , and applying these to the corresponding rotations , the two dimensional matrix elements are grouped based on the rank of destination node @xmath30 from a given sending node @xmath132 .
@xmath135 note that , here , @xmath132 is the node index ( the index of the sender ) and @xmath136 , so the @xmath28 and @xmath30 are the row and column indices of the matrix .
we assume column - major order data access in mpi data communications which distribute columns of matrix @xmath137 to nodes of rank @xmath30 in the next step .
* step 2 : * the columns of the two dimensional matrix @xmath137 are distributed among all nodes . at the sending nodes ,
each column of @xmath137 is sent to a destination node labeled by @xmath30 .
the standard approach is to use _
mpi_alltoall_. however , as we show later , this task can be done using different combinations of point - to - point communications.@xcite in particular , non - blocking communication protocols can be applied to overlap communications and local computations .
overall , this procedure is applied to all the tensor rotations and can be written as @xmath138 as shown in eq .
[ step2 ] , the rank of destination nodes is determined by the column index @xmath30 of @xmath137 in sending nodes . the rank of sending nodes becomes column index @xmath132 of @xmath139 in the receiving nodes .
the rank of sending nodes @xmath132 must be provided to receiving nodes in order to assign the correct column index to the received messages .
* step 3 : * once messages have arrived at the destination nodes , the columns of the two dimensional matrix @xmath139 are rearranged to complete the tensor rotations .
the column index of the rotated received matrix is related to the rank of the sending and receiving nodes by eqs .
[ rot12s ] and [ rot34s ] .
then , the rotations are finalized by using the following relations @xmath140 where @xmath30 is the index of a given receiving node and @xmath141 .
this step is a local process , i.e. , no internode communication is necessary .      while steps 1 and 3 are strictly local processes , step 2 is the only stage involving nonlocal mpi communications .
the nature of the collective communications among all nodes in step 2 makes it suited to the use of _ mpi_alltoall_. in such a case , step 2 can start only after the completion of step 1 .
because _ mpi_alltoall _ is a blocking communication , step 3 must wait to start until step 2 is finished .
therefore , the total elapsed time to complete a tensor rotation is the sum of elapsed times of the three steps . when the problem size is large , the communication efficiency of _ mpi_alltoall _ is reduced significantly due to the increased network complexity associated with the bandwidth and latency among all participating nodes .
our approach to handle these rotations more efficiently is to implement a latency hiding strategy by overlapping message communications ( step 2 ) and local computations ( steps 1 and 3 ) .    to enable this ,
we have developed our own version of a routine that performs communications from all nodes to all nodes .
at a basic level , the functionality of this routine is identical to that of the generic _ mpi_alltoall _ routine .
however , our routine allows further data manipulations such that local computations are embedded between communications in the following way : on the sending node , the first column of @xmath137 is computed from the equations of step 1 .
_ mpi_isend _ sends out the first column of @xmath142 while this column is being sent out , the next column of @xmath137 is prepared with step 1 .
this procedure is repeated until all @xmath130 columns of @xmath137 , the group of the selected elements from @xmath143 , are sent out .
this process overlaps steps 1 and 2 .
latency hiding is also implemented in receiving nodes .
we note that the sending nodes are also receiving nodes .
they only differ by whether they are operating in the sending or the receiving _
mode_. on the receiving nodes , _
mpi_irecv _ is set to receive messages from arbitrary nodes by using _
mpi_any_source _ as a tag identifying the source of the message . for efficiency reasons , _
mpi_irecv _ is posted before _
mpi_isend _ of the sending process .
mpi_test _ calls are used to check the completion of the arrival of the message .
once message arrival is confirmed , the rank of the node that sent this message can be identified by inquiring using _
mpi_status_. this provides @xmath132 to assign to a corresponding column and to be used in step 3 .
since the message arrival is column - by - column , the processing of each column of @xmath139 continues to step 3 while the next column is traveling through the network .
this process is repeated until all columns are completed .
this procedure completely overlaps steps 2 and 3 .
depending on the size of problem , it is desirable to define a virtual node containing several cores ( assuming multicore hardware architecture ) based on the memory availability per node . among the cores ,
mpi communications are assigned to one core .
the other cores are utilized by implementing openmp@xcite that parallelizes the local computational tasks in a node to all cores within the node .
thus , openmp thread depth is set to match with the total number of cores per virtual node .
specifically , we applied the _ do _ directive of openmp for iterations of index @xmath28 in the column selection processes of steps 1 and 3 .
we test the efficiency of this latency hiding scheme using a non - blocking protocol against the standard _ mpi_alltoall_.
all the experimental comparisons are conducted on the cray xt5 ( jaguar ) at the national center for computational sciences ( nccs ) at the oak ridge national laboratory .
jaguar consists of 12 cores per node , with six cores per numa ( non - uniform memory access ) node , and two numas per node .
first , we discuss hardware - driven constraints in implementing latency hiding . the non - blocking _
mpi_isend _ does not check for the arrivals of messages . with larger tensor size
, the node usage and the size of individual columns becomes large .
the _ mpi_isend _ from all participating nodes tries to dump a large column in each iteration .
the next iteration starts regardless of message arrivals in the receiving nodes .
as a consequence , a large amount of data rushes onto the network faster than the data can be absorbed by the receiving nodes .
eventually , this causes memory overflow to the system buffer assigned to the message processing unit . to avoid this
we have allocated more memory space to the system buffer .      for simplicity ,
we assign one virtual node to a physical node . on the jaguar cray xt5 ,
this means one virtual node containing 12 cores . to utilize all cores in a node ,
the value of openmp thread depth is set to 12 .
we gradually increase the problem size @xmath130 until jobs end with error indicating buffer overflow .
then , we set a higher buffer size by controlling environmental variable _
mpich_unex_buffer_size_. for every incidence of error , we add 60 mb buffer size . the default value of _
mpich_unex_buffer_size _ is 60 mb on jaguar xt5 ( the total number of cores is less than 50,000 ) .
the results are shown in the fig .
[ minbuffer ] .
up to @xmath144 , the 60 mb default buffer size is enough to handle the data traffic .
increasing @xmath130 further forces us to use a larger buffer size .
overall , the amount of added buffer size increases for larger problem sizes .
we note that the results presented in fig .
[ minbuffer ] are with the maximum number of cores per a virtual node .
smaller core usage per node alleviates the buffer restriction .
for example , hardware setup with a numa node per virtual node consumes less buffer memory due to the reduced total number of physical nodes participating in internode communication .
we did not observe buffer memory overflow with the generic blocking _
mpi_alltoall _ routine .
the performance of the latency hiding approach is evaluated in terms of wallclock time spent on a single tensor rotation and compared with the case of the standard _
mpi_alltoall _ applied for step 2 . for this , the elapsed time to complete the tensor rotation
is averaged over nine independent runs .
each run contains 40 repetitions of identical tensor rotations . at the end of each run ,
the elapsed time is also averaged for the 40 rotations . for all runs , we choose rotation 1 and the minimum buffer sizes shown in the fig .
[ minbuffer ] are assumed .
the comparison results are shown in fig .
[ compare1 ] . except for @xmath145 and @xmath146
, latency hiding outperforms the case without latency hiding in significant amount .
the performance differences are even higher for @xmath147 . for the _ mpi_alltoall
_ case , there is a sudden speed - up at @xmath148 .
we are exploring this behavior further .
we believe that it is caused by changes in the data traffic controlled by the hardware .
overall , latency hiding provides a higher speed - up for larger tensor sizes and core count . from the general trends
, it can be expected that two - fold or more efficiency improvement for @xmath130 greater than @xmath149 can be obtained by implementing latency hiding with our non - blocking adaptation of the all - to - all routine for tensor rotations .
e. anderson , z. bai , c. bischof , s. blackford , j. demmel , j. dongarra , j. du croz , a. greenbaum , s. hammarling , a. mckenney , and d. sorensen , _ lapack users guide , 3rd . edition _ ,
( society for industrial and applied mathematics , 1999 ) .