astrophysicists have always been keen to exploit technology to better understand the universe .
@xmath0-body simulations predate the use of digital computers with the use of light bulbs and light intensity measurements as an analog of gravity for manual simulations of a many - body self - gravitating system @xcite .
astrophysical objects including planets , individual stars , interstellar clouds , star clusters , galaxies , accretion disks , clusters of galaxies through to large scale structure have all the been the subject of numerical investigations .
the most challenging extreme is probably the evolution of space - time itself in computational general relativistic simulations of colliding neutron stars and black holes . since the advent of digital computers , improvements in storage and processing power
have dramatically increased the scale of achievable simulations .
this , in turn , has driven remarkable progress in algorithm development . increasing problem sizes have forced simulators who were once content with @xmath1 algorithms to pursue more complex @xmath2 and , with limitations , even @xmath3 algorithms and adaptivity in space and time . in this context
we present gasoline , a parallel @xmath0-body and gasdynamics code , which has enabled new light to be shed on a range of complex astrophysical systems .
the discussion is presented in the context of future directions in numerical simulations in astrophysics , including fundamental limitations in serial and parallel .
we begin by tracing past progress in computational astrophysics with an initial focus on self - gravitating systems ( @xmath0-body dynamics ) in section  [ gravity ] .
gasoline evolved from the pkdgrav parallel @xmath0-body tree code designed by @xcite .
the initial modular design of pkdgrav and a collaborative programming model using cvs for source code management has facilitated several simultaneous developments from the pkdgrav code base .
these include inelastic collisions , gas dynamics ( gasoline ) and star formation . in section  [ grav ]
we summarize the essential gravity code design , including the parallel data structures and the details of the tree code as applied to calculating gravitational forces .
we complete the section with a brief examination of the gravitational force accuracy .    in section  [ gas ]
we examine aspects of hydrodynamics in astrophysical systems to motivate smoothed particle hydrodynamics ( sph ) as our choice of fluid dynamics method .
we describe the gasoline sph implementation in section  [ sph ] , including neighbour finding algorithms and the cooling implementation .
interesting astrophysical systems usually exhibit a large range of time scales .
tree codes are very adaptable in space ; however , time - adaptivity has become important for leading edge numerical simulations . in section  [ stepping ] we describe our hierarchical timestepping scheme .
following on in section  [ performance ] we examine the performance of gasoline when applied to challenging numerical simulations of real astrophysical systems . in particular , we examine the current and potential benefits of multiple timesteps for time adaptivity in areas such as galaxy and planet formation .
we present astrophysically oriented tests used to validate gasoline in section  [ tests ] .
we conclude by summarizing current and proposed applications for gasoline .
gravity is the key driving force in most astrophysical systems . with assumptions of axisymmetry or perturbative
approaches an impressive amount of progress has been made with analytical methods , particularly in the areas of solar system dynamics , stability of disks , stellar dynamics and quasi - linear aspects of the growth of large scale structure in the universe . in many systems of interest , however , non - linear interactions play a vital role .
this ultimately requires the use of self - gravitating @xmath0-body simulations .
fundamentally , solving gravity means solving poisson s equation for the gravitational potential , @xmath4 , given a mass density , @xmath5 : @xmath6 where @xmath7 is the newtonian gravitational constant . in a simulations with discrete bodies
it is common to start from the explicit expression for the acceleration , @xmath8 on a given body in terms of the sum of the influence of all other bodies , @xmath9 where the @xmath10 and @xmath11 are the position and masses of the bodies respectively .
when attempting to model collisionless systems , these same equations are the characteristics of the collisionless boltzmann equation , and the bodies can be thought of as samples of the distribution function . in practical work
it is essential to soften the gravitational force on some scale @xmath12 to avoid problems with the integration and to minimize two - body scattering in cases where the bodies represent a collisionless system .
early @xmath0-body work such as studies of relatively small stellar systems were approached using a direct summation of the forces on each body due to every other body in the system @xcite .
this direct @xmath1 approach is impractical for large numbers of bodies , @xmath0 , but has enjoyed a revival due to incredible throughput of special purpose hardware such as grape @xcite .
the grape hardware performs the mutual force calculation for sets of bodies entirely in hardware and remains competitive with other methods on more standard floating hardware up to @xmath13 .    a popular scheme for larger @xmath0 is the particle - mesh ( pm ) method which has long been used in electrostatics and plasma physics .
the adoption of pm was strongly related to the realization of the existence of the @xmath2 fast fourier transform ( fft ) in the 1960 s .
the fft is used to solve for the gravitational potential from the density distribution interpolated onto a regular mesh . in astrophysics sub - mesh resolution
is often desired , in which case the force can be corrected on sub - mesh scales with local direct sums as in the particle - particle particle - mesh ( p@xmath14 m ) method .
pm is popular in stellar disk dynamics , and p@xmath14 m has seen widespread adoption in cosmology .
pm codes have similarities with multigrid and other iterative schemes .
however , working in fourier space is not only more efficient , but it also allows efficient force error control through optimization of the green s function and smoothing .
fourier methods are widely recognised as ideal for large , fairly homogeneous , periodic gravitating simulations .
multigrid has some advantages in parallel due to the local nature of the iterations .
the particle - particle correction can get expensive when particles cluster in a few cells .
both multigrid and p@xmath14 m can adapt to address this via a hierarchy of sub - meshes .
with this approach the serial slow down due to heavy clustering tends toward a fixed multiple of the unclustered run speed .    in applications such as galactic dynamics where high resolution in phase space is desirable and particle noise is problematic , the smoothed gravitational potentials provided by an expansion in modes is useful .
pm does this with fourier modes ; however , a more elegant approach is the self - consistent field method ( scf ) @xcite . using a basis set closely matched to the evolving system
dramatically reduces the number of modes to be modelled ; however , the system must remain close to axi - symmetric and similar to the basis .
scf parallelizes well and is also used to generate initial conditions such as stable individual galaxies that might be used for merger simulations .
a current popular choice is to use tree algorithms which are inherently @xmath2 .
this approach recognises that details of the remote mass distribution become less important for accurate gravity with increasing distance .
thus the remote mass distribution can be expanded in multipoles on the different size scales set by a tree - node hierarchy . the appropriate scale to use
is set by the opening angle subtended by the tree - node bounds relative to the point where the force is being calculated .
the original barnes - hut @xcite method employed oct - trees but this is not especially advantageous , and other trees also work well @xcite .
the tree approach can adapt to any topology , and thus the speed of the method is somewhat insensitive to the degree of clustering .
once a tree is built it can also be re - used as an efficient search method for other physics such as particle based hydrodynamics .
a particularly useful property of tree codes is the ability to efficiently calculate forces for a subset of the bodies .
this is critical if there is a large range of time - scales in a simulation and multiple independent timesteps are employed .
at the cost of force calculations no longer being synchronized among the particles substantial gains in time - to - solution may be realized .
multiple timesteps are particularly important for current astrophysical applications where the interest and thus resolution tends to be focused on small regions within large simulated environments such as individual galaxies , stars or planets .
dynamical times can become very short for small numbers of particles .
p@xmath14 m codes are faster for full force calculations but are difficult to adapt to calculate a subset of the forces .    in order to treat periodic boundaries with tree codes it is necessary to effectively infinitely replicate the simulation volume which may be approximated with an ewald summation @xcite .
an efficient alternative which is seeing increasing use is to use trees in place of the direct particle - particle correction to a particle - mesh code , often called tree - pm @xcite .    the fast multipole method
( fmm ) recognises that the applied force as well as the mass distribution may be expanded in multipoles .
this leads to a force calculation step that is @xmath3 as each tree node interacts with a similar number of nodes independent of @xmath0 and the number of nodes is proportional to the number of bodies . building the tree
is still @xmath2 but this is a small cost for simulations up to @xmath15 @xcite .
the method used spherical harmonic expansions where the desired accuracy is achieved solely by changing the order of the expansions . for the majority of astrophysical applications
the allowable force accuracies make it much more efficient to use fixed order cartesian expansions and an opening angle criterion similar to standard tree codes @xcite .
this approach has the nice property of explicitly conserving momentum ( as do pm and p@xmath14 m codes ) .
the prefactor for cartesian fmm is quite small so that it can outperform tree codes even for small @xmath0 @xcite .
it is a significantly more complex algorithm to implement , particularly in parallel .
one reason widespread adoption has not occurred is that the speed benefit over a tree code is significantly reduced when small subsets of the particles are having forces calculated ( e.g. for multiple timesteps ) .
gasoline is built on the pkdgrav framework and thus uses the same gravity algorithms .
the pkdgrav parallel @xmath0-body code was designed by stadel and developed in conjunction with quinn beginning in the early 90 s .
this includes the parallel code structure and core algorithms such as the tree structure , tree walk , hexadecapole multipole calculations for the forces and the ewald summation .
there have been additional contributions to the gravity code by richardson and wadsley in the form of minor algorithmic modifications and optimizations .
the latter two authors became involved as part of the collisions and gasoline extensions of the original pkdgrav code respectively .
we have summarized the gravity method used in the production version of gasoline without getting into great mathematical detail . for full technical details on pkdgrav
the reader is referred to .
gasoline is fundamentally a tree code .
it uses a variant on the k - d tree ( see below ) for the purpose of calculating gravity , dividing work in parallel and searching . designed pkdgrav from the start as a parallel code .
there are four layers in the code .
the master layer is essentially serial code that orchestrates overall progress of the simulation .
the processor set tree ( pst ) layer distributes work and collects feedback on parallel operations in an architecture independent way using mdl .
the machine dependent layer ( mdl ) is a relatively short section of code that implements remote procedure calls , effective memory sharing and parallel diagnostics .
all processors other than the master loop in the pst level waiting for directions from the single process executing the master level .
directions are passed down the pst in a tree based @xmath16 procedure that ends with access to the fundamental bulk particle data on every node at the pkd level .
the parallel k - d ( pkd ) layer is almost entirely serial but for a few calls to mdl to access remote data .
the pkd layer manages local trees for gravity and particle data and is where the physics is implemented .
this modular design enables new physics to be coded at the pkd level without requiring detailed knowledge of the parallel framework .
pkdgrav departed significantly from the original @xmath0-body tree code designs of by using 4th ( hexadecapole ) rather than 2nd ( quadrupole ) order multipole moments to represent the mass distribution in cells at each level of the tree .
this results in less computation for the same level of accuracy : better pipelining , smaller interaction lists for each particle and reduced communication demands in parallel . the current implementation in gasoline uses reduced moments that require only @xmath17 terms to be stored for the @xmath18 moment .
for a detailed discussion of the accuracy and efficiency of the tree algorithm as a function the order of the multipoles used see @xcite and @xcite .
the original k - d tree @xcite was a balanced binary tree .
gasoline divides the simulation in a similar way using recursive partitioning . at the pst level
this is parallel domain decomposition and the division occurs on the longest axis to recursively divide the work among the remaining processors .
even divisions occur only when an even number of processors remains .
otherwise the work is split in proportion to the number of processors on each side of the division .
thus , gasoline may use arbitrary numbers of processors and is efficient for flat topologies without adjustment . at the pst level
each processor has a local rectangular domain within which a local binary tree is built .
the structure of the lower tree is important for the accuracy of gravity and efficiency of other search operations such as neighbour finding required for sph .
oct - trees are traditional in gravity codes .
in contrast , the key data structures used by gasoline are spatial binary trees .
one immediate gain is that the local trees do not have to respect a global structure and simply continue from the pst level domain decomposition in parallel .
the binary tree determines the hierarchical representation of the mass distribution with multipole expansions , of which the root node or cell encloses the entire simulation volume .
the local gravity tree is built by recursively bisecting the longest axis of each cell which keeps the cells axis ratios close to one .
in contrast , cells in standard k - d trees can have large axis ratios which lead to large multipoles and with correspondingly large gravity errors . at each level the dimensions of the cells are squeezed to just contain the particles .
this overcomes the empty cell problem of un - squeezed spatial bisection trees .
the sph tree is currently a balanced k - d tree ; however , testing indicates that the efficiency gain is slight and it is not worth the cost of an additional tree build .
the top down tree build process is halted when @xmath19 or fewer particles remain in a cell .
stopping the tree with @xmath19 particles in a leaf cell reduces the storage required for the tree and makes both gravity and search operations more efficient . for these purposes
@xmath20 is a good choice .
once the gravity tree has been built there is a bottom - up pass starting from the buckets and proceeding to the root , calculating the center of mass and the multipole moments of each cell from the center of mass and moments of each of its two sub - cells .
gasoline calculates the gravitational accelerations using the well known tree - walking procedure of the algorithm , except that it collects interactions for entire buckets rather than single particles .
this amortizes the cost of tree traversal for a bucket over all its particles .    in the tree building phase
, gasoline assigns to each cell of the tree an _ opening radius _ about its center - of - mass .
this is defined as , @xmath21 where @xmath22 is the maximum distance from a particle in the cell to the center - of - mass of the cell .
the _ opening angle _ ,
@xmath23 , is a user specified accuracy parameter which is similar to the traditional @xmath23 parameter of the barnes - hut code ; notice that decreasing @xmath23 in equation [ eq:1 ] , increases @xmath24 .
the opening radii are used in the _ walk _ phase of the algorithm as follows : for each bucket @xmath25 , gasoline starts descending the tree , opening those cells whose @xmath24 intersect with @xmath25 ( see figure [ fig:2.2 ] ) .
if a cell is opened , then gasoline repeats the intersection - test with @xmath25 for the cell s children otherwise , the cell is added to the _ particle - cell interaction list _ of @xmath25 . when gasoline reaches the leaves of the tree and a _ bucket _
@xmath26 is opened , all of @xmath26 s particles are added to the _ particle - particle interaction _ list of @xmath25 .    once the tree has been traversed in this manner we can calculate the gravitational acceleration for each particle of @xmath25 by evaluating the interactions specified in the two lists .
gasoline uses the hexadecapole multipole expansion to calculate particle - cell interactions .
the particle - particle interactions are softened to lessen two - body relaxation effects that compromise the attempt to model continuous fluids , including the collisionless dark matter fluid . in gasoline
the particle masses are effectively smoothed in space using the same spline form employed for sph in section  [ sph ] .
this means that the gravitational forces vanish at zero separation and return to newtonian @xmath27 at a separation of @xmath28 where @xmath29 is the gravitational softening applied to each particle . in this sense the gravitational forces
are well matched to the sph forces .
a disadvantage of tree codes is that they must deal explicitly with periodic boundary conditions ( as are usually required for cosmology ) .
gasoline incorporates periodic boundaries via the ewald summation technique where the force is divided into short and long range components .
the gasoline implementation differs from that of in using a new technique due to based on an hexadecapole moment expansion of the fundamental cube to drastically reduce the work for the long range ewald sum that approximates the infinite periodic replications . for each particle
the computations are local and fixed , and thus the algorithm scales exceedingly well .
there is still substantial additional work in periodic simulations because particles interact with cells and particles in nearby replicas of the fundamental cube .      the tree opening criteria places a bound on the relative error due to a single particle - cell interaction .
as gasoline uses hexadecapoles the error bound improves rapidly as the opening angle , @xmath23 , is lowered .
the relationship between the typical ( e.g. rms ) relative force error and opening angle is not a straight - forward power - law in @xmath23 because the net gravitational forces on each particle result from the cancellation of many opposing forces . in figure
[ accfig ] , we show a histogram of the relative acceleration errors for a cosmological gasoline simulations at two different epochs for a range of opening angles . we have plotted the error curves as cumulative fractions to emphasize the limited tails to high error values . for typical gasoline simulations we commonly use @xmath30 which gives an _ rms _ relative error of 0.0039 for the clustered final state referred to by the left panel of figure  [ accfig ] .
the errors relative to the mean acceleration ( figure  [ accfigabs ] ) are larger ( rms 0.0083 ) but of less interest for highly clustered cases .
= 5.5 in    = 5.5 in    as a simulated medium becomes more uniform the net gravitational accelerations approach zero . in a tree code
the small forces result from the cancellation of large opposing forces .
this is the case in cosmology at early times when the perturbations ultimately leading to cosmological structures are still small . in this situation
it is essential to tighten the cell - opening criterion to increase the relative accuracy so that the net accelerations are sufficiently accurate .
for example in the right hand panels of figures  [ accfig ] and  [ accfigabs ] the errors are larger , and at @xmath31 , the _ rms _ relative error is 0.041 for @xmath30 .
however , here the absolute errors are lower by nearly a factor of two in rms ( 0.026 ) as shown in figure  [ accfigabs ] . at early times when the medium fairly homogeneous growth is driven by local gradients , and thus acceleration errors normalized to the mean acceleration provide the better measure of accuracy as large relative errors are meaningless for accelerations close to zero . to ensure accurate integrations we switch to values such as @xmath32 before @xmath33 , giving an _ rms _ relative error of 0.0077 and an _ rms _ error of 0.0045 normalized to the mean absolute acceleration for the @xmath31 distribution ( a reasonable start time for a cosmological simulation on this scale ) .
astrophysical systems are predominantly at very low physical densities and experience wide - ranging temperature variations .
most of the material is in a highly compressible gaseous phase . in general
this means that a perfect adiabatic gas is an excellent approximation for the system .
microscopic physical processes such as shear viscosity and diffusion can usually be neglected .
high - energy processes and the action of gravity tend to create large velocities so that flows are both turbulent and supersonic : strong shocks and very high mach numbers are common .
radiative cooling processes can also be important ; however , the timescales can often be much longer or shorter than dynamical timescales . in the latter case isothermal gas
is often assumed for simplicity . in many areas of computational astrophysics ,
particularly cosmology , gravity tends to be the dominating force that drives the evolution of the system .
visible matter , usually in the form of radiating gas , provides the essential link to observations .
radiative transfer is always present but may not significantly affect the energy and thus pressure of the gas during the simulation .
fluid dynamics solvers can be broadly classified into eulerian or lagrangian methods .
eulerian methods use a fixed computational mesh through which the fluid flows via explicit advection terms .
regular meshes provide for ease of analysis and thus high order methods such as ppm @xcite and tvd schemes have been developed .
the inner loops of mesh methods can often be pipelined for high performance .
lagrangian methods follow the evolution of fluid parcels via the full ( comoving ) derivatives .
this requires a deforming mesh or a mesh - less method such as smoothed particle hydrodynamics ( sph ) @xcite .
data management is more complex in these methods ; however , advection is handled implicitly and the simulation tends to naturally adapt to follow density contrasts .
large variations in physical length scales in astrophysics have limited the usefulness of eulerian grid codes
. adaptive mesh refinement ( amr ) @xcite overcomes this at the cost of data management overheads and increased code complexity . in the cosmological context
there is the added complication of dark matter .
there is more dark matter than gas in the universe so it dominates the gravitational potential .
perturbations present on all scales in the dark matter guide the formation of gaseous structures including galaxies and the first stars .
a fundamental limit to amr in computational cosmology is matching the amr resolution to the underlying dark matter resolution .
particle based lagrangian methods such as sph are well matched to this constraint .
a useful feature of lagrangian simulations is that bulk flows ( which can be highly supersonic in the simulation frame ) do not limit the timesteps .
particle methods are well suited to rapidly rotating systems such as astrophysical disks where arbitrarily many rotation periods may have to be simulated ( e.g. sph explicitly conserves angular momentum ) .
a key concern for all methods is correct angular momentum transport .
smoothed particle hydrodynamics is an approach to hydrodynamical modelling developed by and .
it is a particle method that does not refer to grids for the calculation of hydrodynamical quantities : all forces and fluid properties are found on moving particles eliminating numerically diffusive advective terms .
the use of sph for cosmological simulations required the development of variable smoothing to handle huge dynamic ranges @xcite .
sph is a natural partner for particle based gravity .
sph has been combined with p@xmath14 m @xcite , adaptive p@xmath14 m , grape @xcite and tree gravity @xcite .
parallel codes using sph include hydra mpi , parallel treesph @xcite and the gadget tree code @xcite .
the basis of the sph method is the representation and evolution of smoothly varying fluid quantities whose value is only known at disordered discrete points in space occupied by particles .
particles are the fundamental resolution elements comparable to cells in a mesh .
sph functions through local summation operations over particles weighted with a smoothing kernel , @xmath34 , that approximates a local integral .
the smoothing operation provides a basis from which to obtain derivatives .
thus , estimates of density related physical quantities and gradients are generated .
the summation aspect led to sph being described as a monte carlo type method ( with @xmath35 errors ) however it was shown by that the method is more closely related to interpolation theory with errors @xmath36 , where @xmath37 is the number of dimensions .
a general smoothed estimate for some quantity @xmath38 at particle @xmath39 given particles @xmath40 at positions @xmath41 takes the form : @xmath42 where @xmath43 is a kernel function and @xmath44 is a smoothing length indicative of the range of interaction of particle @xmath40 .
it is common to convert this particle - weighted sum to volume weighting using @xmath45 in place of @xmath46 where the @xmath47 and @xmath48 are the particle masses and densities , respectively . for momentum and energy conservation in the force terms
, a symmetric @xmath49 is required .
we use the kernel - average first suggested by ,    @xmath50    for @xmath51 we use the standard spline form with compact support where @xmath52 if @xmath53 @xcite .    we employ a fairly standard implementation of the the hydrodynamics equations of motion for sph @xcite . density is calculated from a sum over particle masses @xmath47 , @xmath54 the momentum equation is expressed , @xmath55 where @xmath56 is pressure , @xmath57 velocity and the artificial viscosity term @xmath58 is given by , @xmath59 where @xmath60 , @xmath61 and @xmath62 is the sound speed .
@xmath63 and @xmath64 are coefficients we use for the terms representing shear and von neumann - richtmyer ( high mach number ) viscosities respectively . when simulating strongly rotating systems we use the multiplicative switch @xmath65 to suppress the viscosity in non - shocking , shearing environments .
the pressure averaged energy equation ( analogous to equation  [ sphmom ] ) conserves energy exactly in the limit of infinitesimal time steps but may produce negative energies due to the @xmath56 term if significant local variations in pressure occur .
we employ the following equation which also conserves energy exactly in each pairwise exchange but is dependent only on the local particle pressure ,    @xmath66    where @xmath67 is the internal energy of particle @xmath39 , which is equal to @xmath68 for an ideal gas . in this formulation entropy
is closely conserved making it similar to alternative entropy integration approaches such as that proposed by .
sph derivative estimates , such as the rate of change of thermal energy , vary sufficiently from the exact answer so that over a full cosmological simulation the cooling due to universal expansion will be noticeably incorrect . in this case
we use comoving divergence estimates and add the cosmological expansion term explicitly .
finding neighbours of particles is a useful operation .
a neighbour list is essential to sph , but it can also be a basis for other local estimates , such as a dark matter density , as a first step in finding potential colliders or interactors via a short range force .
stadel developed an efficient search algorithm using priority queues and a k - d tree ball - search to locate the @xmath69-nearest neighbours for each particle ( freely available as the smooth utility at http://www-hpcc.astro.washington.edu ) .
for gasoline we use a parallel version of the algorithm that caches non - local neighbours via mdl .
the sph interaction distance @xmath70 is set equal to the @xmath69-th neighbour distance from particle @xmath39 .
we use an exact number of neighbours .
the number is set to a value such as 32 or 64 at start - up .
we have also implemented a minimum smoothing length which is usually set to be comparable to the gravitational softening .    to calculate the fluid accelerations using sph we perform two smoothing operations .
first we sum densities and then forces using the density estimates . to get a kernel - averaged sum for every particle ( equations [ sum],[kernelavg ] )
it is sufficient to perform a gather operation over all particles within @xmath70 of every particle @xmath39 .
if only a subset of the particles are active and require new forces , all particles for which the active set are neighbours must also perform a gather so that they can scatter their contribution to the active set .
finding these scatter neighbours requires solving the @xmath69-inverse nearest neighbour problem , an active research area in computer science .
fortunately , during a simulation the change per step in @xmath71 for each particle typically less than 2 - 3 percent , so it is sufficient to find scatter neighbours , @xmath40 for which some active particles is within @xmath72 .
we use a tree search where the nodes contain sph interaction bounds for their particles estimated with @xmath73 .
a similar scheme has been employed by . for the forces
sum the inactive neighbours need density values which can be estimated using the continuity equation or calculated exactly with a second inverse neighbour search .      in astrophysical systems
the cooling timescale is usually short compared to dynamical timescales which often results in temperatures that are close to an equilibrium set by competing heating and cooling processes .
we have implemented a range of cases including : adiabatic ( no cooling ) , isothermal ( instant cooling ) , and implicit energy integration .
hydrogen and helium cooling processes have been incorporated .
ionization fractions are calculated assuming equilibrium for efficiency .
gasoline optionally adds heating due to feedback from star formation , an uniform uv background or using user defined functions .
the implicit integration uses a stiff equation solver assuming that the hydrodynamic work and density are constant across the step .
the second order nature of the particle integration is maintained using an implicit predictor step when needed .
internal energy is required on every step to calculate pressure forces on the active particles .
the energy integration is @xmath3 but reasonably expensive . to avoid integrating energy for every particle on the smallest timestep
we extrapolate each particle forward on its individual dynamical timestep and use linear interpolation to estimate internal energy at intermediate times as required .
the range of physical scales in astrophysical systems is large . for example
current galaxy formation simulations contain 9 orders of magnitude variation in density .
the dynamical timescale for gravity scales as @xmath74 and for gas it scales as @xmath75 . for an adiabatic gas
the local dynamical time scales as @xmath76 . with gas cooling ( or the isothermal assumption ) simulations can achieve very high gas densities . in most cases
gas sets the shortest dynamical timescales , and thus gas simulations are much more demanding ( many more steps to completion ) than corresponding gravity only simulations .
time adaptivity can be very effective in this regime .
= 5.5 in    gasoline incorporates the timestep scheme described as kick - drift - kick ( kdk ) in .
the scheme uses a fixed overall timestep . starting with all quantities synchronised
, velocities and energies are updated to the half - step ( half - kick ) , followed by a full step position update ( drift ) .
the positions alone are sufficient to calculate gravity at the end of the step ; however , for sph , velocities and thermal energies are also required and obtained with a predictor step using the old accelerations .
finally another half - kick is applied synchronising all the variables . without gas forces
this is a symplectic leap - frog integration .
the leap - frog scheme requires only one force evaluation and minimum storage .
while symplectic integration is not required in cosmology it is critical in solar system integrations or any system where natural instabilities occur over very many dynamical times .
an arbitrary number of sub - stepping rungs factors of two smaller may be used as shown in figure  [ timestep ] .
the scheme is no longer strictly symplectic if particles change rungs during the integration which they generally must do to satisfy their individual timestep criteria .
after overheads , tree - based force calculation scales approximately with the number of active particles so large speed - ups may be realised in comparison to single stepping ( see section  [ performance ] ) .
figure  [ timestep ] compares kdk with drift - kick - drift ( dkd ) , also discussed in .
for kdk the force calculations for different rungs are synchronised . in the limit of many rungs
this results in half as many force calculation times with their associated tree builds compared to dkd .
kdk also gives significantly better momentum and energy conservation for the same choice of timestep criteria .
we use standard timestep criteria based on the particle acceleration , and for gas particles , the courant condition and the expansion cooling rate .
@xmath77    @xmath78 is the maximum value of @xmath79 ( from equation  [ artifvisc ] ) over interactions between pairs of sph particles .
for cosmology in place of the comoving velocity @xmath80 , we use the momentum @xmath81 which is canonical to the comoving position , @xmath82 .
as described in detail in appendix a of @xcite , this results in a separable hamiltonian which may be integrated straightforwardly using the drift and kick operators , @xmath83 where @xmath4 is the perturbed potential given by @xmath84 , @xmath85 is the cosmological expansion factor and @xmath86 is the mean density .
thus no hubble drag term is required in the equations of motion , and the integration is perfectly symplectic in the single stepping case .
= 5.5 in    gasoline was built on the pkdgrav @xmath0-body code which achieves excellent performance on pure gravity in serial and parallel .
performance can be measured in floating point calculations per second but the measure of most interest to researchers is the science rate .
we define this in terms of resolution elements updated per unit wallclock time . in the case of gasoline
this is particles updated per second .
this measure directly determines how long it takes to finish the simulation .
figure  [ fig : particlespersecond ] shows the scaling of particles per second with numbers of alpha processors for a single update for velocities and positions for all particles .
this requires gravitational forces for all particles and sph for the gas particles ( half the total ) .
the simulation used is a @xmath87 gas and @xmath87 dark matter particle , purely adiabatic cosmological simulation ( @xmath88cdm ) in a 200 mpc box at the final time ( redshift , @xmath89 ) . at this time
the simulation is highly clustered locally but material is still well distributed throughout the volume .
thus , it is still possible to partition the volume to achieve fairly even work balancing among a fair number of processors . as a rule of thumb we aim to have around 100,000 particles per processor .
as seen in the figure , the gravity scales very well . for pure gravity around @xmath90 scaling efficiency can be achieved in this case
. the cache based design has small memory overheads and uses the large amounts of gravity work to hide the communication overheads associated with filling the cache with remote particle and cell data . with gas the efficiency is lowered because the data structures to be passed are larger and there is less work per data element to hide the communication costs . thus the computation to communication ratio
is substantially lowered with more processors .
fixed costs such as the treebuilds for gravity and sph scale well , but the gas - only costs peel away from the more efficient gravity scaling .
when the overall rate is considered , gasoline is still making substantial improvements in the time to solution going from 32 to 64 processors ( lowest curve in the figure ) .
the overall rate includes costs for domain decomposition and @xmath3 updates of the particle information such as updating the positions from the velocities .
the net science rate compares well with other parallel tree and sph codes .
= 5.5 in    figure  [ fig : particlespersecond ] only treats single timestepping .
multiple timesteps provide an additional speed - up in the science rate of a factor typically in the range of 2 - 5 that is quite problem dependent .
the value of individual particle timesteps is illustrated separately in figure  [ fig : multistep ] .
for this example we analysed the time spent by gasoline doing a single major step ( around 13 million years ) of a million particle galaxy formation simulation at redshift @xmath91 .
the simulation used a range of 128 in substeps per major step .
the top curve in the figure is the cost for single stepping .
it rises exponentially , doubling with every new bin , and drops off because the last bin was not always occupied .
sample numbers of particles in the bins ( going from 0 to 7 ) were 473806 , 80464 , 63708 , 62977 , 85187 , 129931 , 1801 and 20 respectively .
using individual timesteps gasoline was running 4 times faster than an equivalent single stepping code .
the test was performed in parallel on 8 processors , and it is heartening that despite the added difficulties of load balancing operations on subsets of the particles the factor of 4 benefit was realized .
tree building and other fixed costs that do not scale with the number of active particles can be substantially reduced using tree - repair and related methods which would bring the speed - up to around 5 . in the limit of uniform costs per particle independent of the number of active particles the speed - up would approach 10 . in this example
the timestep range was limited partly due to restrictions imposed on the sph smoothing length . in future work
we anticipate a wider range of timestep bins . with a few particles consistently on a timestep 1/256th of the major step
the theoretical speed - up factor is over 30 . in practice
, overheads will always limit the achievable speed - up . in the limit of very large numbers of rungs
the current code would asymptotically approach a factor of 10 speedup . with the fixed costs of tree
builds and domain decomposition made to scale with the number of active particles ( e.g. tree - repair ) , the asymptotic speed up achievable with gasoline would be 24 times .
the remaining overheads are increasingly difficult to address . for small runs , such as the galaxy formation example used here
, the low numbers of the particles on the shortest timesteps make load balancing difficult .
if more than 16 processors are used with the current code on this run there will be idle processors for a noticeable fraction of the time .
though the time to solution is reduced with more processors , it is an inefficient use of computing resources .
the ongoing challenge is to see if better load balance through more complex work division offsets increases in communication and other parallel overheads .
= 5.5 in      there are three key points to keep in mind when evaluating the performance of sph on highly symmetric tests .
the first is that the natural particle configuration is a semi - regular three dimensional glass rather than a regular mesh .
the second is that individual particle velocities are smoothed before they affect the dynamics so that the low level noise in individual particle velocities is not particularly important .
the dispersion in individual velocities is related to continuous settling of the irregular particle distribution .
this is particularly evident after large density changes .
thirdly , the sph density is a smoothed estimate .
any sharp step in the number density of particles translates into a density estimate that is smooth on the scale of @xmath92 particle separations .
when relaxed irregular particle distributions are used , sph resolves density discontinuities close to this limit . as a lagrangian method sph
can also resolve contact discontinuities just as tightly without the advective spreading of some eulerian methods .
we have performed standard shock tube tests used for the original treesph @xcite .
we find the best results with the pairwise viscosity of equation  [ artifvisc ] which is marginally better than the bulk viscosity formulation for this test .
the one dimensional tests often shown do not properly represent the ability of sph to model shocks on more realistic problems .
the results of figure  [ tube ] demonstrate that sph can resolve discontinuities in a shock tube very well when the problem is setup to be comparable to the environment in a typical three dimensional simulation .
= 5.5 in    the shocks of the previous example have a fairly low mach number compared to astrophysical shocks found in collapse problems . @xcite
first introduced a spherical adiabatic collapse as a test of gasdynamics with gravity .
this test is nearly equivalent to the self - similar collapse of and has comparable shock strengths .
we compare gasoline results on this problem with very high resolution 1d lagrangian mesh solutions in figure  [ adcollfig ] .
we used a three - dimensional glass initial condition .
the solution is excellent with 2 particle spacings required to model the shock .
the deviation at the inner radii is a combination of the minimum smoothing length ( 0.01 ) and earlier slight over - production of entropy at a less resolved stage of the collapse .
the pre - shock entropy generation ( bottom left panel of fig .
[ adcollfig ] occurs in any strongly collapsing flow and is present for both the pairwise ( equation  [ artifvisc ] ) and divergence based artificial viscosity formulations .
the post - shock entropy values are correct .
the rotating isothermal cloud test examines the angular momentum transport in an analogue of a typical astrophysical collapse with cooling .
grid methods must explicitly advect material across cell boundaries which leads to small but systematic angular momentum non - conservation and transport errors .
sph conserves angular momentum very well , limited only by the accuracy of the time integration of the particle trajectories .
however , the sph artificial viscosity that is required to handle shocks has an unwanted side - effect in the form of viscous transport away from shocks .
the magnitude of this effect scaling with the typical particle spacing , and it can be combatted effectively with increased resolution .
the switch detects shearing regions so that the viscosity can be reduced where strong compression is not also present .
we modelled the collapse of a rotating , isothermal gas cloud .
this test is similar to a tests performed by @xcite and @xcite except that we have simplified the problem in the manner of @xcite .
we use a fixed @xcite ( concentration @xmath93=11 , mass=@xmath94 ) potential without self - gravity to avoid coarse gravitational clumping with associated angular momentum transport .
this results in a disk with a circular velocity of 220 km / s at 10 kpc .
the @xmath95 , @xmath96 kpc gas cloud was constructed with particle locations evolved into a uniform density glass initial condition and set in solid body rotation ( @xmath97 ) corresponding to a rotation parameter @xmath98 .
the gas was kept isothermal at @xmath99 k rather than using a cooling function to make the test more general .
the corresponding sound speed of @xmath100 km / s implies substantial shocking during the collapse .
= 5.5 in    we ran simulations using 64 , 125 , 500 and 4000 particle clouds for @xmath101 gyr ( 43 rotation periods at @xmath100 kpc ) .
we present results for monaghan viscosity , bulk viscosity ( hernquist - katz ) and our default set - up : monaghan viscosity with the balsara switch .
results showing the effect of resolution are shown in figure  [ figrotsummary ] . both monaghan viscosity with the balsara switch and bulk viscosity result in minimal angular momentum transport .
the monaghan viscosity without the switch results in steady transport of angular momentum outwards with a corresponding mass inflow .
the monaghan viscosity case was run with and without multiple timesteps . with multiple timesteps
the particle momentum exchanges are no longer explicitly synchronized and the integrator is no longer perfectly symplectic . despite this the evolution is very similar , particularly at high resolution .
the resolution dependence of the artificial viscosity is immediately apparent as the viscous transport drastically falls with increasing particle numbers .
it is worth noting that in hierarchical collapse processes the first collapses always involve small particle numbers .
our results are consistent with the results of @xcite . in that paper self - gravity of the gas
was also included , providing an additional mechanism to transport angular momentum due to mass clumping . as a result ,
their disks show a gradual transport of angular momentum even with the balsara switch ; however , this transport was readily reduced with increased particle numbers .
= 5.5 in    in figure  [ figrotzoom ] we focus on the bounce that occurs during the collapse using bulk viscosity .
bulk viscosity converges very slowly towards the correct solution and displays unwanted numerical behaviours even at high particle numbers . in cosmological adiabatic tests
it tends to generate widespread heating during the early ( pre - shock ) phases of collapse .
thus we favour monaghan artificial viscosity with the balsara switch .
= 5.5 in    the cluster comparison involved simulating the formation of an extremely massive galaxy cluster with adiabatic gas ( no cooling ) .
profiles and bulk properties of the cluster were compared for many widely used codes .
we ran this cluster with gasoline at two resolutions : @xmath102 and @xmath103 .
these were the two resolutions which other sph codes used in the original paper .
profiles of the simulated cluster have previously been shown in .
as shown in figure  [ figcluscomp ] , the gasoline cluster bulk properties are within the range of values for the other codes at @xmath89 .
the gasoline x - ray luminosity is near the high end ; however , the results are consistent with those of the other codes .
the large variation in the bulk properties among codes was a notable result of the original comparison paper .
we investigated the source of this variation and found that a large merger was occurring at @xmath89 with a strong shock at the cluster centre .
the timing of this merger is significantly changed from code - to - code or even within the same code at different resolutions as noted in the original paper . running the cluster with gasoline with @xmath102 ( solid ) instead @xmath103 ( dotted ) particles changes the timing noticeably .
the different resolutions have modified initial waves which affect overall timing and levels of substructure .
these differences are highlighted by the merger event near @xmath89 .
as shown in the figure , the bulk properties are changing very rapidly in a short time .
this appears to explain a large part of the code - to - code variation in quantities such as mean gas temperature and the kinetic to thermal energy ratio .
timing issues are likely to be a feature of future cosmological code comparisons .
the authors have applied gasoline to provide new insight into clusters of galaxies , dwarf galaxies , galaxy formation , gas - giant planet formation and large scale structure .
these applications are a testament to the robustness , flexibility and performance of the code
.    used gasoline at high resolution to show convincingly that giant planets can form in marginally unstable disks around protostars in just a few orbital periods .
these simulation used 1 million gas particles and were integrated for around thousand years until dense proto - planets formed .
current work focusses on improving the treatment of the heating and cooling processes in the disk .
borgani ( 2001,2002 ) used gasoline simulations of galaxy clusters in the standard @xmath88cdm cosmology to demonstrate that extreme levels of heating in excess of current estimates from supernovae associated with the current stellar content of the universe are required to make cluster x - ray properties match observations .
the simulations did not use gas cooling , and this is a next step .    used gasoline with star formation algorithms to produce a realistic disk galaxy in the standard @xmath88cdm cosmology .
current work aims to improve the resolution and the treatment of the star formation processes .
mayer ( 2001a , b ) simulated the tidal disruption of dwarf galaxies around typical large spiral galaxy hosts ( analogues of the milky way ) , showing the expected morphology of the gas and stellar components .
used a 270 million particle cosmological gasoline simulation to provide detailed estimates of the sunyaev - zeldovich effect on the cosmic microwave background at small angular scales .
this simulated 400 mpc cube contains thousands of bright x - ray clusters and provides a statistical sample that is currently being matched to observational catalogues @xcite .
a common theme in these simulations is large dynamic ranges in space and time .
these represent the cutting edge for gasdynamical simulations of self - gravitating astrophysical systems , and were enabled by the combination of the key features of gasoline described above .
the first feature is simply one of software engineering : a modular framework had been constructed in order to ensure that pkdgrav could both scale well , but also be portable to many parallel architectures . using this framework , it was relatively straightforward to include the necessary physics for gasdynamics .
of course , the fast , adaptable , parallel gravity solver itself is also a necessary ingredient .
thirdly , the gasdynamical implementation is state - of - the - art and tested on a number of standard problems .
lastly , significant effort has gone into making the code adaptable to a large range in timescales .
increasing the efficiency with which such problems can be handled will continue be a area of development with contributions from computer science and applied mathematics .
the authors would like to thank carlos frenk for providing the cluster comparison results and matthias steinmetz for providing high resolution results for the spherical collapse .
development of pkdgrav and gasoline was supported by grants from the nasa hpcc / ess program .
bond , j. , contaldi , c. , pen , u - l . , pogosyan , d. , prunet , s. , ruetalo , m. , wadsley , j. , zhang , p. , mason , b. , myers , s. , pearson , t. , readhead , a. , sievers , j. , udomprasert , p. 2002 , apj ( submitted ) , astro - ph/0205386                          frenk , c. , white , s. , bryan , g. , norman , m. , cen , r. , ostriker , g. , couchman , h. , evrard , g. , gnedin , n. , jenkins , a. , pearce , f. , thomas , p. , navarro , h. , owen , j. , villumsen , j. , pen , u - l . , steinmetz , m. , warren , j. , zurek , w. , yepes , g. & klypin , a. , 1999 , ap .
j. , 525 , 554                                                                      anderson , r. & tjaden , b. 2001 , `` the inverse nearest neighbor problem with astrophysical applications '' , proceedings of the twelfth annual acm - siam symposium on discrete algorithms ( soda ) , january 2001 .