the process of star formation in molecular gas is often described as multi - physics .
a myriad of physical effects are thought to be important at regulating the process , through establishing the structure of molecular gas and the subsequent distribution of stellar masses that result .
fluid dynamics shaped by gravitation , magnetic fields , chemistry , and radiation lies at the heart of the process and each of these effects is thought to contribute significantly to the outcome .
while analytic approaches guide our thinking , numerical simulations have been essential for modeling the nonlinear interaction of these effects .
as numerical simulations become more sophisticated , they provide a better emulation of the processes at work in molecular clouds . by finding a good match between the observations and simulations ,
we recognize the underlying physical parameters governing the simulation as a possible ( though not unique ) set of conditions within molecular clouds .    given this improvement in numerical methods , many groups have begun to compare simulations of star forming molecular clouds to observations of the same ( e.g. , * ? ? ?
* ; * ? ? ?
* ; * ? ? ?
* ; * ? ? ?
* ; * ? ? ?
these numerical studies all produced _ synthetic observations _ , which can be compared to observations of actual molecular clouds .
the synthetic observation approach avoids the complexity of inferring the real physical properties of molecular gas from the incomplete information present in observations .
these comparisons usually leverage an underlying statistic to gauge the degree of similarity .
the most common route is to identify a population of objects within the simulation ( e.g. , dense molecular cores ) and compare their properties to those of the known population .
the studies claim a successful emulation when the distributions of certain properties ( mass , angular momentum , size , etc . )
mimic those of the observed population .
population comparisons work well when there is a well - defined population of observed objects to which the simulations can be compared .    when comparing continuous structure such as that of the gas in a molecular cloud , different sets of statistics that rely on the continuous nature of the data can be employed . in general
, these statistics can be broadly divided into those that rely on the theoretical understanding of turbulence @xcite and those inspired by observations @xcite .
in addition to those statistics that are constructed around the continuous nature of the star forming clouds , catalog - based approaches are commonly adopted .
the canonical approach is that of clump finding , which entails dividing or extracting a catalog of substructures from the molecular cloud ( called clumps ) , and comparing the resulting population of objects @xcite .    with the improving quality of both simulations and observations ,
there is a growing need for a general framework by which the comparisons between observations and simulations can be rigorously assessed .
many statistical tools are developed _
ad hoc _ , and it is unclear what each individual tool is telling us . a given tool may not be sensitive to different physical parameters .
for example , fractal statistics may not be significantly altered by changes in the magnetic field .
similarly , simulation efforts may claim to reproduce the structure of molecular clouds , but the assessment of that similarity is frequently qualitative . in this paper
, we present a framework by which the sensitivity of different statistics to physical parameters can be measured .
we further use this framework to evaluate the quality of these statistics .
we focus this work on the structure of star - forming molecular clouds
. observations of molecular cloud structure use both spectral lines ( typically co ) and the dust continuum . of these , we focus on the co line mapping of clouds , which produces position - position - velocity ( ppv ) data cubes containing information about both the distribution and dynamics of molecular clouds .
such ppv data sets display a wealth of structure , representing a challenging modeling problem for numerical simulations . in this paper
, we will outline a basic analysis framework and provide several candidate statistical measures for comparing data sets (  [ section - comparison ] ) .
we will then present a suite of numerical simulations that exhaustively probe a simple parameter space (  [ section - simulations ] ) . using this set of simulations
, we describe how _ linear models _ provide a means to analyze the significance of the statistical measures (  [ section - linearmodel ] ) . using the results of the analysis
, we critically evaluate the different proposed statistics in  [ section : results ] .
this work emphasizes the comparison of a family of simulations to each other .
future work will use this approach to compare simulations to observations and the optimization of simulation parameters to match observations .
the comparison between simulations and observations is best carried out in the observational domain . our observational window onto reality
is restricted to information that can be inferred from the observed emission of these molecular clouds . inferring the full suite of physical cloud properties ( density , temperature , velocities ) from the emergent radiation is complicated by radiative transfer effects ( particularly in optically thick emission ) , chemistry , spatially and temporally varying heating and cooling as well as imperfections in observations .
compensating for these effects is difficult and always leaves caveats attached to the conclusions . by creating synthetic observations from the known physics in simulations , we avoid having to invert these non - linear effects . instead , we rely on the same caveats affecting both our emulated and actual views of star formation .
thus , a rigorous comparison of simulations and observations necessarily requires the comparison of synthetic observations with actual observations ( e.g. , * ? ? ?
even though the present work does not directly incorporate observational data , we operate on synthetic observations and deploy statistical tools that can also operate on observational data .    given two ppv data sets , @xmath0 and @xmath1 , we propose a formalism based around a pseudo - distance metric that measures the differences between them . like other distance metrics ,
we require @xmath2 and , ideally , this pseudo - distance measures the difference in the underlying physics producing the data sets , such as temperature , magnetic field , and mach number of the fluid flow .
in contrast , it should not be affected by other features such as coordinate shifts .
indeed , a good similarity statistic will be unaffected by any representation effects such as the pixelization ( though it should depend on the resolution ) .
for example , @xmath3 if @xmath1 is just @xmath0 shifted along any of its axes , e.g. , @xmath4 . any statistic that ignores the spatial relationships between pixels
will trivially fulfill this criterion .
the distance metric should be sensitive to difference in physical scale for a scale factor @xmath5 : @xmath6 .    for observational data ,
the statistic should be independent of the noise levels of the data set .
we denote the normal distribution with mean @xmath7 and variance @xmath8 as @xmath9 .
for example , @xmath10
if @xmath11 and @xmath12 for the same @xmath13 but different @xmath14 and @xmath15 . practically , this means any statistic involving simulation data should not rely on any data that are at low intensity levels with respect to the maximum .
such data would be impossible to measure in real observational data sets .
the distance metric should be near zero for different realizations of the same physical processes .
in our notation , @xmath10 if @xmath0 and @xmath1 are synthetic observations generated with the same physical conditions but different random seeds .
likewise , observations of regions in a similar evolutionary state should also show minimal difference under these statistics .
finally , there is significant physical information in the spatial ordering of the pixels .
self - gravitation , for example , leads to neighboring pixels begin correlated in their density and consequently their emission .
thus , @xmath16 if @xmath17 is the same data as @xmath0 with the positions randomized ( either in position or in position plus velocity ) .
given these considerations , we propose several possible similarity statistics as follows .
not all statistics fulfill all of the above criteria , and the different features they incorporate lead to different responses under the changing physical conditions (  [ section : results ] ) .
@xcite first proposed an application of principal component analysis ( pca ) to spectral line data cubes of molecular clouds .
subsequently , pca has been shown to be sensitive to the structure function of the underlying turbulent flow @xcite .
we adapt part of the pca formalism to parameterize the physical differences between datasets . given two data sets , @xmath18 and @xmath19 , we construct a covariance matrix , comparing the variance between different velocity channels in the data cube : @xmath20 and a parallel construction for @xmath1 yielding @xmath21 . following the pca approach , we compute eigenvalues @xmath22 and eigenvectors @xmath23 following @xmath24 . to construct a distance measure , we sort the vectors eigenvalues in order of absolute value ( @xmath25 , @xmath26 ) and propose a distance metric : @xmath27^{1/2}. \label{equation - dpca}\ ] ] here , the sums are carried out over the elements of the vectors @xmath28 .
the length of these vectors is equal to the number of velocity channels in the data . in practice ,
nearly all of the power is found in the first @xmath29 eigenvalues , and thus we truncate the sums in equation [ equation - dpca ] after 50 terms .
this facilitates comparing the results to cubes with different extent in velocity .
the spectral correlation function ( scf ) was first proposed by @xcite , and subsequently demonstrated to be a useful discriminant between the properties of turbulence in comparing simulations and observations @xcite .
the scf has a variety of different forms ; here we express the scf in terms of a normalized root - mean - square difference between spectra separated by an offset vector @xmath30 : @xmath31 here , @xmath32 is a vector representing the two spatial coordinates within a datacube .
note that @xmath33 and generally decreases to zero as @xmath34 grows large .
computing the scf for a pair of data cubes will yield two different surfaces .
we calculate a distance metric as @xmath35 ^ 2}.\ ] ] again , we truncate the summation over an area where the scf is significant .
we chose a 23-pixel squared patch for calculating @xmath36 and subsequently @xmath37 .
motivated by the need for multivariate two - sample testing , the cramer test statistic was first proposed by @xcite . given two dimensional data sets @xmath38 and @xmath39 with sizes @xmath40 and @xmath41 respectively , the test statistic is calculated by viewing each row as a point in a @xmath42-dimensional space .
the test statistic is computed by comparing the typical euclidean inter - point distances between the data @xmath38 and the data in @xmath39 to the distances between the data within each of @xmath38 and @xmath39 respectively : @xmath43 the above norm is the euclidean norm between two points in @xmath42 dimensions : @xmath44 .
one advantage of the cramer statistic is that its significance can be determined by bootstrapping .
the bootstrap draws two mock data sets of the same sizes as @xmath38 and @xmath39 from the combined set @xmath45 and evaluates the statistic over a large number of repetitions .
additionally , the statistic has been shown to be more sensitive than the multivariate kolmogorov - smirnov test .    in the context of the current work
, we must recast ppv data cubes into sets of @xmath42-dimensional points suitable for the cramer test .
we experimented with several methods for representing a ppv data cube as one or more @xmath46-dimensional points .
given a data cube @xmath47 , one approach is to consider each spectrum with @xmath42 velocity channels as a datum and consider the sets @xmath38 and @xmath39 as spectra drawn from the two different data cubes .
this comparison requires that the data sets being compared have matched spectral resolution and eliminates spatial information in the data .
alternatively , we construct a vector from each velocity channel in the data cube by sorting the data into a vector of intensity values for each channel .
we found that only retaining the top quartile of the brightness data maintained the sensitivity of the statistic while improving performance .
we use the latter approach in this initial study and evaluate the application of the cramer test to ppv data cubes in other work ( flynn et al . , in prep . ) .
the cramer statistic is sensitive to the absolute values of the data set . in order to eliminate this sensitivity
, we normalize the transformed arrays by their respective spectral norms .
the spectral norm of an array @xmath38 is constructed by first taking the eigenvalues , @xmath28 of the square array @xmath48 .
the spectral norm is then @xmath49 .
we note that clearly the cramer statistic is not sensitive to pixel ordering in our implementation .
the statistic performs well in the task of classifying similar and dissimilar data cubes , and experimentally satisfies the requirements of a well behaved distance metric .
we ran 16 simulations of self - gravitating , magnetized turbulence using enzo version 1.9 @xcite .
the simulations were designed to emulate a star forming cloud undergoing gravitational collapse .
we used a base grid with dimensions 128@xmath50 over a box size of @xmath51 pc with periodic boundary conditions .
we set the initial density field of the box to be uniform @xmath52 and the initial direction of the magnetic field to be parallel to one face of the cube . for each of the simulations , we initially ran the simulations without self - gravity to allow the driving mechanism to develop turbulence .
the random driving pattern for the turbulence is solenoidal with power on wavenumbers from @xmath53 to @xmath54 .
depending on the run , we set the shape of the initial velocity spectrum to have power on a wide range of scale ( from @xmath55 to @xmath56 ) or only large scale velocity structure ( @xmath57 to @xmath55 ) .
we scale the time of the simulation to the crossing time @xmath58 where @xmath59 is the mach number and @xmath60 is the sound speed . at @xmath61 , we turned on self - gravity , adaptive mesh refinement and enabled the formation of sink particles .
we used a jeans length refinement criterion , refining the grid by a factor of two whenever the local jeans length became shorter than four cells @xcite .
we limited the number of refinement levels to four .
we systematically varied the input mach number ( @xmath59 , which sets the amplitude of the forcing field ) , initial magnetic field strength ( @xmath62 ) , initial velocity spectrum shape ( @xmath63 ) , and temperature ( @xmath64 ) as described in table [ table : codes ] .
the table also reports the `` text codes '' we give to describe the varying parameters in the results .
we analyze the simulations at intervals of 0.1 crossing times from @xmath65 to @xmath66 .
ccccc low & 6 & 1 & @xmath67 $ ] & 10 + high & 18 & 100 & @xmath68 $ ] & 30 +   + & m & b & k & t [ table : codes ]    we ran one simulation for each of the @xmath69 possible combinations of the parameter settings .
we post - processed the data using the radmc-3d code @xcite to produce ppv data cubes of @xmath70 emission . in radmc-3d ,
the level populations are calculated using the large velocity gradient ( lvg ) approximation @xcite .
the @xmath71co collisional and emission coefficients are taken from the leiden atomic and molecular database ( lamda ; * ? ? ?
the simulated observations are generated as if the simulation domain were located a distance of 260 pc ( the distance of the perseus molecular cloud , chosen for future comparison with observational data from the complete survey ; * ? ? ?
we also selected @xmath71co as a tracer since it highlights the main structure of a molecular cloud while not being as limited by opacity effects in the @xmath72co data . in the radiative transfer ,
we include `` microturbulence '' with @xmath73 .
microturbulence represents turbulent motions on scales too small to be resolved by the hydrodynamic grid and contributes to line broadening .
the velocity resolution of the data cubes is @xmath74 .
we produced ppv data sets for each of the three orthogonal orientations of the cube to capture the effects of observations along and parallel to the initial magnetic field .
regions where the adaptive mesh undergoes refinement are averaged over so that the ppv data have the same spatial size as the base grid , namely 10 pc resolved into 128 resolution elements .
our final data suite consists of ( 16 simulations)@xmath75(8 time steps)@xmath75(3 orientations)@xmath76 synthetic ppv data cubes .
figure [ fig : compcube ] shows the @xmath77 slice through the data cubes for four different parameter settings at @xmath66 .
additional data from the simulations are presented in appendix [ appendix - simprops ] .
the work leverages the framework from the statistical subfield of experimental design .
the goal of this analysis is to determine how changes in physical conditions manifest in the synthetic data .
we refer to these changes as _ effects _ and aim to estimate their significance . to make these estimates , we use similarity metrics to compare the values at the fiducial ( or low ) parameter values to those at the high values ( table [ table : codes ] ) .
we define a scalar response when comparing the synthetic data from one simulation to another .
since there are multiple timesteps available for each set of physical parameters , these need to be aggregated into a single statistic
typifying how distinct simulations with different parameters are .
we define a scalar response value for the @xmath78th simulation compared to a fiducial simulation as @xmath79 .
\label{eq : response}\ ] ] this is the mean of a given comparison statistic over the last @xmath80 outputs of the simulation ( i.e. , from @xmath65 to 3.0 ) . each simulation is compared to the fiducial simulation @xmath0 , which has @xmath81 , @xmath82 , @xmath83  k and @xmath84 $ ] .
the averaging occurs when self - gravity is active , and some simulations are evolving significantly .
the use of an average over time aggregates these effects , but the time evolution can be significant ( see  [ subsection - evolution ] ) .      typically , parameter studies of star formation use `` one factor at a time '' ( ofat ) variation of parameters ( e.g. , * ? ? ?
* ; * ? ? ?
* ; * ? ? ?
* ; * ? ? ?
* ; * ? ? ?
one parameter ( e.g. , the magnetic field strength ) is varied while all the other factors in the simulation are held constant .
the response of the simulation to the variation of this single parameter is then interpreted as the effect of changing this parameter .
this approach is reasonable when simultaneous parameter changes will not result in a different degree response relative to individual changes , i.e. , when there are no interactions between the parameters .
analyzing a suite of ofat parameter studies may be misleading when parameter interactions exist .
we establish quantitative measures of parameter changes by studying the resulting runs as a standard linear model .
we consider the case where we have @xmath85 different numerical simulations and @xmath46 different combinations of parameter settings .
note that repeated experiments with the same parameter settings but different random seeds will increase @xmath85 but not @xmath46 . in matrix form
we have @xmath86 where @xmath87 is an @xmath88 model matrix , @xmath89 is the @xmath90 vector of responses ( i.e. , equation [ eq : response ] ) , @xmath91 is the @xmath92 vector of independent normal errors and @xmath93 is the @xmath94 vector of unknown parameters ( including the intercept ) .
these fit parameters characterize the response of the simulations to changing the physical conditions in the simulation . in this framework ,
the ofat design has a model matrix @xmath95 where the first column corresponds to a constant intercept , while the latter four columns code the factor setting for the four factors used in this study ( see table [ table : codes ] ) . in ofat
, we represent the low / fiducial settings for parameters with a _ code _ of 0 and the high setting of parameters with a code of 1 .
we utilize codes instead of the actual parameter values for simplicity in developing the design matrix ( equation [ eq : base_ofat ] ) and interpreting the statistical properties thereof .
there is substantial latitude in choice of coding since the values of @xmath96 . under certain coding schemes , the values of these effect estimates
can be interpreted in terms of quantitative effects @xcite , but here we are only interested in the significance of the values .
the standard least squares estimator , @xmath97 , of the fit parameters @xmath96 is given by : @xmath98    the covariance matrix for @xmath99 is given by @xmath100 where @xmath101 is the unknown error variance characterizing @xmath91 . determining @xmath8
usually requires running multiple experiments with the same physical parameter settings but different random seeds .
the repeated experiments indicate what fraction of the measured variation is the result of statistical fluctuations , enabling these effects to be distinguished from changes that can be attributed to changing simulation parameters .
the covariance matrix for the parameters @xmath91 is given by @xmath102 indicating that the parameter estimates in the ofat approach are correlated with each other meaning that changes in one will affect changes in another parameter .
in the case of an experimental setup and measuring of parameter significance it is desirable to have independent ( i.e. , uncorrelated ) parameter estimates .
additionally , the above model does not allow for interactions between the parameters to be studied through that set of simulations .
we run a _ full factorial _
design where all possible changes of the parameters are considered . using only two values for each of the four factors
, this setup results in @xmath103 runs . in order to aid the resulting analysis
, we adopt a different coding for the levels , which utilizes the linear independence between the columns of orthogonal arrays to produce independent estimates of the importance of different physical effects .
we note again that the choice of coding is arbitrary as we only seek to measure the significance of the effect estimates .
the fiducial / low will be coded as @xmath104 ( previously 0 ) and a @xmath105 is used for the high level .
the first five columns of the model matrix are given as @xmath106 where the first column represents the intercept and the remaining 4 columns show the level setting for the parameters .
for example the third row shows that enzo would be run with @xmath81 , @xmath82 , @xmath107 $ ] and @xmath108  k. also notice that rows 15 , 16 , 14 , 10 and 7 respectively are the 5 runs in the ofat design described in equation [ eq : base_ofat ] .    the above design allows for the estimation of all of the two , three and four factor interactions between the resulting parameter settings .
these terms are included in the model matrix by adding an additional 10 columns to @xmath87 that are constructed by taking the product of all possible subsets of the columns 2 through 5 in equation [ eq : ortho_full ] .
when using this parameterization , @xmath109 where @xmath110 is the identity matrix , so we describe this coding as an _
orthogonal design_. in this case , the covariance matrix ( corresponding to equation [ eq : cov_ofat ] ) is diagonal , and the estimated effects @xmath97 are statistically independent . under this framework
each parameter estimate is given by @xmath111 where @xmath112 and @xmath113 are the means of the responses at the high and low values of the parameter respectively ( as defined in table [ table : codes ] ) .
the goal is to evaluate the statistical significance of these effects . without a measure of the run - to - run variation in the response statistics
, any derived effect could just be the result of random fluctuations .
ideally , there would be several replicates of each experimental run which would allow for the estimation of @xmath8 in the model .
replicated simulations are runs with the same physical parameters , but different random seeds for the turbulent driving . with an estimate of @xmath8 , the statistical significance of each of the regression coefficients ( @xmath114 )
can be assessed to determine what parameters are causing the real changes in the underlying ppv data cubes . usually , replicated numerical experiments entail a considerable investment of computational resources and using those resources to produce a different experimental run can provide substantially more information .
in the case of the two - level full factorial design , parameter significance can still be determined even without replicated runs .
@xcite introduced a formal approach to judge parameter significance in unreplicated experiments .
lenth s method assumes that most effects and interactions are insignificant so they can be used to estimate the `` noise '' in the effect estimates .
this assumption is checked after analysis to verify that only a small number of significant effects are indeed significant .
the method can not be relied upon if it finds nearly all or none the effects are significant .
a pseudo - standard error is calculated using a robust estimator @xmath115 given by @xmath116 where @xmath117 the approach proceeds by constructing statistic similar to a student s t : @xmath118 and comparing @xmath119 to a critical value set by the experimental design . in the case of a 16 run experiment with 15 factors ,
any @xmath120 is judged as significant , where the critical value is determined using the so - called individual error rate and changes depending in the number of runs @xcite .
using the simulations described above , we calculate response vectors @xmath89 for each of the three similarity metrics described in  [ section - comparison ] . in this section ,
we examine how the different similarity metrics respond to changes in the physical parameters defined in the initial conditions of the simulation .
we begin by examining what can be determined from the full suite of 16 simulations , representing all parameters and then we examine how the results change for an ofat study . finally , we propose using fractional factorial designs , where a carefully chosen subset of the 16 simulations is used in the analysis .      using the full factorial design , we can estimate the effects @xmath121 and their statistical significance . the effects measure
how responsive a statistic is to changing physical parameters in the simulations .
the influence of the four principal parameters ( known as _
main effects _ in the design literature ) and their interactions is probed with a full factorial design .
for example , the analysis measured how the scf statistic responds to changes in input mach number and temperature as well as the interaction term between these effects ( coded as m : t ) that measures the change in the response statistic in excess of the change seen from the main effects alone .
these additional interaction terms probe cases where the physical parameters amplify the effects of each other .
figure [ fig : full_lenth ] shows the analysis of the full factorial design across all three similarity metrics .
the coefficients @xmath122 are shown as the lengths of the bars .
the differences in the range of values shown for each @xmath114 arises because the individual statistics have different numerical ranges in their response .
only their value relative to the significance threshold of @xmath123 for each statistic is important .
the degree by which @xmath122 exceeds the threshold can be interpreted as a degree of significance .
for example , the scaling of the three statistics with changing the input mach number is more significant than any other effect for the three response statistics .
the analysis of the three comparison statistics consistently highlights two significant main effects : input mach number and temperature .
notably , there is no significant effect due to changing magnetic field alone .
since the input mach number of the turbulent driving in the simulation determines the density contrasts in shocks , this result is not surprising .
note that the input mach number is most closely tied to the amplitude of the turbulent forcing field and does not mean that actual mach number of the flows in the simulation is this value ( see  [ subsection - driving ] ) .
the temperature changes manifest in slightly larger line widths due to thermal pressure , but these effects should be insignificant because the sonic mach numbers are large .
the temperature increase also controls the amplitude of the spectral lines after radiative transfer ( see appendix a ) . while the amplitude of the spectral lines is compensated for in the three statistics under comparison , the compensation has varying degrees of success .
the full - factorial analysis indicates an interaction effect between the input mach number and the temperature which is significant for pca and cramer statistics but not for the scf .
the interaction effect is completely eliminated because of the nature of the scf normalization which compensates for absolute values of the brightness in the cube comparison on a comparison - by - comparison basis .
the cramer statistic also normalizes the data but appears to be more sensitive to variations due to the brightness of the data sets .
given the strong effects of the magnetic field visually apparent in figure [ fig : compcube ] , it is surprising that the magnetic field does not manifest as an active parameter in the response statistics .
this indicates a shortcoming of the statistics as formulated in the analysis : they are not sensitive to magnetic effects .
even though the principal analysis is formulated for a line of sight parallel to the initial magnetic field , we performed a parallel analysis for the two faces perpendicular to the magnetic field and found no significant magnetic effects .
this lack of sensitivity to the strong effect on the emission structure in the highly magnetized case can be explained because the spectral line profiles for the @xmath124 are not significantly different from the fiducial case ( see figure [ fig - samplespec ] ) .
again , the analysis highlights the relative merits of the comparison statistics and indicates different statistics will need to be developed in order to measure the sensitivity to magnetic field . despite this lack of sensitivity to the main effect ,
both the pca and scf indicate a negative interaction effect between the magnetic field and the input mach number .
this interaction indicates that joint increases in both the input mach number and magnetic field produce simulations that these similarity measures find are more similar to the fiducial value than if these parameters are increased independently .
we attribute these significant interaction effects to a difference between the input mach number , which is the parameter in the simulation , and the actual mach number which establishes the density structure in the simulation ( see  [ subsection - driving ] ) .    despite the comparatively low resolution of the simulation suite ,
the analysis of the full factorial design indicates the initial success at adopting the formalism .
the results show that the framework can identify some of the significant physical effects ( input mach number and temperature ) that govern the structure of ppv data cubes .
however , the nondetection of effects that are clearly significant ( magnetic field ) shows that the construction of the similarity statistics needs to be adapted to detect magnetism .
alternatively , other similarity measures which emphasize magnetic effects could be recast into this framework ( e.g. , * ? ? ?
* ; * ? ? ?
we emphasize that the results identify the significance of both the effects and their interactions , representing the primary utility in employing the fully developed statistical framework to this analysis .
.effect estimates of the various statistics based on the one factor at a time experiment ( ofat ) .
[ cols=">,>,>,>",options="header " , ]     where the full factorial analysis identifies significant main effects as well as their interaction terms , ofat analysis can only detect main effects .
furthermore , ofat analysis without multiple realizations of the physical effects yields no estimate of the variance that arises because of the random fluctuations between simulations with the same physical conditions . the ofat analysis can be used to compare the magnitude of the main effects as given in table [ tb : ofat ] for the three different similarity statistics . as with the full factorial design
, input mach number remains the dominant effect .
temperature effects and magnetic effects also seem to be detectable and the initial velocity spectrum is least important affect . the intercept value is estimated as 0.0 for all statistics because of the definition of the response , namely @xmath125 when the fiducial simulation is compared to itself .
we note that , in practice , the fiducial data will likely be observational data sets to which the simulations are being compared . for analyzing a suite of simulations without observational data , a simulation at the center of the parameter space would make the best choice for a fiducial simulation .
the effect estimate for the magnetic field seems like it could be detected by the similarity statistics ; however , the full - factorial design indicates that this apparent detection arises because of the interaction effects seen between magnetic field strength and input mach number .
a purely ofat analysis will incorrectly conclude that the influence of the magnetic field strength has been detected .
furthermore , ofat provides no assessment of the variance in the similarity statistics and the importance of each of these effect estimates can not be determined .
this analysis highlights the shortcomings of the ofat approach .
specifically , the analysis can be misled in the presence of parameter interactions and is inefficient at testing the significance of different simulation parameters @xcite .
the full factorial design provides a more nuanced interpretation of the responses as a function of changing initial conditions .
however , running a full factorial design can require substantial computational resources .
we ran the two - level , full factorial design , because we were using 128@xmath50 resolution simulations .
in addition , we chose a model where there are only four factors and each factor only has two values .
there are situations where one would like to run simulations with higher resolution , more factors , and a continuous range of values . to reduce the computational cost
, it is possible to run a subset of the full factorial design with the ability to estimate both the main effects and some of the interactions . in many situations
, prior knowledge can be used to identify a limited subset of the interactions that are likely to be important .
starting from the full factorial design of 16 runs , we select the 8 runs that correspond to having all factors ( m , b , k , t ) at the high level .
this results in the following 8 runs , which are commonly referred to as a @xmath126 fractional factorial design .
@xmath127 again the columns 2 through 5 are used to assign the level setting for each of the four factors .
since we have reduced the run size by a factor of two we lose the ability to estimate 8 of the 15 effects that were estimated in
[ section - fullfac ] . of the 7 remaining degrees of freedom
, one can attribute the significance of the estimated effect to one of two terms .
for example , if the effect of column 2 is significant then either the main effect of input mach number ( m ) or the b : k : t interaction is important .
we assume third and higher order terms are negligible , so we can estimate each of the four main effects .
the remaining three columns can be used to estimate the following pairs of effects , m : b = k : t , m : t = k : b and b : t = k : m .
the notation m : b = k : t is an indication that , in fractional factorial design , estimates of the two - factor interaction m : b are aliased with the k : t interaction .
if significant , we would not be able to discriminate between these two effects .
this aliasing is a cost of running a factional factorial design .
however , we can use our prior knowledge to decide whether a given significant effect should be interpreted as , for example , a m : t interaction or a k : b interaction .
we believe that the effects from the initial velocity spectrum are negligible because those effects are washed out by the turbulent driving .
our assumption is well justified by the insignificance of velocity spectrum effects in the full factorial response analysis . in larger simulations ,
it is possible the velocity spectrum and magnetic field could show dependence .
thus , for this set of simulations , we believe that the significant interactions would be attributed to m : b , m : t and b : t as opposed to the effects involving the initial velocity spectrum ( k ) . ignoring the aliasing of the parameters in the above design , the 8
runs simply correspond to a full factorial in three factors .
hence , one can use the same unreplicated analysis proposed by @xcite to determine which parameters are significant . in this case
the appropriate critical value is @xmath128 .
figure [ fig : frac_lenth ] shows the results from the fractional factorial analysis .
[ fig : frac_lenth ]    the significant parameters from both the full factorial and fractional factorial are consistent in relative magnitude and sign .
the effect significance , particularly for the pca is reduced under the fractional factorial design .
some information is lost when the fractional factorial is used over the full factorial , but the fractional factorial does provide some information about the interactions .
in addition , it only requires an additional 3 runs in comparison to the ofat design , while establishing an understanding of the parameter interactions that compares well to running the full suite simulations at twice the computational cost .
the results from this design illustrate how a carefully chosen design can maximize information from a parameter studies that are subject to limited computational resources .
the savings of a fractional factorial design over a full factorial grows as the number of parameters explored or the number of level settings increases .
the dominant physical effect we see in our simulations is the response of our statistics to the input mach number parameter set .
this input mach number does not actually set the true mach number of the fluid flow in the simulation .
rather , it determines the amplitude of the driving field used to establish the turbulent velocity field ( * ? ? ?
* outlines an approach similar to that used in enzo ) .
the default enzo distribution uses a heuristic formulation to specify the amplitude of the turbulent driving force per unit mass : @xmath129 , where @xmath130 is a _ driving efficiency _ , @xmath60 is the sound speed , @xmath131 is the initial rms velocity measured in the simulation and @xmath132 is the mean free - fall time in the domain . the parameter @xmath59 is the input mach number for the simulation but
does not necessarily mean that the mach number of the simulation will have that value .
the driving efficiency was taken as @xmath133 for all simulations , but this value only produces real mach numbers close to the initial mach number parameter for weak @xmath134-field cases .
the initial mach number should thus be thought as a characteristic of the strength of the driving field rather than the true mach number characterizing the simulation .
when compared to the input mach number , we found that the actual mach numbers were between 37% ( @xmath124 , @xmath135 $ ] ) and 95% ( fiducial ) of the input mach number . of note , the input mach number parameter also establishes the crossing time for the simulation .
the actual crossing time will be different from the crossing time reported in the simulation .
we also constructed a linear model of the actual mach number in the simulation as a function of the changing input parameters . as expected , the input mach number was the dominant effect in establishing the real mach number , but the initial magnetic field strength has a strong negative effect ( increasing @xmath62 decreased actual mach number relative to the input mach number ) .
the damping effect of the magnetic field is expected on physical grounds and serves to explain why the magnetic field appears significant in ofat analysis , but the effect is not present in the full - factorial design . instead
, interaction effects between the magnetic field and temperature or input mach number appear significant .
the interaction effects in the actual simulation response can be attributed partly to the magnetic field driving a discrepancy between input and actual mach number .
additionally , a modest mach - temperature interaction emerged from the model which may explain the same effect seen in the response of the comparison metrics .      our primary analysis averages over the last eight time steps of the simulation ( from @xmath65 to 3.0 ) .
since self - gravity is turned on at time @xmath136 , simulations that are undergoing gravitational collapse will see these effects manifest over the time period contributing to the average .
real effects may be averaged over or diminished by using the aggregate of all the response values over the self - gravitating phase of the simulation . to address this concern , we plot the main effect estimates as a function of time in figure [ fig : timevar ] .
this figure compares each simulation at a fixed time step ( though note that @xmath137 will vary by simulation based on the input mach number as above ) .
figure [ fig : timevar ] shows that there is not a significant change in the main effect estimates over the period when the simulation is self - gravitating .
there is some variation in the sensitivity of the scf to the magnetic field at the last timesteps of the simulation , but this effect is modest .
the significance of the effects for mach and temperature changes appears significant over the whole period .
in this work , we have proposed a formalism by which ppv data cubes of spectral line data can be compared in order to assess physical effects .
the formalism relies on the methods utilized in the statistical subfield of experimental design .
we have proposed three similarity metrics based on existing techniques in the literature and assess their reliability using the formalism applied to a suite of simulations .
the three similarity metrics are principal component analysis , the spectral correlation function and a new method using the cramer multivariate two - sample test .
all three similarity statistics showed consistent results in our analysis , though the relative sensitivities of these measures varied .
in particular , the statistics are most sensitive to mach number and temperature effects .
we have also examined the benefits and liabilities for using the `` one factor at a time '' ( ofat ) analysis that is common in the astrophysical literature . while the results of ofat analysis are relatively easy to interpret , the analysis is unable to assess whether the physical effects explored in the simulations interact with each other .
this can lead to overinterpreting the importance of certain parameters in the simulation .
for example , we found that our measurement statistics appeared sensitive to magnetic field strength based on the ofat analysis .
however , in exploring the parameter space fully , the statistics were found to be unable to detect changes in the magnetic field strength , instead finding an interaction effect between magnetic field strength and the input mach number .
we attribute this interaction to the reduction of the real mach number relative to the input for increasing magnetic field strength .
a full exploration of parameter space is invariably computationally prohibitive for high quality simulations .
thus , we advocate selecting a fractional factorial design that is able to recover those effects most likely to be significant through the study .
a major shortcoming of this study is the exaggerated ranges used for the parameters ( e.g. , 100 @xmath7 g fields on 10 pc scales are not observed ) .
even so , this work demonstrates proof - of - concept for three avenues of study .
first , the comparison of simulations with observations can be explored under this framework .
frequently , simulations are compared to observations and the conditions for a best match are assessed under qualitative terms . by establishing observations as the fiducial to which the simulation suite is compared , we can establish those effects which most closely govern similarity to simulations . while this would formalize the comparison between simulations and observations
, it will require amending the similarity statistics to account for the effects of telescope observations ( variable data set sizes , limited resolution , receiver noise ) .
the second set of work that this study enables is the ability to assess the reliability of comparison metrics .
there are a wealth of such metrics present in the literature , but they are only validated under a limited set of conditions . using this test suite of simulations or a higher quality suite that studies a full range of parameter combinations , we can assess how reliable the simulations are at detecting certain physical effects .
the framework also reveals when a statistic can be conflating the effects of two parameters .
for example , the cramer statistic is primarily sensitive to mach and temperature effects and is surprisingly immune to interaction effects between these parameters and the magnetic field .
the pca and scf metrics detect significant statistical interactions between these parameters and the magnetic field strength .
however , none of these statistics detect the magnetic field as a main effect indicating limitations in their current formulation .
finally , this study offers guidance for explorations of parameter space in suites of simulations .
given the prevalence of ofat analysis in astronomy and the complexity of interpreting the simultaneous variation of multiple physical effects , using a statistical response model to assess parameter effects offers some clear guidance .
extensions to this framework allow for multiple values or continuous exploration of parameter spaces , non - linear response models and efficient exploration of high - dimensional parameter spaces for simulation .
this project was supported by discovery grants from nserc of canada and a collaborative research grant from the i.k .
barber school of arts and sciences at ubc .
ssro is supported by a nasa through a hubble fellowship grant # 51311.01 awarded by the space telescope science institute , which is operated by the association of universities for research in astronomy , inc .
, for nasa , under contract nas 5 - 26555
. we are grateful for simulation assistance from tyler berg .
we acknowledge productive discussions with doug johnstone and alyssa goodman in formulating this research .
westgrid and compute canada resources were used in executing the simulations .
this research made heavy use freely - available , community - developed software , including enzo ( http://enzo-project.org ) , yt ( http://yt-project.org/ ; * ? ? ? * ) and astropy ( http://astropy.org ; * ? ? ? * ) and we are grateful for the developer efforts on those projects .
this research has made use of nasa s astrophysics data system .
we acknowledge the constructive comments from the anonymous referee , which improved the quality and presentation of the paper .    42 natexlab#1#1    attwood , r.  e. , goodwin , s.  p. , stamatellos , d. , & whitworth , a.  p. 2009 , a&a , 495 , 201    baringhaus , l. & franz , c. 2004 , journal of multivariate analysis    box , g. , hunter , j.  s. , & hunter , w.  g. 2005 , statistics for experimenters : design , innovation , and discovery , 2nd edition ( wiley )    brunt , c.  m. & heyer , m.  h. 2002 , apj , 566 , 276    . 2002 , apj , 566 , 289    burkhart , b. , falceta - gonalves , d. , kowal , g. , & lazarian , a. 2009 , apj , 693 , 250    burkhart , b. , lazarian , a. , goodman , a. , & rosolowsky , e. 2013 , apj , 770 , 141    dib , s. , hennebelle , p. , pineda , j.  e. , csengeri , t. , bontemps , s. , audit , e. , & goodman , a.  a. 2010 , apj , 723 , 425    dickman , r.  l. & kleiner , s.  c. 1985 , apj , 295 , 479    dullemond , c.  p. 2012 , astrophysics source code library , 02015    falgarone , e. , lis , d.  c. , phillips , t.  g. , pouquet , a. , porter , d.  h. , & woodward , p.  r. 1994 , apj , 436 , 728    falgarone , e. , phillips , t.  g. , & walker , c.  k. 1991 , apj , 378 , 186    goodman , a.  a. 2011 , computational star formation , 270 , 511    heyer , m. , gong , h. , ostriker , e. , & brunt , c. 2008 , apj , 680 , 420    heyer , m.  h. & schloerb , f.  p. 1997 , apj , 475 , 173    houlahan , p. & scalo , j. 1990 , apjs , 72 , 133    kleiner , s.  c. & dickman , r.  l. 1985 , apj , 295 , 466    kramer , c. , stutzki , j. , rohrig , r. , & corneliussen , u. 1998 , a&a , 329 , 249    lazarian , a. & pogosyan , d. 2000 , apj , 537 , 720    . 2004 , apj , 616 , 943    lenth , r.  v. 1989 , technometrics , 31 , 469    mac  low , m. & klessen , r.  s. 2004 , reviews of modern physics , 76 , 125    mac  low , m .- m . 1999 ,
apj , 524 , 169    miesch , m.  s. & bally , j. 1994 , apj , 429 , 645    offner , s. s.  r. , bisbas , t.  g. , viti , s. , & bell , t.  a. 2013 , apj , 770 , 49    offner , s. s.  r. , krumholz , m.  r. , klein , r.  i. , & mckee , c.  f. 2008 , aj , 136 , 404    oshea , b.  w. , bryan , g. , bordner , j. , norman , m.  l. , abel , t. , harkness , r. , & kritsuk , a. 2004 , arxiv , 3044    padoan , p. , goodman , a.  a. , & juvela , m. 2003 , apj , 588 , 881    padoan , p. , juvela , m. , kritsuk , a. , & norman , m.  l. 2006 , apjl , 653 , l125    ridge , n.  a. , di  francesco , j. , kirk , h. , li , d. , goodman , a.  a. , alves , j.  f. , arce , h.  g. , borkin , m.  a. , caselli , p. , foster , j.  b. , heyer , m.  h. , johnstone , d. , kosslyn , d.  a. , lombardi , m. , pineda , j.  e. , schnee , s.  l. , & tafalla , m. 2006 , aj , 131 , 2921    rosolowsky , e.  w. , goodman , a.  a. , wilner , d.  j. , & williams , j.  p. 1999 , apj , 524 , 887    scalo , j.  m. 1984 , apj , 277 , 556    schier , f.  l. , van  der tak , f.  f.  s. , van dishoeck , e.  f. , & black , j.  h. 2005 , a&a , 432 , 369    shetty , r. , glover , s.  c. , dullemond , c.  p. , & klessen , r.  s. 2011 , mnras , 412 , 1686    stutzki , j. , bensch , f. , heithausen , a. , ossenkopf , v. , & zielinsky , m. 1998 , a&a , 336 , 697    stutzki , j. & gsten , r. 1990 , apj , 356 , 513    .
2013 , arxiv , 6212    truelove , j.  k. , klein , r.  i. , mckee , c.  f. , holliman , j. h.  i. , howell , l.  h. , & greenough , j.  a. 1997 , astrophysical journal letters v.489 , 489 , l179    turk , m.  j. , smith , b.  d. , oishi , j.  s. , skory , s. , skillman , s.  w. , abel , t. , & norman , m.  l. 2011 , apjs , 192 , 9    vzquez - semadeni , e. , gonzlez , r.  f. , ballesteros - paredes , j. , gazol , a. , & kim , j. 2008 , mnras , 390 , 769    williams , j.  p. , de  geus , e.  j. , & blitz , l. 1994 , apj , 428 , 693    wu , c. f.  j. & hamada , m.  s. 2011 , experiments , planning , analysis , and optimization ( wiley )
in figure [ fig - samplespec ] we present two summaries of the simulations in the simulation suite . rather than plotting curves for all 16 of the simulations ,
we only illustrate the fiducial case with all parameters at the low setting , the curves for only one factor changing from its low to high setting and finally the curves for all parameters set to their high value .
the two summaries present density pdfs and spectra for these simulations . of note ,
the density pdfs show that only the fiducial case and the large scale initial spectrum ( @xmath138 $ ] ) show significant gravitational collapse and a tail of values off to high densities @xcite . in the other simulations , including those not shown in the figure , the local jeans refinement condition is never met and there is no local gravitational collapse .
the initial driving scale likely plays little role at governing gravitational collapse since the fixed driving pattern dominates the structure of the turbulence at the time of gravitational collapse .
the average spectra of the simulated observations also show roughly expected behavior .
the line width is larger for higher mach numbers and temperature .
the amplitude of the line is also larger for the higher temperatures as expected from radiative transfer effects .
spectra along individual lines of sight show significant non - gaussian structure as is frequently seen in simulated data .