complex systems are usually described as consisting of mutually interacting objects , often exhibiting complex global behavior resulting from the interactions between these objects .
this behavior is typically characterized as `` emergent '' or `` self - organizing '' as the system s constituting parts do not usually obey a central controller @xcite .
analytic treatment generally does not yield the complete theory of a complex system .
therefore , modeling and simulation techniques play a major role in our understanding of how these systems work @xcite .
methodologies such as agent - based modeling ( abm ) , system dynamics , discrete event simulation , among others , are frequently employed for this purpose @xcite . of these
, abm provides an instinctive approach for describing many complex systems , as agents are regularly a suitable match to the individual and heterogeneous objects composing these systems .
the local interactions of these objects , as well as their individual and adaptive behavior , are often critical for understanding global system response @xcite .
abms are commonly implemented as a stochastic process , and thus require multiple runs ( observations ) with distinct pseudo - random number generator ( prng ) seeds in order to have appropriate sample sizes for testing hypotheses and differentiating multiple scenarios under distinct parameterizations @xcite .
computational models of complex systems in general , and abms in particular , are usually very sensitive to implementation details , and the influence that seemingly negligible aspects such as data structures , discrete time representation and sequences of events can have on simulation results is notable @xcite .
furthermore , most model implementations are considerably elaborate , making them prone to programming errors @xcite .
this can seriously affect model validation when data from the system being modeled can not be obtained easily , cheaply or at all .
model verification can also be compromised , to the point that wrong conclusions may be drawn from simulation results .
a possible answer to this problem is the independent replication of such models @xcite .
replication consists in the reimplementation of an existing model and the replication of its results @xcite .
replicating a model in a new context will sidestep the biases associated with the language or toolkit used to develop the original model , bringing to light dissimilarities between the conceptual and implemented models , as well as inconsistencies in the conceptual model specification @xcite .
additionally , replication promotes model verification , model validation @xcite , and model credibility @xcite .
more specifically , model verification is promoted because if two or more distinct implementations of a conceptual model yield statistically equivalent results , it is more likely that the implemented models correctly describe the conceptual model @xcite . thus , it is reasonable to assume that a computational model is untrustworthy until it has been successfully replicated @xcite .
model parallelization is a an illustrative example of the importance of replication .
parallelization is often required for simulating large models in practical time frames , as in the case of abms reflecting systems with large number of individual entities @xcite . by definition ,
model parallelization implies a number of changes , or even full reimplementation , of the original model .
extra care should be taken in order to make sure a parallelized model faithfully reproduces the behavior of the original serial model .
there are inclusively reports of failure in converting a serial model into a parallel one @xcite .
although replication is considered the scientific gold standard against which scientific claims are evaluated @xcite , most conceptual models have only been implemented by the original developer , and thus , have never been replicated @xcite .
several reasons for this problem have been identified , namely : a ) lack of incentive @xcite ; b ) below par communication of original models @xcite ; c ) insufficient knowledge and uncertainty of how to validate results of a reimplemented model @xcite ; and , d ) the inherent difficulty in reimplementing a model @xcite .
this work targets c ) , with positive influence on d ) .
replication is evaluated by comparing the output of the reimplementation against the output of the original model @xcite , and this process , as will be discussed , is empirically driven and model - dependent ( or even parameter - dependent ) .
furthermore , it is sometimes unclear as to what output features best describe model behavior . a robust and ready to use output comparison method
would thus reduce or eliminate uncertainty of how to validate reimplementation results ( reason c ) , eliminating this obstacle in the overall process of model replication ( reason d ) .
we present a model comparison technique , which uses principal component analysis ( pca ) to convert simulation output into a set of linearly uncorrelated statistical measures , analyzable in a consistent , model - independent fashion .
it is appropriate for ascertaining statistical equivalence of a model replication with its original implementation . besides model - independence , this technique has three additional desirable features : a ) it automatically selects output features that best explain implementation differences ; b ) it does not depend on the distributional properties of simulation output ; and , c ) it simplifies the modelers work , as it can be used directly on model output , avoiding manual selection of specific points or summary statistics .
we evaluate this technique against classic model comparison methods using the pphpc abm @xcite as a test case .
this model is thoroughly studied in terms of simulation output for a number of parameter configurations , providing a solid base for this discussion .
the paper is organized as follows .
first , in section [ sec : background ] , we review commonly used methods for comparing the output of simulation models , as well as previous work on model replication using these methods .
section [ sec : simmod ] introduces pphpc , the simulation model used as a test case for the proposed model - independent comparison methodology , which is described in section [ sec : micomp ] .
next , section [ sec : expsetup ] delineates the experimental setup for assessing this methodology . in section [ sec : results ] , results of the classic and model - independent comparison approaches are presented .
a discussion and an evaluation of these results is undertaken in section [ sec : discussion ] .
section [ sec : conclusions ] summarizes what was accomplished in this paper .
the typical process of model comparison is described by wilensky and rand @xcite , and can be roughly divided into three steps : 1 ) choice of replication standard ( rs ) ; 2 ) selection of output statistical summaries , i.e. focal measures ( fms ) , to compare ; and , 3 ) comparison of fms using statistical techniques .
[ [ choice - of - replication - standard ] ] choice of replication standard + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +    axtell et al .
@xcite defined three rss for the level of similarity between fms ( carley @xcite calls the rs the emphasis of demonstration ) : _ numerical identity _ , _ distributional equivalence _ and _ relational alignment_. the first , _ numerical identity _ , implies exact numerical output , but it is difficult to demonstrate for stochastic models and not critical for showing that two such models have the same dynamic behavior . to achieve this goal , _ distributional equivalence _ is a more appropriate choice , as it aims to reveal the statistical similarity between two outputs . finally , _
relational alignment _ between two outputs exists if they show qualitatively similar dependencies with input data , which is frequently the only way to compare a model with another which is inaccessible ( e.g. , implementation has not been made available by the original author ) , or with a non - controllable `` real '' system ( such as a model of the human immune system @xcite ) . for the remainder of this text
we assume the _ distributional equivalence _
rs when discussing model replication .
[ [ selection - of - focal - measures ] ] selection of focal measures + + + + + + + + + + + + + + + + + + + + + + + + + + +    after selecting the rs criteria , a set of statistical summaries representative of each output must also be defined .
it is these statistical summaries , and not the complete model outputs , that will be compared in order to assert the similarity between the original model and the replication . as models may produce large amounts of data , the statistical summaries should be chosen as to be relevant to the actual modeling objective .
consequently , the selection of output statistical summaries , i.e. fms , is an empirical exercise and is always dependent of the model under study @xcite .
[ [ comparison - of - fms - using - statistical - techniques ] ] comparison of fms using statistical techniques + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +    there are three major statistical approaches used to compare fms : 1 ) statistical hypothesis tests ; 2 ) confidence intervals ; and , 3 ) graphical methods @xcite .    in statistical hypothesis testing , a null hypothesis is tested against an alternative hypothesis .
the test can have two results : 1 ) fail to reject the null hypothesis ; or , 2 ) reject the null hypothesis in favor of the alternative hypothesis .
in this case , the tests of interest are two - sample ( or multi - sample ) hypothesis tests which test for the null hypothesis that the observations in each sample are drawn from the same distribution , against the alternative that they are not .
these observations are the statistical summaries obtained from the outputs of two ( or more ) model implementations , or from the outputs of a model and the system being modeled .
although statistical procedures for comparing model and system outputs using hypothesis tests have been proposed @xcite , confidence intervals are usually preferred for such comparisons , as they provide an indication of the magnitude by which the statistic of interest differs from model to system .
confidence intervals are also commonly used when evaluating different models that might represent competing system designs or alternative operating policies @xcite .
graphical methods , such as q - q plots , can also be employed for comparing output data , though their interpretation is more subjective than the previous methods . in this paper
we will focus on hypothesis tests , which are more common for comparing a model implementation and its replication @xcite .
the choice of which statistical test(s ) to use depends on a number of factors , namely : 1 ) whether a specific fm follows a normal distribution ; 2 ) the number of observations ( i.e. , number of runs ) in each sample ( i.e. , model implementation ) ; 3 ) the dimensionality of the fms ( i.e. , unidimensional or multidimensional ) ; and , 4 ) the number of samples ( i.e. , model implementations ) to be compared . for example , in @xcite , the authors discuss the use of the hotelling s @xmath0 test , for assessing the validity of a multivariate response simulation model representing a real , observable system with multiple outputs .
the study is focused towards relating the cost of sampling against the probability of type i and ii errors according to an acceptable similarity range between model and system output averages .
this analysis was later extended to confidence intervals and joint confidence regions @xcite .      in reference @xcite ,
axtell et .
all compared two initially different models , with one iteratively modified in order to be aligned with the other .
the authors evaluated how can distinct equivalence standards be statistically assessed using non - parametric statistical tests ( namely the kolmogorov - smirnov @xcite and mann - whitney u @xcite tests ) , and how minor variations in model design affect simulation outcomes .
they concluded that comparing models developed by different researchers and with different tools ( i.e. , programming languages and/or modeling environments ) , can lead to exposing bugs , misinterpretations in model specification , and implicit assumptions in toolkit implementations .
the concepts and methods of `` computational model alignment '' ( or `` docking '' ) were first discussed in this work .
edmonds and hales @xcite performed two independent replications of a previously published model involving co - operation between self - interested agents .
several shortcomings were found in the original model , leading the authors to conclude that unreplicated simulation models and their results can not be trusted .
this work is one of the main references in model replication , describing in detail the process of running two model implementations with different parameters , selecting comparison measures and performing adequate statistical tests .    in reference
@xcite , the authors presented an abm replication case study , describing the difficulties that emerged from performing the replication and determining if the replication was successful .
a standard @xmath1-test was used for comparing model outputs .
the authors concluded that model replication influences model verification and validation and promotes shared comprehension concerning modeling decisions .
miodownik et al .
@xcite replicated an abm of social interaction @xcite , originally implemented in matlab , using the ps - i environment for abm simulations @xcite .
a statistical comparison of the mean levels of `` civicness '' at the end of the simulation ( over 10 runs ) was performed using the mann - whitney u test .
results showed that , while distributional equivalence was obtained in some cases , the two models were mostly only relationally aligned .
the authors attribute this mainly to the fact that some aspects of the original model were not implementable with ps - i .
a method for replicating insufficiently described abms was discussed in reference @xcite , consisting in modeling ambiguous assumptions as binary parameters and systematically applying statistical tests to all combinations for their equivalence to the original model .
the approach was used to replicate epstein s demographic prisoner s dilemma model @xcite , with only partial success , suggesting the existence of some undefined assumptions concerning the original model .
the authors also conducted a number of statistical tests regarding the influence of specific design choices , highlighting the importance that these be explicitly documented .
alberts et al .
@xcite implemented a cuda @xcite version of toy infection model @xcite , and compared it with the original version implemented in netlogo @xcite , as well as to an in - house optimized serial version .
statistical validation was performed visually using q - q plots .
multiple serial and parallel variants of the pphpc model were compared in reference @xcite .
simultaneous comparison of the several variants was accomplished by applying the multi - group non - parametric kruskal - wallis test @xcite to predetermined fms of the several model outputs .
results showed that all model variants could not be considered misaligned for a number of different parameters .
this methodology has a number of disadvantages .
first , it relies on fms which are model - dependent and , probably , user - dependent .
in order to apply it to a different model , the process of selecting fms and the appropriate statistical tests must be redone .
furthermore , for different model parameters , the originally selected fms may be of no use , as simulation output may change substantially ( e.g. , the warm - up period for the steady - state statistical summaries can be quite different ) .
also , it might not be clear which fms best capture the behavior of a model .
the predator - prey for high - performance computing ( pphpc ) model is a reference model for studying and evaluating spatial abm ( sabm ) implementation strategies , capturing important characteristics of sabms , such as agent movement and local agent interactions .
it is used in this work as a test case for the proposed model comparison method . the model is thoroughly described in reference @xcite using the odd protocol @xcite . here we present a summarized description of the model .
pphpc is a predator - prey model composed of three entity classes : _ agents _ , _ grid cells _ and _ environment_. _ agents _ can be of type prey or predator . while prey consume passive cell - bound food , predators consume prey . agents
have an energy state variable , @xmath2 , which increases when feeding and decreases when moving and reproducing .
when energy reaches zero , the agent is removed from the simulation .
instances of the _ grid cell _ entity class are where agents act , namely where they try to feed and reproduce .
grid cells have a fixed grid position and contain only one resource , cell - bound food ( _ grass _ ) , which can be consumed by prey , and is represented by the countdown state variable @xmath3 .
the @xmath3 state variable specifies the number of iterations left for the cell - bound food to become available .
food becomes available when @xmath4 , and when a prey consumes it , @xmath3 is set to @xmath5 ( given as an initial simulation parameter ) .
the set of all grid cells forms the _ environment _ entity , a toroidal square grid where the simulation takes place .
the environment is defined by its size and by the restart parameter , @xmath5 .
the temporal scale is represented by a positive integer @xmath6 , which represents the number of iterations .
a simulation starts with an initialization process , where a predetermined number of agents are randomly placed in the simulation environment .
cell - bound food is also initialized at this stage .
after initialization , and to get the simulation state at iteration zero , outputs are collected .
the scheduler then enters the main simulation loop , where each iteration is sub - divided into four steps : 1 ) agent movement ; 2 ) food growth in grid cells ; 3 ) agent actions ; and , 4 ) gathering of simulation outputs .
note that the following processes are explicitly random : a ) initialization of specific state variables ( e.g. , initial agent energy ) ; b ) agent movement ; c ) the order in which agents act ; and , d ) agent reproduction .
for process c ) , this implies that the agent list should be explicitly shuffled before agents can act .
six outputs are collected at each iteration @xmath7 : @xmath8 , @xmath9 , @xmath10 , @xmath11 , @xmath12 , and @xmath13 .
@xmath14 and @xmath15 refer to the total prey ( _ sheep _ ) and predator ( _ wolf _ ) population counts , respectively , while @xmath16 holds the quantity of available cell - bound food .
@xmath17 and @xmath18 contain the mean energy of prey and predator populations . finally , @xmath13 refers to the mean value of the @xmath3 state variable in all grid cells .
reference parameters for the pphpc model are specified in reference @xcite .
parameters are qualitatively separated into size - related and dynamics - related groups .
although size - related parameters also influence model dynamics , this separation is useful for parameterizing simulations .    concerning size - related parameters , a base grid size of @xmath19
is associated with @xmath20 prey and @xmath21 predators .
different grid sizes should have proportionally assigned agent population sizes , such that the initial agent density and the initial ratio between prey and predators remains constant .
we define _ model size _ as the association between grid size and initial agent population . for example
, model size 200 corresponds to a grid size of @xmath22 with prey and predators at iteration 0 .    for the dynamics - related parameters , two parameter sets , 1 and 2 ,
are proposed .
the two parameter sets generate distinct dynamics , with the second set typically yielding more than twice the number of agents than the first
. we will refer to a combination of model size and parameter set as `` size@xmath23set '' , e.g. @xmath24 for model size 400 , parameter set 1 .
the reference number of iterations , @xmath6 , is , not counting with the initial simulation state at iteration 0 .      in order to statistically compare output behaviors of different pphpc implementations
, we should first select a set of fms , i.e. , of statistical summaries representative of individual outputs @xcite .
we can then compare these fms for distributional equivalence instead of directly comparing outputs .
typically , statistical summaries consist of long - term or steady - state means .
however , being limited to analyze average system behavior can lead to incorrect conclusions @xcite .
consequently , we extract and compare a broader range of statistical summaries from individual outputs , namely : 1 ) maximum value ( @xmath25 ) ; 2 ) iteration where maximum value occurs ( @xmath26 ) ; 3 ) minimum value ( @xmath27 ) ; 4 ) iteration where minimum value occurs ( @xmath28 ) ; 5 ) steady - state mean ( @xmath29 ) ; and , 6 ) steady - state sample standard deviation ( @xmath30 ) .
thus , we compare a total of 36 fms ( six statistical summaries from six outputs ) . in order to discard the initial transient period and avoid initialization bias , steady - state measures , @xmath29 and @xmath30 ,
are taken for @xmath31 and @xmath32 for parameter sets 1 and 2 , respectively @xcite .    assuming that individual model implementations are executed @xmath33 times
, there will be @xmath33 observations for each associated output .
as such , each of the 36 fms will be represented by a sample of @xmath33 observations per implementation .
individual fms are tested for the null hypothesis that the points in each sample are drawn from the same distribution , against the alternative that they are not .
as discussed in @xcite , all measures , except @xmath26 and @xmath28 , are amenable to be compared using parametric methods , as they follow ( or approximately follow ) normal distributions .
thus , for these we use the two - sample @xmath1-test , while for @xmath26 and @xmath28 we use the non - parametric mann - whitney u test .    if two implementations produce the same dynamic behavior it is reasonable to expect that any statistical summary collected over a number of independent runs also comes from the same distribution @xcite .
thus , conclusions on the alignment of two configurations can be drawn from the @xmath34-values produced by the statistical tests , more specifically by observing if all or most of the 36 @xmath34-values ( one per fm ) are above the typical 1% or 5% significance levels .
as this effectively constitutes a multiple hypothesis testing scenario , @xmath34-values can be adjusted with the bonferroni procedure or similar method @xcite .
two implementations of the pphpc model are used for evaluating the model comparison technique proposed in the next section .
the first is developed in netlogo @xcite , while the second is a java implementation with several parallel variants @xcite .
for the results discussed in this paper , simulations performed with the java implementation were executed with the ex parallelization strategy using eight threads . in this strategy
, each thread processes an equal part of the simulation environment , and reproducible simulations are guaranteed .
individual threads use their own sub - sequence of a global random sequence , obtained using a random spacing approach @xcite . the mersenne twister pseudo - random number generator ( prng )
@xcite is used by both implementations for driving the model s random processes .
complete details of both implementations are available in the provided references , and their source code is available at https://github.com / fakenmc / pphpc/.
a model - independent comparison or alignment method should work directly from simulation output , automatically selecting the features that best explain potential differences between the implementations being compared , thus avoiding the disadvantages of the classical approach .
additionally , such a method should not depend on the distributional properties of simulation output , and should be directly applicable by modelers .
our proposal consists of automatically extracting the most relevant information from simulation output using pca @xcite .
pca is a widely used technique @xcite which extracts the largest variance components from data by applying singular value decomposition to a mean - centered data matrix . in other words
, pca is able to convert simulation output into a set of linearly uncorrelated measures which can be analyzed in a consistent , model - independent fashion .
this technique is especially relevant for abms , as it considers not only equilibrium but also dynamics over time @xcite .
procedure  [ proc : micomp ] summarizes this process for a time - series type output , although the general idea is applicable for other types of multivariate output @xcite .    1 .
perform @xmath33 replications for each model implementation , and collect the respective outputs .
2 .   for each model output @xmath35 ( e.g. , predator population , @xmath36 ) : 1 .
group the @xmath35 s from all replications row - wise in matrix @xmath37 , i.e. the @xmath38 output from the @xmath39 replication will be the @xmath39 row of matrix @xmath37 . if we are comparing @xmath40 implementations and performing @xmath33 replications of @xmath6 time steps for each , the size of this matrix will be @xmath41 .
2 .   determine matrix @xmath42 , which is the column mean - centered version of @xmath37 .
3 .   apply pca to matrix @xmath42 , considering that rows ( replications ) correspond to observations and columns ( iterations or time steps ) to variables .
this yields : 1 ) matrix @xmath43 , containing the representation of the original data in the principal components ( pcs ) space ; and , 2 ) vector @xmath44 , containing the eigenvalues of the covariance matrix of @xmath42 in descending order , each eigenvalue corresponding to the variance of the columns of @xmath43 .
the columns of @xmath43 correspond to pcs , and are orderer by decreasing variance , i.e. the first column corresponds to the first pc , and so on .
rows of @xmath43 correspond to observations .
the @xmath45 column of @xmath43 contains @xmath46 modelindependent observations for the @xmath45 pc , @xmath33 for each implementation .
groups of @xmath33 observations associated with model implementations can be compared using statistical methods .
more specifically , hypothesis tests can be used to check if the output projections on the pc space , grouped by model implementation , are drawn from populations with the same distribution .
there are two possible lines of action :    1 .   a multivariate approach , consisting of applying a manova test @xcite on the @xmath46 output projections , organized in @xmath40 groups ( one per model implementation ) of @xmath33 @xmath47-dimensional observations each , along the first @xmath47 pcs ( dimensions ) such that these explain a userdefined minimum amount of variance .
2 .   a univariate approach , consisting of applying a hypothesis test to individual pcs on the @xmath46 output projections , organized in @xmath40 groups ( one per model implementation ) of @xmath33 observations each .
possible tests include the @xmath1-test and the mann - whitney u test for comparing two groups ( @xmath48 ) , or anova @xcite and kruskal - wallis test , which are the respective parametric and non - parametric versions for comparing more than two groups ( @xmath49 ) .
the use of the manova multivariate test has the advantage of yielding a single @xmath34-value from the simultaneous comparison of output projections along multiple dimensions .
an equally succinct answer can be obtained with the univariate approach using the bonferroni correction or a similar method for handling @xmath34-values from multiple comparisons @xcite . however
, both approaches will not prioritize dimensions , even though the first pcs are more important for characterizing model differences , as they explain more variance . in the univariate case
one can subjectively attribute higher relevance to the first pcs , or objectively prioritize dimensions according to the explained variance using the weighted bonferroni procedure @xcite .
parametric tests such as manova and @xmath1-test make several assumptions about the data being compared , namely that @xcite :    * all samples are mutually independent . * each sample is drawn from a normally distributed population ( multivariate normality for manova ) .
* samples are drawn from populations with equal variances ( for manova , the variance - covariance matrix should be the same for each population ) .
the first assumption is guaranteed by the way the samples are obtained , as described in section [ sec : expsetup ] .
the second and third assumptions can be checked using appropriate statistical tests .
more specifically , group sample normality can be assessed using the shapiro - wilk test @xcite ( royston test @xcite in the multivariate approach ) , while equality of variances among groups can be verified with the bartlett test @xcite ( box s m test @xcite for homogeneity of variance - covariance matrices in the multivariate approach ) .
however , box s m test is very sensitive and can lead to false negatives ( type ii errors ) .
fortunately , manova is considerably robust to violations in the homogeneity of variance - covariance matrices when groups have equal sample sizes @xcite .
if these assumptions are not met in the univariate approach , non - parametric tests ( e.g. , mann - whitney u test or the kolmogorov - smirnov test ) can be used instead .
non - parametric alternatives , e.g @xcite , exist for the multivariate approach , but they are not as well established as the aforementioned methods .
the eigenvalues vector @xmath44 is also important for this process , for two reasons : 1 ) to determine a minimum number of pcs for the manova test , such that these explain a prespecified percentage of variance ; 2 ) alignment or otherwise of the @xmath40 model implementations can be empirically assessed by analyzing how the explained variance is distributed along pcs .
the percentage of variance explained by each pc can be obtained as shown in eq .
[ eq : pcvar ] .    @xmath50    where @xmath7 identifies the @xmath51 pc , @xmath52 is the eigenvalue associated with the @xmath51 pc , and @xmath53 is the sum of all eigenvalues .
if the variance is well distributed along many pcs , it is an indication that the compared implementations are aligned , at least for the output being analyzed . on the other hand ,
if most of the variance is explained by the first pcs , it is a clear sign that at least one model implementation is misaligned .
the rationale being that if all implementations show the same dynamical behavior , then the projection of their outputs in the pc space will be close together and have similar statistics , i.e. means , medians and variance .
as such , pca will be unable to find components which explain large quantities of variance , and the variance will be well distributed along the pcs .
if at least one model implementation is misaligned , the projection of its outputs in the pc space will be farther apart than the projections of the remaining implementations . as such
, pca will yield at least one component which explains large part of the overall variance .
the alignment of two or more implementations can be assessed by analyzing the following information : 1 ) the @xmath34-values produced by the univariate and multivariate statistical tests , which should be above the typical 1% or 5% significance levels in case of implementation alignment ; in the univariate case , it may be useful to adjust the @xmath34-values using the weighted bonferroni procedure to account for multiple comparisons ; 2 ) in case of misalignment , the total number of pcs required to explain a prespecified amount of variance should be lower than in case of alignment ; also , more variance should be explained by the first pcs of the former than by the same pcs of the latter ; and , 3 ) the scatter plot of the first two pc dimensions , which can offer visual , although subjective feedback on model alignment ; e.g. , in case of misalignment , points associated with runs from different implementations should form distinct groups .
determining the alignment of models with multiple outputs may be less straightforward .
if model implementations are clearly aligned or misaligned , conclusions can be drawn by analyzing the information provided by the proposed method for each individual output .
otherwise , an objective approach may be required .
we suggest two such approaches : a ) a bonferroni or similar multiple comparison @xmath34-value correction ; or , b ) the concatenation of all outputs , centered and scaled , followed by a model - independent examination of the concatenated output .
the first method can be appropriate for the multivariate case , in which we have one @xmath34-value per output .
it can be extended to the univariate case if we only consider the uncorrected @xmath34-value from the first pc .
if we consider multiple pcs or their already corrected @xmath34-values ( weighted bonferroni ) , the analysis becomes more complex and it can become unclear how to interpret the results . additionally , multiple comparison correction methods often assume independency of test statistics , which might not be possible to assure when testing different outputs of the same simulation model , which are most likely correlated .    in the second approach , we concatenate all outputs and examine the resulting concatenated output .
this reduces a model with @xmath54 outputs to a model with one output . in order to perform output concatenation , we center and scale the outputs such that their domains are in the same order of magnitude
this can be performed using range scaling on each output @xmath35 , for example , as shown in the following eqs .
:    @xmath55    @xmath56 \label{eq : scaledoutput}\ ] ]    where @xmath57 is the value of output @xmath35 at iteration @xmath7 , @xmath58 is the mean of output @xmath35 , and @xmath59 is the range scaled version of @xmath35 .
other centering and scaling methods , such as auto - scaling or level scaling @xcite , can be used as an alternative to range scaling in eq .
[ eq : rangescale ] . for a model with @xmath54 outputs , @xmath60
, the resulting concatenated output is given by    @xmath61    where @xmath62 is the concatenation operator , and @xmath63 is the concatenation of all model outputs .
model implementations can thus be compared with the proposed method using the `` single '' model output @xmath63 .
in order to test the model comparison methods , we define a base pphpc configuration using the netlogo implementation and compare it with three configurations using the java implementation .
all configurations are tested for model sizes 100 , 200 , 400 and 800 , and parameter sets 1 and 2 , as described in section [ sec : simmod : params ] .
the four configurations follow the conceptual model , except when stated otherwise :    1 .
netlogo implementation .
2 .   java implementation .
3 .   java implementation : no agent list shuffling prior to agent actions
java implementation : the restart parameter , @xmath5 , is set to one unit less than specified ( 9 instead of 10 for parameter set 1 , 14 instead of 15 for parameter set 2 ) .
the goal is to assess how the two model comparison methods ( classic and model - independent ) expose the increasing differences of comparing configuration 1 with configurations 2 to 4 .
more specifically , we are interested in checking if the proposed model - independent comparison method is able to expose these differences in the same way as the classic approach . as such ,
we define three comparison cases :    case i : :    compare configuration 1 with configuration 2 .
these configurations    should yield distributionally equivalent results .
case ii : :    compare configuration 1 with configuration 3 .
a small misalignment is    to be expected .
case iii : :    compare configuration 1 with configuration 4 .
there should be a    mismatch in the outputs .
for each `` size@xmath23set '' combination , independent samples of the six model outputs are obtained from @xmath64 replications for each configuration , in a total of @xmath65 runs .
each replication @xmath66 is performed with a prng seed obtained by taking the md5 checksum of @xmath67 and converting the resulting hexadecimal string to an integer ( limited to 32-bits for netlogo and 128-bits for the java implementation ) , guaranteeing independence between seeds , and consequently , between replications .
the same samples were used for the evaluation of both classical and model - independent comparison approaches .    in order to evaluate how the tested methodologies respond to larger sample sizes
, we performed additional @xmath68 replications per configuration for the @xmath24 combination , in a total of @xmath69 runs .
the prng seeds for individual replications were obtained in the same fashion .    for the model - dependent comparisons ,
results were obtained with the simoututils software @xcite .
the micompr r package @xcite provides an implementation of the proposed model - independent comparison approach , and was used to produce the corresponding results .
the data generated by this computational experiment , as well as the scripts used to set up the experiment , are made available to other researchers at https://zenodo.org/record/46848 .
in this section we mainly focus on the results for the @xmath24 combination .
results for the remaining size / set combinations are provided as supplementary material , and are referred to when appropriate .
.[tab : mdcomps]@xmath70-values for the classic model comparison method for model size 400 , parameter set 1 , and @xmath64 runs per configuration .
@xmath70-values were obtained with the @xmath1-test for @xmath25 , @xmath27 , @xmath29 and @xmath30 statistics , and with the mann - whitney u test for @xmath26 and @xmath28 statistics .
case i weights two similar configurations , case ii compares configurations with a small implementation difference , and case iii compares configurations with a different parameter .
values lower than @xmath71 are underlined , while values lower than @xmath72 are double - underlined .
[ cols="^ , < , > , > , > , > , > , > " , ]     the shapiro - wilk test does not reject univariate normality in approximately 95% of samples used for the @xmath1-tests . of the remaining 5% ,
only 1% show significant rejection at the @xmath73 level .
this percentage is even lower for pc1 samples , the most critical for evaluating configuration alignment using the univariate approach .
samples also seem to generally have similar variance , as denoted by the 90% of non - significant results for the bartlett test .
multivariate normality is also verified by the royston test in most instances . since @xmath34-values are the probability of obtaining a result at least as extreme than what was actually observed ( assuming that the null hypothesis is true ) , type i error rates are close to what would be expected , namely 95% for @xmath74 , 4% for @xmath75 and 1% for @xmath76 .
as such , the assumptions tested by the shapiro - wilk , bartlett and royston tests seem to be generally verified .
the same does not appear to hold for the manova assumption of homogeneity of variance - covariance matrices , evaluated with box s m test .
however , as stated in section [ sec : micomp ] , this test is extremely sensitive and manova is resilient to violations in this assumption when samples have equal sizes , which is the case here .
larger sample sizes make hypothesis test more powerful , i.e. , more likely to reject the null hypothesis when it is false , thus decreasing the probability of committing a type ii error .
there are a number of methods to determine the adequate sample size ( number of runs in this case ) , required to obtain a specific statistical power @xcite .
however , we are essentially interested in determining , within the discussed frameworks , if and how conclusions concerning configuration alignment change with sample size . for this purpose ,
the classic and the proposed model - independent comparison methods were applied to samples of @xmath77 and @xmath68 replicates for the @xmath24 combination .
the comparison with samples of size 10 used the first 10 observations from the @xmath64 runs per configuration setup , while the comparison with samples of size 100 used a new set of @xmath68 runs per configuration .
results from this analysis are provided in supplementary tables s5.1 ( classic approach , 10 runs ) , s5.2 ( model - independent approach , 10 runs ) , s6.1 ( classic approach , 100 runs ) and s6.2 ( model - independent approach , 100 runs ) .
using samples of 10 observations , the classic model comparison method still manages to find configuration differences in cases ii and iii , although not as clearly as for @xmath64 . in case ii , the difference is found on the steady - state mean of the @xmath36 and @xmath78 outputs . with @xmath64 ,
differences are detected in several more fms .
the proposed model - independent comparison approach is also able to spot differences in cases ii and iii , likewise detecting differences in the @xmath36 and @xmath78 outputs for case ii .    for @xmath68
, both methods confirm what was observed for smaller sample sizes , although results are more pronounced , with very significant @xmath34-values for cases ii and iii .
two details emerge , however .
first , for case ii , the classic approach hardly recognizes configuration differences for the mean predator energy output , @xmath79 .
such differences are exposed by the proposed model - independent method , both by the @xmath1-test and manova .
second , the model - independent method perceives a difference where there should not be any , namely in output @xmath78 of case i , via the manova test . the pc1 @xmath1-test @xmath34-value is not significant , as is the case for the vast majority of remaining pcs .
for the 72 pcs ( dimensions ) considered by manova , the @xmath1-test @xmath34-value is significant at @xmath73 in three cases : pc11 , pc22 and pc48 .
after weighted bonferroni correction , only the pc11 @xmath34-value , associated with 2.4% of the variance , remains significant , with a value of @xmath80 . since these pcs , and pc11 in particular , account for a small amount of output variability , conclusions concerning output similarity ( and consequently , model alignment ) , do not necessarily change .
however , this result raises some concerns about the sensitivity of manova to type i errors ( i.e. , rejecting a true null hypothesis ) in this context .
overall , what seems clear for a sample of @xmath68 replicates , is that a @xmath1-test on the first pc of each individual output ( or even the concatenated output ) , is enough to pick up implementation differences .
the classic comparison approach and the proposed model - independent method were able to identify the changes introduced in configurations 3 and 4 , even though these were of different nature .
the change introduced in configuration 3 , case ii , confirms the sensitivity of this type of models : a simple and seemingly innocuous implementation discrepancy lead to statistically different behavior .
however , the change in configuration 4 , case iii , was expected to be noticed , since it implied the alteration of a model parameter . nonetheless , in specific conditions , namely for parameter set 2 , configuration 3 presented more dissimilar behavior than configuration 4 , when compared to configuration 1 .
since parameter set 2 generates more agents during simulations ( as discussed in section [ sec : simmod : params ] ) , the effect of not shuffling the agent list in configuration 3 , case ii , may be more pronounced .
additionally , the @xmath5 parameter is larger in parameter set 2 than in parameter set 1 .
as such , the smaller relative difference of decreasing @xmath5 by one unit in configuration 4 may lead to less significant discrepancies in case iii .    in the context of a smaller sample size , namely for @xmath77 runs per configuration , both methods were capable of discerning differences in cases ii and iii .
differentiation was better with increasing number of replicates , i.e. , when going from @xmath77 , then @xmath64 , and finally @xmath68 .
in practice , however , more replicates did not yield different conclusions regarding output and configuration alignment .
globally , results for the proposed model - independent method were in accordance with the classic approach on an output - per - output basis , with the advantage that the outputs are compared as a whole , instead of being summarized into a number of fms .
the proposed method was also able to expose a few implementation differences not detected by the classic approach , and , in other instances , it identified these differences with additional detail .
as such , the proposed method was shown to be a direct , and often more effective replacement for the classic model - dependent comparison approach . however , in spite of these advantages , interpretation of results should be done thoroughly . in other words , the different types of information produced by the method , detailed in section [ sec : micomp ]
, should be considered and evaluated before concluding on model alignment .
the @xmath34-values yielded by the @xmath1 and manova tests offer an objective assessment concerning the alignment of individual outputs .
while in most instances the two tests provided the same response , there were situations in which results differed . in case of alignment ,
the @xmath1-test on the first pc never produced @xmath34-values significant at @xmath73 , even prior to weighted bonferroni correction .
manova , however , signaled a few false positives for the @xmath78 output , the most worrying of which for the @xmath68 observations per sample case , which also affected the @xmath63 output .
a careful analysis revealed the cause to be a misalignment in one or more pcs , which in themselves did not account for much output variance , but were sufficient to provoke a type i error in the manova test . in case of configuration misalignment , manova was generally more sensitive than the pc1 @xmath1-test , and with the exception of @xmath81 , flagged a few misalignments unnoticed by the @xmath1-test .
overall , we recommend both tests are performed , and in case of disagreement , try to find the cause .
namely , 1 ) if the pc1 @xmath1-test @xmath34-value is significant ( and the manova @xmath34-value is not ) , adjust it with the weighted bonferroni correction , and consider that instead ; 2 ) if the manova @xmath34-value is significant ( and the pc1 @xmath1-test @xmath34-value is not ) , perform the @xmath1-test on other pcs to find which ones are likely to be influencing the manova result .
decide on alignment considering how much variance the culprit pcs explain . as an alternative to performing both tests
, it is also possible to perform the @xmath1-test on all pcs , and consider the resulting @xmath34-values after weighted bonferroni correction .
this also has the advantage of automatically privileging the most important pcs , i.e. , the ones that explain more variance .
furthermore , if samples do not appear to be drawn from a normal distribution , a non - parametric test can be used instead of the @xmath1-test ( e.g. , mann - whitney or kolmogorov - smirnov ) .
the eigenvalue structure , reflected in the number of pcs required to explain a prespecified percentage of variance , as well as in the variance explained by the first pc or pcs , is also an interesting metric . for aligned outputs , in contrast with misaligned ones , more pcs are required to explain the same amount of variance , and less variance is explained by the first pc or pcs .
the information provided by this metric in general , and by the number of pcs in particular , is more useful when assessing multiple cases , for which it can be relatively compared . nonetheless ,
concerning the variance explained by the first pc , some notes can be provided .
it seems to stay at about 5 - 20% for aligned outputs , and usually 25% or more , up to 95% , otherwise .
there are exceptions , namely for model size 100 , so unless this value is extremely high , e.g. above 40% , it can be difficult to draw conclusions without having an aligned baseline to compare with ( e.g. , case i ) . this aligned baseline can be obtained with different sets of runs ( using distinct prng seeds ) from the same configuration .
however , the specified variance is above all necessary for selecting the number of pcs for manova .
while conclusions concerning configuration alignment do not appear to change with different percentages of variance , it seems preferable to use more variance than less , otherwise one may not be able to use manova .
our selection of 90% seems appropriate .
another possibility is to use a fixed number of pcs for manova instead of specifying a percentage of variance to explain .
a potential problem with this approach is that manova could eventually consider very different percentages of variance from output to output and/or case to case .
the scatter plot of the first two pc dimensions is also useful for quickly assessing possible output misalignments .
if the outputs have a clear mismatch , points associated with runs from different implementations form visibly distinct groups .
although this technique is basically a type of face validation , it provides a useful intuition on the alignment of two or more implementations for any given output .
the concatenated output provides a single answer on whether two or more model implementations are aligned or not . in terms of false positives ,
i.e. , reporting a misalignment where none is expected , this was observed once by a manova test for the @xmath68 runs per configuration instance .
the @xmath1-test , however , did not flag any configuration difference .
false negatives , i.e. failure to detect output mismatch , occurred a few times for both manova and the @xmath1-test , and were essentially associated with smaller model sizes .
additionally , neither tests were able to detect differences in case ii for the @xmath77 runs per configuration instance .
overall , output concatenation does provide a practical and quick assessment of model alignment , but should not be fully relied upon to make a final decision .
additionally , the analysis of individual outputs allows to diagnose which model aspects are most affected by possible mismatches , something which the concatenated output does not permit .
altogether , no single metric of the proposed method provides a definitive answer , and decisions concerning model alignment should be taken based on the different types of information it produces .
a technique for comparing simulation models in a model - independent fashion was presented in this paper .
the method dispenses the need for selecting model - specific fms , and is appropriate for comparing multiple implementations of a conceptual model or replications of an existing model .
the technique was validated against a classic model comparison approach using the pphpc abm on a number of model sizes and parameter sets .
results showed that the proposed method offered similar conclusions concerning output and model alignment , while providing more detailed feedback in some situations .    although the proposed framework is presented in the context of simulation output comparison , it is a generic statistical comparison methodology .
as such , it can be employed in circumstances where observations yield a large number of variables @xcite .
this work was supported by the fundao para a cincia e a tecnologia ( fct ) projects uid / eea/50009/2013 , uid / mat/04561/2013 , and partially funded with grant sfrh / bd/48310/2008 , also from fct .
the author vitor v. lopes acknowledges the financial support from the prometeo project of senescyt ( ecuador ) .
s.  alberts , m.  k. keenan , r.  m. dsouza , and g.  an .
data - parallel techniques for simulating a mega - scale agent - based model of systemic inflammatory response syndrome on graphics processing units . , 88(8):895907 , 2011 .
r.  a. berg , h.  c. hoefsloot , j.  a. westerhuis , a.  k. smilde , and m.  j. werf .
centering , scaling , and transformations : improving the biological information content of metabolomics data .
, 7(1):142 , 2006 .
n.  fachada , m.  a.  t. figueiredo , v.  v. lopes , r.  c. martins , and a.  c. rosa .
spectrometric differentiation of yeast strains using minimum volume increase and minimum direction change clustering criteria .
, 45:5561 , aug .
2014 .
n.  fachada , v.  v. lopes , and a.  rosa .
simulating antigenic drift and shift in influenza a. in _ proc .
of the 2009 acm symposium on applied computing _ , sac 09 , pages 20932100 , new york , ny , usa , 2009 .
acm .
lee , t.  filatova , a.  ligmann - zielinska , b.  hassani - mahmooei , f.  stonedahl , i.  lorscheid , a.  voinov , j.  g. polhill , z.  sun , and d.  c. parker .
the complexities of agent - based modeling output analysis .
, 18(4):4 , 2015 .
b.  mller , s.  balbi , c.  m. buchmann , l.  de  sousa , g.  dressler , j.  groeneveld , c.  j. klassert , q.  b. le , j.  d. millington , h.  nolzen , d.  c. parker , j.  g. polhill , m.  schlter , j.  schulze , n.  schwarz , z.  sun , p.  taillandier , and h.  weise .
standardised and transparent model descriptions for agent - based models : current status and prospects . , 55:156163 ,
may 2014 .
h.  r. parry and m.  bithell .
large scale agent - based modelling : a review and guidelines for model scaling . in a.
j. heppenstall , a.  t. crooks , l.  m. see , and m.  batty , editors , _ agent - based models of geographical systems _ , pages 271308 .
springer netherlands , 2012 .
r.  g. sargent . a new statistical procedure for validation of simulation and stochastic models .
technical report syr - eecs-2010 - 06 , department of electrical engineering and computer science , syracuse university , nov . 2010 .
o.  will and r.  hegselmann . a replication that failed  on the computational model in ` michael w. macy and yoshimichi sato : trust , cooperation and market formation in the u.s . and japan .
proceedings of the national academy of sciences , may 2002 ' .
, 11(3):3 , 2008 .