a fundamental problem of statistical mechanics is the determination and understanding of classical ground states of many - particle systems - the zero - temperature particle arrangement that minimizes potential energy per particle .
although perfect crystalline ( periodic ) structures are often energetically favorable among all configurations , there is incomplete mathematical and intuitive understanding of the formation of order at low temperature , @xcite introducing the possibility of disordered ground states .
a disordered many - particle system is one that lacks long - range order .
more precisely , disordered systems have a pair correlation function @xmath0 ( defined below ) that decays to unity faster than @xmath1 , for spatial dimension @xmath2 and some positive @xmath3 , in the infinite - volume limit .
@xcite recently , a collective coordinate approach has been used to identify certain possibly disordered ground states .
@xcite although the previous studies are suggestive of the relation between disordered ground states and collective coordinates for finite systems , a systematic investigation of disordered ground states , including whether they exist in the infinite - volume limit , is not yet available .    in this paper
, we use an `` inverse '' approach to construct classical disordered ground states with precisely tuned wave scattering characteristics via the aforementioned collective coordinate procedure . in a recent example of a `` forward '' problem ,
the scattering from glass ceramics with nanometer - sized crystals was likened to that of random sequential adsorption ( rsa ) of hard spheres , @xcite a well - known , disordered many - body configuration .
@xcite these ceramics are of interest in photonics applications because they are mechanically rigid and nearly suppress all scattering at long wavelengths .
@xcite    by contrast , our method utilizes an inverse approach : we prescribe scattering characteristics ( _ e.g. , _ absolute transparency ) and construct many - body configurations that give rise to these targeted characteristics .
potential applications include designing ground - state materials as radiation filters or scatterers , and materials transparent to specific wavelengths of radiation , among others .
we apply our methodology initially for structureless ( i.e. , point ) particles .
however , it could be generalized to structured particles , colloids , or as bodies using the appropriate structure factors for finite - sized particles as is done for random media .
@xcite    since previous studies utilized small periodic systems , @xcite we first establish that systematically increasing the system size has no effect on the degree of disorder .
extrapolation from these results indicates that the constructed configurations remain disordered in the infinite - volume limit ; a seemingly counterintuitive proposition .
we then construct disordered ground states with special scattering properties : `` stealth materials , '' `` super - ideal gases , '' and `` equi - luminous materials . ''
we use the term `` stealth '' materials to refer to many - particle configurations that completely suppress scattering of incident radiation for a set of wave vectors , and thus , are transparent at these wavelengths .
@xcite periodic ( i.e. , crystalline ) configurations are , by definition , `` stealthy '' since they suppress scattering for all wavelengths except those associated with bragg scattering .
however , we construct disordered stealth configurations that prevent scattering only at prescribed wavelengths with no restrictions on any other wavelengths .    we define a `` super - ideal '' gas as a single many - particle configuration whose scattering _ exactly _ matches that of an ensemble of ideal gas configurations , or poisson point distributions , for a set of wave vectors .
a super - ideal gas has a structure factor that is identically unity for specified wave vectors , and thus , this single configuration would be impossible to differentiate from an ensemble of ideal gas configurations for the specified wave vectors .
we define an `` equi - luminous '' material to be a system whose scattering is constant for a set of wavelengths .
although stealth materials and super - ideal gases are subsets of equi - luminescent materials , we use this term to refer to materials that scatter radiation more intensely relative to an ideal gas . these materials that scatter radiation much more intensely than an ideal gas for a set of wave vectors have enhanced density fluctuations and show local clustering similar to polymers and aggregating colloids .
@xcite typically , scattering experiments on these systems are used to shed light on the characteristic length scales of a system.@xcite with our inverse procedure , we impose the degree of clustering by tuning the scattering characteristics for certain wavelengths .    upon generating ensembles of ground - state configurations for each class of materials described above ,
we characterize local order of each ensemble .
we place emphasis on pair information in real space via the pair correlation function @xmath0 and in reciprocal space through the structure factor @xmath4 as these functions are experimentally accessible and used widely in many - body theories .
@xcite    the pair correlation function @xmath0 is the normalized two - particle probability density function @xmath5 and is proportional to the probability of observing a particle center at @xmath6 relative to a particle at the origin.@xcite for a statistically homogeneous and isotropic medium , the pair correlation function @xmath0 depends only on the magnitude of @xmath7 , and is commonly referred to as the radial distribution function @xmath8 , which henceforth is the designation used in this paper .
the structure factor @xmath4 is proportional to the intensity of scattering of incident radiation from a configuration of @xmath9 particles and is defined as @xmath10 where @xmath11 are the collective coordinates and @xmath12 are the wave vectors associated with the system volume and boundary conditions .
collective coordinates @xmath11 are the fourier coefficients in the expansion of the density field : @xmath13 where @xmath14 denotes the location of particle
_ j_. when @xmath4 depends only on the the magnitude of @xmath15 , the structure factor @xmath16 is related to the fourier transformation of @xmath17 , ignoring the forward scattering associated with @xmath18 , @xmath19d{\bf r},\ ] ] where @xmath20 is the number density . for highly ordered systems ,
both @xmath8 and @xmath16 contain a series of @xmath21-functions or peaks at large @xmath22 and @xmath23 , indicating strong correlations at the associated pair distance . in configurations without long - range order , both @xmath8 and @xmath16 approach unity at large @xmath22 and @xmath23 .
several inverse methods have sought to construct systems using pair information in real space , particularly in addressing the question of pair correlation function `` realizability,''@xcite which asks whether a given pair correlation function , at number density @xmath20 , can be realized by spatial arrangements of particles .
typically , in real space , a `` target '' radial distribution function is chosen and many - particle configurations are found that best match the target @xmath8 .
stochastic optimization techniques have been a popular reconstruction method in finding spatial arrangements of particles that best approximate a target correlation function.@xcite    in contrast to these real - space methods , we target pair information in reciprocal space to construct configurations whose structure factor _ exactly _ matches the candidate structure factor for a set of wavelengths . in addition , our procedure guarantees that the resulting configuration is a ground - state structure for a class of potential functions .
the remainder of this paper is as follows .
section [ sec : background ] presents background on disordered ground states and motivates our choices for candidate structure factors , while section [ sec : methods ] outlines the numerical procedure .
structure factors , radial distribution functions , and representative particle patterns for stealth materials , super - ideal gases , and equi - luminous materials are found in secs .
[ sec : stealth ] , [ sec : superideal ] , and [ sec : lumin ] .
lastly , general conclusions and discussion relevant to classical disordered ground states and this procedure are found in sec .
[ sec : conclusions ] . appendix [ sec : energy ] compares minimization algorithms and analyzes the energy landscapes associated with our potential functions .
physical intuition and experimental facts suggest that in the zero - temperature limit , classical systems of interacting particles adopt a periodic structure to minimize potential energy .
the `` crystal problem '' has attempted to determine the fundamental mechanism that forces particles into ordered states , but the existence of these mechanisms has yet to be fully understood .
@xcite the notion of disordered ground states is particularly mysterious because of the lack of symmetry , lack of long - range order , and degeneracy of ground - state configurations .
the characterization of order in solid phases in the low - temperature limit has been well studied . in addressing the crystal problem ,
it has been suggested that nonanalyticity of thermodynamic functions may yield `` turbulent , '' or nonperiodic , gibbs states at positive temperature .
@xcite as a consequence , a turbulent crystal , characterized by fuzzy diffraction peaks , is possible as a nonperiodic solid phase , in addition to periodic and quasiperiodic structures . @xcite
turbulent crystals have been examined previously @xcite and evidence has been presented that at low temperature , equilibrium states may contain disorder . @xcite theoretical work has created classical lattice models with short - range interactions whose ground states contain the property of disorder .
@xcite in addition , a simple gradient model was used to develop a disordered state , a labyrinth , within a pattern forming system .
although this state was ultimately excluded from being a ground state , the authors were unable to exclude other models as potentially yielding disordered ground states .
@xcite despite significant research attention , understanding of the attainability of disordered configurations as classical grounds states is incomplete .
we choose to limit this investigation to the study of stealth materials , super - ideal gases , and equi - luminous materials based on the potential applications and fundamental interest .
constructing systems with transparency at specified wavelengths , including wavelengths outside of the low-@xmath23 region , is the primary motivation .
a stealth material has a structure factor that is exactly zero for some set of wavelengths . in the low-@xmath23 limit
, several disordered systems nearly suppress all scattering , though most are not ground states .
for example , when crystallizing hard colloidal spheres , stacking faults of close - packed layers create deviation from perfect crystallinity .
the structure factor is nearly zero in the low-@xmath23 region followed by a strong bragg peak and a diffuse peak before decaying to unity .
@xcite    the examples of stealth configurations can also be described as `` hyperuniform . ''
hyperuniform systems have the property that @xmath24 i.e. , infinite - wavelength density fluctuations vanish.@xcite hyperuniform point patterns arise in the structure of the early universe,@xcite maximally random jammed packings,@xcite certain tilings of space,@xcite and the ground - state configurations of certain repulsively interacting particle systems.@xcite    the structure factor for a super - ideal gas is exactly unity for a set of wave vectors and unconstrained for the remaining wave vectors .
we narrow our study to constraining wave vectors in the small @xmath23 region .
we choose super - ideal gases based on interest in poisson point distributions .
the poisson point process has the simplest candidate @xmath16 and @xmath8 , both being exactly unity for all @xmath23 and @xmath22 , respectively . for a single finite configuration ,
the structure factor exhibits random fluctuations about unity .
but in a super - ideal gas , we constrain a set of wave vectors so that @xmath16 is exactly unity and , for all wave vectors outside of this unconstrained set , the intuitive expectation is that the structure factor would ensemble - average to unity . in fact
, this does not necessarily happen and an interesting alternative behavior arises , discussed in detail below .
`` equi - luminous '' describes materials that scatter light equally intensely for a set of wave vectors .
the structure factor for this class of materials is simply a constant for a set of wave vectors .
subsets of equi - luminous materials include super - ideal gases ( @xmath25 ) and stealth materials ( @xmath26 ) . here
, we focus on materials whose structure factor is a constant greater than unity , as these systems show strong local clustering and intense scattering relative to an ideal gas .
the numerical optimization procedure follows that of uche , stillinger and torquato @xcite used to tailor the small @xmath23 behavior of the structure factor .
the structure factor @xmath16 and collective coordinates @xmath11 , defined in eqs .
( [ eq : sk ] ) and ( [ eq : rhok ] ) , are related to the quantity @xmath27 , @xmath28 where @xmath29.\ ] ] for a system interacting via a pair potential @xmath30 , the total potential energy can be written in terms of @xmath27 , @xmath31 where @xmath32 is the system volume and @xmath33 is the fourier transform of the pair potential function @xmath34 for a region of space with dimensions @xmath35 , @xmath36 , or @xmath37 in one , two , or three dimensions , subject to periodic boundary conditions , the infinite set of corresponding wave vectors has components @xmath38 where @xmath39 are positive or negative integers , or zero and @xmath40=@xmath41 as needed .
for example , in three dimensions , the set of wave vectors are @xmath42    we introduce a square mound @xmath33 that is a positive constant @xmath43 for all * k * @xmath44 , where @xmath45 is the set of wave vectors such that @xmath46 , and zero for all other * k*. in the infinite - volume limit , this corresponds to a system of particles interacting via a real - space pair potential function that is bounded , damped , and oscillating about zero at large @xmath22.@xcite for a cutoff radius @xmath47 , there are @xmath48 wave vectors in the set @xmath45 , where @xmath49 is the number of independently constrained collective coordinates
. that is , constraining @xmath27 implicitly constrains @xmath50 due to the relation @xmath51 for a system of @xmath9 particles in @xmath2 dimensions , there are @xmath52 total degrees of freedom .
we introduce the dimensionless parameter @xmath53 to conveniently represent the ratio of the number of constrained degrees of freedom relative to the total number of degrees of freedom @xmath54 the global minimum of the potential energy defined in eq .
( [ eq : phi ] ) has the value of @xmath55 if and only if there exist particle configurations that satisfy all of the imposed constraints , which necessarily occurs for @xmath56 .
minimizing eq .
( [ eq : phi ] ) to its global minimum , for @xmath56 , yields ground - state configurations that are stealthy for all * k * @xmath44 .    to target a specific form of the structure factor to certain nonzero values , such as @xmath16 = 1
, we introduce a second nonnegative objective function , @xmath57 ^ 2,\ ] ] where @xmath58 is associated with the target structure factor by eq .
( [ eq : sktock ] ) .
( [ eq : phitilde ] ) is taken to be the potential energy of an @xmath9-body system , then two- , three- , and four- body interactions are present.@xcite equation ( [ eq : phitilde ] ) has a global minimum of zero , for @xmath59 , if and only if there exist configurations that satisfy all of the imposed constraints . minimizing eq .
( [ eq : phitilde ] ) is used to construct super - ideal gases and equi - luminous materials as ground - state configurations .
three algorithms have been employed previously for minimizing eqs .
( [ eq : phi ] ) and ( [ eq : phitilde ] ) : steepest descent,@xcite conjugate gradient,@xcite and minop.@xcite steepest descent and conjugate gradient methods are line search methods that differ only in their choice of search directions .
@xcite the minop algorithm is a trust - region method .
when far from the solution , the program chooses a gradient direction , but when close to the solution , it chooses a quasi - newton direction.@xcite upon each iteration , the program makes an appropriate update to approximate the hessian.@xcite    we find that that neither the conjugate gradient method nor minop algorithm significantly biases any subset of ground - state configurations .
the resulting configurations are visually similar , and the ensemble - averaged radial distribution function and structure factor produced by both methods have similar features .
we chose the minop algorithm because it has been demonstrated to be better suited to the collective coordinate procedure than the conjugate gradient method .
we refer the reader to appendix [ sec : energy ] for characterization of the energy landscape and comparison between line search methods and minop .
three sets of initial conditions were considered : random placement of particles ( poisson distributions ) , random sequential addition ( rsa ) , and perturbed lattices ( integer , triangular , and face centered cubic in one , two , and three dimensions respectively ) .
for an rsa process , particles are assigned a diameter and randomly and irreversibly placed in space such that particles are not overlapping .
@xcite at sufficiently high @xmath53 , usually @xmath60 0.6 , the constructed ground - state systems apparently lose all memory of their initial configurations .
the analyses presented in the following sections will be those of random initial conditions .
in some cases at large @xmath53 , a global minimum is not found .
for the results discussed here , eqs .
( [ eq : phi ] ) and ( [ eq : phitilde ] ) were minimized to within @xmath61 of their respective minimum value .
all other trials were excluded from the analysis .
the region of space occupied by the @xmath9 particles was limited to a line in one dimension , a square in two dimensions , and a cube in three dimensions , with periodic boundary conditions .
for stealth materials , particular attention was paid to the choice of @xmath9 for two and three dimensions .
minimizing eq .
( [ eq : phi ] ) for large @xmath53 is known to yield crystalline ground states.@xcite we choose to be consistent with previous studies . in two dimensions ,
@xmath9 was chosen as a product of the integers @xmath62 , and @xmath63 is a rational approximation to @xmath64 so that all particles could be placed in a triangular lattice configuration without substantial deformation . in three dimensions , @xmath9 was usually chosen so that @xmath65 , where @xmath66 is an integer , so that the particles could be placed in a face centered cubic lattice without deformation . in minimizing eq .
( [ eq : phitilde ] ) , @xmath9 occasionally was assigned other values .
previous work suggesting the existence of disordered ground states utilized small simulation boxes containing up to several hundred particles .
@xcite our goal here is to show that constructed systems continue to show no long - range order in the infinite - volume limit . for @xmath2=2 and 3 , systems containing up to several thousand particles
were constructed by minimizing eq .
( [ eq : phi ] ) for small @xmath53 values .
we find that ground - state configurations differing only by @xmath9 , with @xmath20 and @xmath53 fixed , are disordered and exhibit the same local structure .
figures [ fig : stealth3dinfsk ] and [ fig : stealth3dinfgr ] demonstrate the behavior of @xmath16 and @xmath8 for stealth materials constrained at @xmath53=0.05 .
the structure factors for systems only differing in @xmath9 have identical characteristics .
the structure factor is exactly zero for constrained wave vectors and subsequently peaks above and fluctuates about unity .
the averaged @xmath16 initially peaks to a value of 1.10 and decays rapidly to unity , a feature that is more apparent for a large ensemble of configurations .
the radial distribution function remains essentially invariant as the number of particles in the simulation box increases from 108 to 6912 , with @xmath53 and @xmath20 fixed , as shown in figure [ fig : stealth3dinfgr ] . for a single realization
, a system containing 108 particles has the same shape of @xmath8 as that of the larger system but shows significant statistical noise . in the figure , we ensemble - average the results for the smaller system to make clear the structural similarities . in the smaller system , the large-@xmath22 behavior is unavailable due to the minimum image convention of the periodic box .
thus , the figure only displays local structure , which is clearly disordered . in both cases
, @xmath8 dips slightly below unity for small @xmath22 and quickly approaches and oscillates about unity with a diminishing amplitude .
it was previously reported that minimizing collective density variables for sufficiently high @xmath53 induces crystallization , @xcite therefore , for disordered , stealth ground states , we have minimized eq .
( [ eq : phi ] ) , focusing on the low @xmath53 regime .    for @xmath67
, crystallization occurs for @xmath68 0.5.@xcite for most values of @xmath53 below the crystallization threshold , the structure factor is zero for all constrained wave vectors , it then peaks above unity immediately outside @xmath47 and decays toward one . as @xmath53 is increased , the height of the peak decreases and at @xmath53 higher than 0.30 , a second peak forms .
below the crystallization value , @xmath16 dampens to unity for @xmath23 larger than @xmath47 .    for @xmath69 ,
three @xmath53 regimes have been reported : disordered for @xmath700.57 , `` wavy crystalline '' for @xmath71 , and crystalline for @xmath72 .
@xcite we choose to investigate well below these ordered regions . in the ensemble - averaged structure factor , a peak forms in @xmath16 immediately beyond @xmath47 and decays toward unity . as @xmath53 increases ,
the magnitude of the peak increases , in contrast to @xmath67 , and at the wavy crystalline threshold , several peaks begin to form beyond @xmath47 .
the height of the peak is density dependent , however , @xmath16 generally has a maximum between @xmath73 in the disordered region .    for @xmath74 , the transition from disordered to crystalline regimes was identified previously to occur at @xmath53 near 0.5.@xcite constraining @xmath53 below 0.45 , @xmath16 peaks immediately beyond @xmath47 and smoothly decays to unity , but at @xmath53 = 0.45467 , @xmath16 smoothly oscillates about unity .
the magnitude of the peak is generally smaller than for systems similarly constrained in lower dimensions . in figure
[ fig : stealth3dsk ] , we compare the structure factor for several @xmath53 values for 500 particles in a unit cube .
we include a nearly crystalline system of 500 particles constrained at @xmath53=0.54867 .
the order of the system is apparent by the series of sharp peaks in @xmath16 that persist at large @xmath23 .
we find that particles have a repellent core that increases in strength with increasing @xmath53 .
figure [ fig : stealth3dgr ] demonstrates the repellent core effect via the radial distribution function associated with the 500 particle system described above . for this particular system , at @xmath53
= 0.45467 , an exclusion region develops where @xmath8 is exactly zero for a region near the origin . at @xmath53
= 0.54867 , the peaks demonstrate crystallinity .
increasing @xmath53 tends to increase the the net repulsion of the potential , which is clearly observed in particle patterns . since differentiating between disordered stealth systems is most instructive in two dimensions , we present particle patterns in this dimension only .
figure [ fig : config2dstealth ] compares particle patterns of 168 particles with eq .
( [ eq : phi ] ) constrained for small @xmath53 .
the circular window in the figure represents the length scale of the wavelength associated with @xmath47 . at the lowest @xmath53
considered , the particles do not appear to have any spatial correlation . at higher @xmath53 values ,
particles develop an exclusion shell about their center but do not have any long - range order .
the entirety of disordered stealth materials studied have involved constraining wave vectors in a spherical shell near the origin .
since this procedure is capable of constraining wave vectors of choice , we have constructed configurations in which two disconnected , concentric regions of wave vectors are constrained , one near the origin and a second further from the origin .
we define parameters @xmath75 as magnitudes of limits for constrained wave vectors . in this class of stealth configurations , we constrain collective coordinates so that @xmath16 is zero for all wave vectors in two spherical shells about the origin in reciprocal space .
specifically , @xmath16 is zero for all @xmath76 and all @xmath77 .
the region @xmath78 is defined as the intermediate region , where the structure factor can be free to fluctuate or to be controlled .
figure [ fig : stealth3dsplit ] shows the location of parameters .
these systems are constructed by introducing a @xmath33 that contains two square mounds .
spherically constrained stealth configurations were constructed in two and three dimensions .
we present ensemble - averaged radial distribution functions and structure factors for @xmath74 and particle patterns for @xmath69 as these provide the clearest representation of the general trends .
the peaking phenomenon in @xmath16 , as originally observed in simple stealth materials , is evident both immediately beyond @xmath79 and beyond @xmath80 .
the structure factor increases above unity slightly in the intermediate region but peaks and decays rapidly to unity beyond @xmath80 .
this is seen for all test cases and it is not immediately clear if this phenomenon persists for large separations of constrained regions ( i.e. , @xmath81 ) .
figure [ fig : stealth3dsplit ] shows the ensemble - averaged radial distribution function and structure factor for configurations of 500 particles constrained so that @xmath16 = 0 for @xmath82 and @xmath83
.    an important feature of this procedure is the ability to suppress scattering for wave vectors that are normally bragg peaks in a crystalline material . with a 500 particle , three - dimensional system , minimizing eq .
( [ eq : phi ] ) for a single square mound @xmath33 for @xmath53 = 0.54867 creates a crystalline ground state where the first bragg peak occurs just beyond @xmath23 = 14@xmath84 .
the constructed system in figure [ fig : stealth3dsplit ] suppresses scattering for a range of @xmath23 surrounding 14@xmath84 . because the intermediate set is free to fluctuate , the total number of constrained wave vectors in this stealth configuration is less than that of a crystalline configuration .
the stealth region away from the origin can be shifted to larger @xmath23 and can be tailored in magnitude .
the scattering of the intermediate set of wave vectors can also be controlled using eq .
( [ eq : phitilde ] ) .
one such configuration that can be developed is a stealth / ideal gas hybrid . in these many - particle systems
, there is behavior typical of crystalline solids at selected regions of reciprocal space and ideal gas at others . in figure
[ fig : stealth3dsplitid ] , a 500 particle system in three dimensions is constrained so that @xmath16 = 0 for @xmath82 and @xmath83 .
the intermediate set is constrained to @xmath16 = 1 .
the characteristics of the radial distribution function vary depending on the constraints of the intermediate set . in all cases ,
the radial distribution function shows weak neighbor peaks before approaching unity .
the second neighbor peak , though , is stronger than the first neighbor peak , a trait uncommon to most conventional many - body systems .
the extent of the repelling - core region varies depending on the chosen value of @xmath79 and the extent to which the intermediate set is controlled , evidenced by figures [ fig : stealth3dsplit ] and [ fig : stealth3dsplitid ] . further reducing the the value
to which the intermediate set is controlled is likely to increase the repelling - core region given the relation between @xmath16 and @xmath8 in eq .
( [ eq : sktogr ] ) .    spherically constrained stealth particle patterns consisting of 168 particles with the intermediate set respectively unconstrained and constrained are shown in fig .
[ fig : stealth2dcompare ] . to serve as a basis of comparison ,
a realization of a wavy crystalline configuration generated by suppressing scattering for all wave vectors up to 22@xmath84 is also shown . the wavy crystalline material , fig .
[ fig : stealth2dcompare]a , stands in sharp contrast to the stealth materials since particles tend to align in well - patterned strings .
the spherically constrained stealth materials lack any order . with the intermediate region
uncontrolled , fig .
[ fig : stealth2dcompare]b , particles tend to align into weak `` strings . ''
creating a stealth / ideal gas hybrid , fig .
[ fig : stealth2dcompare]c , decreases the tendency to align in weak strings .
the diameter of the circular window is equivalent to the length scale of @xmath80 , indicating the ability to impose system features with a specified length scale .
super - ideal gases were constructed at various @xmath53 values for @xmath85 , and 3 .
the maximum attainable @xmath53 value varied depending on spatial dimension and system size but was generally near @xmath86 for most initial configurations . at @xmath53 near unity ,
the minimization routine sometimes failed to find a global minimum of @xmath87 .
however , in @xmath74 and @xmath53 near unity , the success rate for finding global minimum was much improved over that of lower dimensions .
the results from all dimensions studied have similar characteristics . for a single realization ,
the structure factor is exactly unity for @xmath88 .
outside of the constrained region , the structure factor seemingly fluctuates about unity .
however , for an ensemble of super - ideal gases , the structure factor had a small peak immediately beyond @xmath47 that slowly decayed to unity .
this small peak is unexpected since we impose no constraints on @xmath16 for wave vectors outside @xmath47 and would expect that correlations not exist at these @xmath23 .
the construction of super - ideal gases reveals a subtle coupling between @xmath16 within the constrained region and @xmath16 outside the constrained region that manifests upon ensemble averaging . for @xmath67 , @xmath89 , and @xmath90 , @xmath16
never exceeds a value of 1.25 , 1.18 , and 1.10 respectively when ensemble averaged . for @xmath91 and @xmath92
, the peak generally decays to unity rapidly . for @xmath93 ,
the peak is rather long ranged , decaying much more slowly .
figure [ fig : super3dhighandlow ] shows the ensemble - averaged @xmath8 and @xmath16 for two systems containing 500 particles in @xmath74 .
the radial distribution function has characteristics common across dimensions studied . for @xmath94 ,
@xmath8 shows very small fluctuations about unity .
the local behavior of @xmath8 , @xmath95 , varies depending on @xmath53 and is most sensitive to @xmath53 in @xmath74 .
figure [ fig : super3dhighandlow ] shows that super - ideal gases at @xmath96 exhibit severe local clustering as the radial distribution function has a contact value @xmath97 near 7 .
however , at @xmath98 , the structure more closely resembles an ideal gas .
figure [ fig : supergrpeak ] tracks the contact value of @xmath8 for various @xmath53 . at significantly large @xmath53 , local clustering is suppressed and the super - ideal gas structure closely resembles that of an ensemble of ideal gas configurations .
an interesting consequence is that the local structure of a super - ideal gas at very small @xmath53 resembles that of a super - ideal gas at very large @xmath53 .
two - dimensional particle patterns reveal subtle differences between super - ideal gas configurations and poisson distributions .
figure [ fig : super2dconfig ] compares a realization of poisson distribution of 418 particles and a super - ideal gas at @xmath53=0.90 . at certain @xmath53 , super - ideal gases exhibit local order and a tendency to align into weak `` strings ''
that is best revealed only upon ensemble averaging . at smaller @xmath53 ,
super - ideal gases are particularly difficult to discern from a poisson distribution .
equi - luminous materials represent a broad class of materials that scatter light equally intensely for a set of wave vectors , where a super - ideal gas is a special case .
we choose to focus here on equi - luminous materials whose scattering in the small-@xmath23 region is much more intense relative to that of an ideal gas .
for all cases considered , we observe qualitative similarities in the ensemble - averaged radial distribution function and structure factor .
the structure factor is exactly equal to the chosen constant for all @xmath88 . beyond @xmath47
, the ensemble - averaged structure factor decays to unity very slowly at a rate dependent on the specified constant in constrained region .
figure [ fig : lumin2dsk ] compares @xmath16 for several equi - luminous configurations containing 168 particles for @xmath69 .
these configurations have constrained @xmath16 values that are exactly 1,2,3 , and 4 , respectively , for @xmath99 . achieving @xmath53 above 0.37 for @xmath16 = 4.0 for @xmath69 proved challenging as the minimization procedure often failed to find global minima .    in real space
, we observe that constraining @xmath16 to be increasingly large has no affect on the long - range behavior of @xmath8 . generally , for @xmath94 , @xmath8 averages near unity .
however , for @xmath100 , strong local correlations rapidly vanish for increasing @xmath22 . by increasing the constrained value of @xmath16 , the contact value of @xmath8 increases significantly but does not change the large @xmath22 behavior , which remains near unity . figure [ fig : lumin2dgr ] demonstrates the behavior of @xmath8 corresponding to the systems described above .
realizations of equi - luminous materials demonstration the aggregation of particles common to this class of equi - luminous materials .
figure [ fig : lumin2dconfig ] shows configurations for which @xmath16 = 2 and @xmath16 = 4 for @xmath53 = 0.37 .
particles tend to cluster and align into well - formed strings , or filamentary structures , as opposed to clustering radially .
filamentary structures arise in astrophysical systems , particularly for the distributions of galaxies . @xcite for a larger target @xmath16 , the aggregation is increasingly severe and particles nearly stack on top of each other .    in the extreme case of targeting @xmath16 toward its maximum value of @xmath9 for very small @xmath53 , particles tend to collapse upon each other , yielding an overall rescaling of the system length scale .
it should be noted that in these extreme cases , global minima were rarely found and required successive iterations to achieve optimality .
for example , in seeking a ground - state configuration with @xmath101 for @xmath102 , the ground - state configuration in which @xmath103 for @xmath102 was used as the initial condition .
previous studies concerning constraints on the collective coordinates of particle configurations @xcite were suggestive that potentials defined by eqs .
( [ eq : phi ] ) and ( [ eq : phitilde ] ) yield classical disordered ground - state configurations .
these studies restricted the system size to small periodic boxes consisting of at most 12 particles in @xmath67,@xcite 418 particles in @xmath69 , @xcite and 500 particles in @xmath74.@xcite in this investigation , we find that increasing the system size while fixing @xmath20 and @xmath53 does not affect the structure of resulting configurations , suggesting that there are no long - range correlations in the infinite - volume limit .
constructing ground state materials via minop for systems significantly larger than that presented in section [ sec : stealth ] becomes computationally challenging and currently is the limitation on system size .
regardless , our numerical evidence for disordered ground states in the infinite - volume limit corroborates recent analytical work that suggests the existence of energetically degenerate and aperiodic ground states in the infinite - volume limit via a similar potential.@xcite    three novel classes of ground - state materials with potential radiation scattering applications have been introduced : stealth materials , super - ideal gases , and equi - luminous materials .
each provides an unique opportunity to impose an underlying structure for a ground - state configuration with a known potential . with stealth materials
, we have the precise control to suppress scattering at specified wavelengths . for a single square - mound @xmath33 , increasing the number of constraints on wave vectors near the origin drives systems toward crystallinity .
@xcite however , by introducing @xmath33 with two square mounds and choosing to suppress scattering in two disconnected regions of reciprocal space , we disrupt the drive toward crystallinity .
particle patterns then have strong local correlations with an imposed length scale , but lack long - range order .
it is interesting to note that all stealth ground - state configurations generated in this investigation are also hyperuniform point patterns ; i.e. , configurations that suppress density fluctuations in the infinite - wavelength limit . in
choosing which regions of reciprocal space to be stealthy , we exhibit the unique ability to control and suppress density fluctuations at specified wavelengths .
equi - luminous materials investigated in section [ sec : lumin ] demonstrate ground - state structures that scatter light for a set of wavelengths more strongly than an ideal gas . in choosing the the number of constrained wave vectors and the value of @xmath16
, one can design ground - state configurations that have increasingly clustered behavior for increasing @xmath53 and @xmath16 . by adjusting @xmath47
, we can impose correlations over a certain length scale .
the differences between super - ideal gases and ideal gases are not intuitively obvious .
the resulting local structure deviates significantly from that of an ideal gas as there is some degree of local clustering , with the exception of @xmath53 near 0 and @xmath53 near 1 .
the propensity for super - ideal gases to exhibit local clustering suggests that , among the many energetically degenerate global minima in the landscape , configurations with some degree of clustering dominate relative to those that do not .
we also find that this local order is particularly sensitive to the number of constraints imposed suggesting that the energy landscape changes significantly for small changes in @xmath53 . despite the local clustering , the resulting configurations exhibit no long - range order .
one potential application of this procedure is in addressing the realizability question . despite receiving significant attention ,
only necessary conditions have been placed on the pair correlation function @xmath8 and its corresponding structure factor @xmath16 for the realizability of a point process.@xcite general sufficient conditions have yet to be developed .
this method can potentially address realizability from reciprocal space , which stands in contrast to real - space numerical reconstruction techniques.@xcite a limitation in its use for the question of realizability is the peaking phenomenon in @xmath16 . in
all cases studied , @xmath16 has a peak immediately beyond @xmath47 , suggesting collective coordinates are not necessarily independent of each other and that the nonlinearity of the coordinate transformation plays an important role in constraining collective coordinates .
additionally , our results show that the magnitude of the peak is reduced for higher dimensions , suggesting that dimensionality plays a key role in the coupling among collective coordinates .
the understanding of constraints on collective coordinates is incomplete .
new questions arise : how are the wave vectors beyond @xmath47 influenced by constraints below @xmath47 and what role does dimensionality play in the peaking phenomenon ?
another possible application of this procedure is to produce ground - state configurations that are ordered or quasiperiodic over a specified length scale for the design of photonic materials .
materials with a photonic band gap are of significant interest due to their technological applications , @xcite and materials with desired band gaps have been designed.@xcite from first - order perturbation theory , scattering of radiation is related to the photonic band gap of a material.@xcite by targeting a structure factor that is maximal for certain wave vectors and zero for others , we may be able to construct ground - state structures with a specified desired photonic band gap to a first - order approximation .    in the present investigation ,
the potential @xmath33 was chosen to be a square mound with compact support at @xmath47 . however , the constructed ground - state configurations are equivalent to the ground - state configurations associated with a broad class of @xmath33 . if a ground - state configuration of density @xmath20 is constructed by minimizing eq .
( [ eq : phi ] ) or eq .
( [ eq : phitilde ] ) to its global minimum value , eq .
( [ eq : min ] ) or zero respectively , then the ground - state configuration is also a ground state at density @xmath20 for any positive @xmath33 that is bounded with the same compact support at @xmath47 .
recently derived duality relations allow us to identify further degeneracies among ground states constructed via eq .
( [ eq : phi ] ) and provide bounds on the minimum energy for various forms of @xmath104 .
these duality relations link the energy of configurations for a bounded and integrable real - space pair potential function to that of its fourier transform and provide a fundamental connection between the ground states of short - range pair potentials to those with long - range pair potentials .
more specifically , if a bravais lattice structure is the ground state energy @xmath105 for a pair potential function @xmath104 , then @xmath105 provides an upper bound on the ground state energy @xmath106 of a system interacting via the fourier - transform of the pair potential function @xmath33 at the associated reciprocal density.@xcite in using eq .
( [ eq : phi ] ) , we used a long - ranged , damped , and oscillatory @xmath104 , whose fourier transform is a square mound @xmath33 . for @xmath59 and at a given density @xmath20 , this @xmath104 had a bravais lattice as a ground - state structure , and for sufficiently large @xmath53 , a bravais lattice becomes its unique ground - state structure at a certain density . by these duality relations ,
we know that the minimum energy of this bravais lattice system , eq .
( [ eq : min ] ) , is the upper bound on the ground - state energy of systems at density @xmath107 interacting via @xmath33.@xcite    targeting specific forms of the structure factor with eq .
( [ eq : phitilde ] ) requires up to four - body interactions in real space to achieve these as ground states .
however , it would be particularly useful to construct disordered ground states using short - range , two - body potentials .
developing effective short - range , pair interactions , potentially via the ornstein - zernike formalism,@xcite to achieve desired scattering properties remains a potential future direction .
the authors thank obioma uche and chase zachary for for helpful discussions regarding several aspects of this work .
s. t. thanks the institute for advanced study for their hospitality during his stay there .
this work was supported by the office of basic energy sciences , u.s .
department of energy , under grant de - fg02 - 04-er46108 .
for large @xmath53 , the procedure sometimes terminates at a potential higher than the absolute minimum , failing to meet our criterion for global minimum .
generally this has been attributed to local minima in the energy landscape associated with eqs .
( [ eq : phi ] ) and ( [ eq : phitilde ] ) and the performance of the minimization algorithm . @xcite since little is known about these landscapes , we shed some light unto them and justify the discrepancy between the performance of the various algorithms .
the landscape of @xmath87 can be visualized for a system of three particles on a unit line . imposing the constraint so that @xmath108 , the energy , @xmath87 , becomes @xmath109 + \cos[2\pi(x_1-x_3 ) ] + \cos[2\pi(x_2-x_3 ) ] + \frac{3}{2 } - d\}^2 .
\label{eq : threepart}\ ] ] @xmath110 must be within the range of realizability for @xmath16 ( i.e. , @xmath111 ) .
the energy landscape possesses translational freedom , so we can fix @xmath112 at the origin , @xmath112=0 . plotting eq .
( [ eq : threepart ] ) versus particle coordinates for the case of @xmath110=0 and @xmath110=1 provides a simple picture of the energy landscape . figures [ fig : lszero ] and [ fig : lssuper ] show the energy landscape and contour plot for @xmath110=0 and @xmath110=1 respectively .    for a stealth configuration , @xmath113 , figure [ fig : lszero ] ,
there is exactly one solution , corresponding to the crystalline arrangement of particles , that is a global minimum of @xmath87 .
the global maximum is a stacking of all particles onto the origin . for @xmath114 , figure [ fig : lssuper ] ,
there is a family of degenerate ground - state solutions located in a ring around configurational points corresponding to the periodic arrangement of particles .
for this system , a single ground state exists only for @xmath113 or @xmath115 . for all other @xmath110 , there exist degenerate ground states .
taylor expansion about the minima for when @xmath110=0 indicates the landscape is quartic in nature .
when @xmath116 , local maxima are located at configurational points associated with the periodic arrangement of particles , as evident in figure [ fig : lssuper ] . as @xmath110 is increased , the local maxima are found increasingly higher in the landscape . above @xmath110=1.5 ,
the periodic arrangements of particle become global maxima and the origin becomes a local maximum .
visualization confirms that the energy landscape is nontrivial , particularly for @xmath116 .
local minima were not found despite our experience with the minimization procedure .
local minima can be found by constraining the next wave vector .
for example , constructing a system for @xmath117 at @xmath53=0.667 for a three - particle system introduces local minima . if @xmath118 and @xmath119 , @xmath87 can not be minimized to zero at @xmath53=0.667 .
the crystalline arrangement achieves @xmath87=0 for @xmath113 and the stacking of particles achieves @xmath87=0 for @xmath115 .      to enumerate and characterize the stationary points ( local / global minimum or maximum , or saddle point ) of the energy landscape @xmath87 , eq .
( [ eq : phi ] ) , we introduce a nonnegative function , @xmath120 , where @xmath121 refers to particle coordinates : @xmath122 the global minima of @xmath120 have a value of zero and correspond to stationary points of the energy landscape @xmath87 .
capture basins are the regions of phase space that , upon minimizing @xmath120 , yield each stationary point . for three particles on a line ,
the stationary points are countable from the visualization of the landscape , and this provides a basis for comparison .
we use steepest descent since it will generate nonoverlapping capture basins and we compare against minop .
if at a certain point in the energy landscape minop s trust region overlaps with a different capture basin , it may favor a search direction that enters the overlapping capture basin and lead to a different stationary point .
table [ tab : threepart ] summarizes the results of the capture basin analysis for a three particle one - dimensional system constrained at @xmath23=1 and @xmath23=2 from several hundred trials .
random positions are assigned and @xmath120 is minimized , yielding a stationary point .
many times @xmath120 was not minimized to a global minimum indicating that a stationary point of @xmath87 was not found . in the table
, the number of capture basins refers to the countable number of basins in the landscape with one basin associated with each stationary point , without accounting for particle permutations .
the fractions in the chart represent the fraction of the total number of stationary points found corresponding to each stationary point , or alternatively , the fraction of phase space that corresponds to a specific capture basin for each stationary point .
the capture basins for a two - dimensional system of 12 particles were found for all available values of @xmath53 , for @xmath113 . because of the increased speed of minop
, we only employ this algorithm to minimize @xmath120 . upon generating a stationary point ,
the potential @xmath87 was calculated and the eigenvalues of its associated hessian matrix were found .
global minima of @xmath87 have nonnegative eigenvalues and @xmath123 .
saddle points have at least one negative eigenvalue and @xmath124 .
interestingly no local minima , local maxima , or global maxima were found .
table [ tab:12part ] summarizes the capture basins and stationary points for 12 particles in two dimensions .
trials that did not yield stationary points are excluded from the reported fractions .
no local maxima , global maxima , or local minima were found , however , they must exist in systems of sufficiently high dimension .
the most unstable saddles generally have a high potential exceeding the global minimum by @xmath125 , while the most stable saddles have potentials @xmath87 that differed from a global minimum by o(@xmath126 ) .
our experience with finding local minima is consistent with these results .
when minop terminates at @xmath87 above its global minimum value , which usually occurs when @xmath127 , the potential of the final configuration is about @xmath128 to @xmath129 above the global minimum value . the strong correlation between the number of negative eigenvalues and potential is consistent with our experience with these algorithms .
the number of saddle points grows rapidly with the total dimensionality of the system ( number of particles times spatial dimension ) , and grows more rapidly than any other stationary points . in the three - particle system at @xmath53=0.667
, we observe that local maxima are located near saddle points in the energy landscape .
it is likely that minimizing @xmath120 introduces a bias toward finding saddle points of @xmath87 rather than local maxima of @xmath87 .
because local maxima are outnumbered by saddle points and the search procedure may favor directions toward saddle points of @xmath87 , it likely that minimizing eq .
( [ eq : phidotphi ] ) would rarely generate local maxima for systems of high dimensionality .
global maxima are attained only by stacking particles in a single location .
it is expected that this stationary point would be so greatly outnumbered by all other stationary points that minimizing @xmath130 may never generate a global maximum .
these simple studies allow us to better justify the inferences regarding the landscape associated with @xmath87 . overconstraining the three particle system for targeting a super - ideal gas introduced local minima .
increases to the system s dimensionality and to @xmath53 increased the number of saddle points and is expected to increase the number of local minima .
comparisons between minop and line search techniques indicate that the trust region often overlaps other capture regions which find global minima of @xmath87 more effectively .
.[tab : threepart]fraction of phase space corresponding to capture basins of landscape @xmath87 for three particles on a line as found by steepest descent ( sd ) and minop algorithms . [ cols="^,^,^,^,^,^,^ " , ]


_  damn referees , i ll miss them less than anybody . " _  abe lemons    scholars have long been dissatisfied with aspects of the peer review system , but now there is an increasing sense that something drastic should and could be done about it . in the present article
we discuss some recent reform proposals given our own experiences publishing in theoretical and applied statistics .
scholarly publishing has many problems , a lot of degrees of freedom , and various dimensions on which one could measure success . put these together and you have the potential for unending opportunities for change , with each new idea promising improvement in some aspect of the system .
thus it is no surprise that many different proposals have arisen for reforming the peer - review system , with each new plan sounding completely reasonable but in contradiction to proposals by others .
we bring this up not as a way to cynically dismiss innovation in this area but rather to explain how it is possible that so many completely different ideas can seem to make so much sense .
it is a bit like reform of primary education : some people recommend back - to - basics while others recommend modern student - centred learning .
either approach could make sense , given the many different ways that such ideas can be implemented and the multiplicity of potential outcome measures .    as statisticians
, it would be most appropriate for us to evaluate reform proposals by analysing existing data , systematically gathering new information , or at the very least proposing a plan for sampling , measurement , and causal inference .
unfortunately , as in much of our professional lives , we do not practice what we preach . instead of systematic data collection and analysis
, we offer the same sort of mix of opinion and personal experience that you might get from any other participant in the system .
thus in this article we speak not as statisticians but merely as a particular set of academic researchers with our own particular perspective .
_  off with the ref ! " _
scientific and scholarly publishing has for many years been centred on peer - reviewed journals , where the authors of published articles are responsible for their correctness , while editors and referees vouch for the novelty and importance of the work .
each field has a set of journals with a rough partial ordering ( top theoretical and applied journals , top sub - field journals , second - tier , and third - tier publications ) .
widely - acknowledged problems with the current system include inefficiency for authors ( time spent preparing and revising a single article for multiple journals ) , waste of reviewers efforts ( long referee reports that are read by only two people  the author and the journal editor ) , a breakdown of the sorting process ( so that it is no longer sufficient for scholars to keep up with a field by reading three or four top journals ) , the exclusion of `` gated '' research from people without access to a major university library , and , most importantly , a profusion of unreplicated or unreplicable claims even in the highest - prestige outlets .    for instance , wasserman s ( @xcite ) remarks that `` we are using a refereeing system that is almost 350 years old .
if we used the same printing methods as we did in 1665 it would be considered laughable . ''
he describes the refereeing process as``noisy , time consuming and arbitrary , '' that it `` limits dissemination '' and that provides an `` illusion '' of quality control .
he likens the process to a `` priesthood '' or `` guild '' and advocates its replacement by a `` marketplace '' of ideas .
concerns with the peer - reviewed publication system are not new ; they are , for example , integral to the relative attribution of credit to newton versus leibniz for the invention of differential calculus in the 17th century @xcite .
proposals for reform typically choose among the following options : ( 1 ) removing journals roles as copyright holders and gatekeepers by setting up incentives for researchers to post their articles freely on the web , ( 2 ) formalising a post - publication peer - review process so that referee reports are open and available for all to read , and ( 3 ) putting more of the burden of proof of replicability on published work by requiring data - based articles to come with full replication materials .
each of these steps has been taken already , to some extent .
personal websites and servers such as arxiv ( physics and mathematics ) and ssrn ( social science ) are widely used for posting of unreviewed preprints .
you ca nt come in off the street and publish at arxiv or ssrn but it is not difficult for a researcher to establish the connections necessary to post there .
post - publication peer review exists in some journals and , more effectively , in an informal network of scientific blogs .
the goal of ensuring replicability is tougher , but some journals ( for example , the _ quarterly journal of political science _ ) do require a full suite of replication materials before allowing any empirical article to be published .    thus , proposed reforms typically involve taking some aspect of the current system and pushing it further . here
are three recent suggestions :    1 .
theoretical statistician larry wasserman ( 2012 ) calls for `` a world without referees '' : + _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ `` we should be disseminating our research as widely as possible .
instead , we let two or three referees stand in between our work and the rest of our field . (  ) we should think about our field like a marketplace of ideas .
everyone should be free to put their ideas out there .
there is no need for referees .
good ideas will get recognised , used and cited .
bad ideas will be ignored . ''
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2 .
cognitive psychologist nikolaus kriegeskorte ( 2009 , 2011 ) proposes `` open post - publication peer - review '' : + _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ `` any scientist can instantly publish a peer review on any published paper . the scientist will submit the review to a public repository .
(  ) the repository will link each paper to all its reviews , such that that readers are automatically presented with the evaluative meta - information .
in addition , the repository allows anyone to rank papers according to a personal objective function computed on the basis of the public reviews and their numerical quality ratings . ''
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 3 .   political scientist brendan nyhan ( 2012 ) , following some ideas that have become popular in medical research , recommends that data - collection protocols be published ahead of time , with the commitment to publish the eventual results : + _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ `` in the case of experimental data , a better practice would be for journals to accept articles before the study was conducted .
the article should be written up to the point of the results section , which would then be populated using a pre - specified analysis plan submitted by the author .
the journal would then allow for post - hoc analysis and interpretation by the author that would be labelled as such and distinguished from the previously submitted material . by offering such an option
, journals would create a positive incentive for preregistration that would avoid file drawer bias .
more published articles would have null findings ( at least 5% ! ) , but that s how science is supposed to work . ''
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _    all three of these proposals are appealing , compelling , and radical  and go in different directions , with the statistician wanting to eliminate referees , the psychologist recommending reviews but in a different structure , and the political scientist proposing a more stringent system of pre - publication quality control .
our goal is not to evaluate these particular proposals but rather ( a ) to consider the relevance of these ideas for the fields of applied and computational statistics , and ( b ) to emphasise the value of the referee system and to focus attention on how to not lose its benefits in this time of change ..    this note summarises our reflections on the worth of the current and admittedly imperfect referee system , while making constructive proposals towards its improvement , rather than its abolition .
we attempt to find a middle ground between what we have now and various proposed reforms . in our opinion ,
the debate is as much about ethics as it is about science , namely how to work out a system of dissemination in which papers are evaluated on the basis of their scientific worth , rather than on the paper s conformity with existing norms ( a problem with the traditional system of peer review ) , its potential popular impact ( an issue with proposed open alternatives ) , the author s reputation or networks , or the reviewer s own long - term plans .
based on our own experiences , we argue that in this era of data explosion , the referee system remains preferable to the frightening morass of an uncontrolled accumulation of self - published documents
.    it can be instructive to compare a concern at hand with a parallel situation . where else do we find this practice of refereeing ?
in sport , of course .
so we will make use of the cries heard during a soccer match as an apt introduction to some of our points throughout the note .
_  whadya call that , ref ? ! " _    each field or sub - field brings its own perspective on publishing . for a mathematician or theoretical statistician such as wasserman ,
what s important in a publication is the idea .
mathematical ideas can be evaluated openly and , in principle , by anyone . from the other direction
, nyhan focuses on the _ difficulty _ of replicating empirical results , especially given the selection problem that positive rather then negative findings tend to get published . as applied statisticians
, we see the merits of both approaches , depending on what we are working on .    at the same time that mathematicians are moving to deregulate the academic publication system
, many experimental scientists are pushing toward more formal registries . beyond their direct benefit in replicability ,
such reforms involve incentives for better behaviour on the part of researchers .
if you know ahead of time that you will have to supply details of your design , methods , data and computer code , you will be motivated to keep better records and clearer codes the first time , which in turn leads to a positive feedback in which later analyses are improved by iterating on existing material .
this is the argument made by orourke and detsky ( @xcite ) .
publication patterns also vary among academic fields .
some of the best mathematicians and economists work alone or in small collaborations and publish papers after they have been honed by workshop and seminar presentations , while , at the other extreme , leading physicists , biologists , and electrical engineers supervise laboratories producing dozens of publications a year . in the first case , one could argue against an extra refereeing stage , however biases in the workshop process also need to be ironed out by this anonymous refereeing step .
turning now to the analogy , we could ask the same question : why do we have referees in sport ? there is a myriad of backyard and self - organised games around the world , but a neutral referee is demanded in major games .
( of course , each side may also bring their own umpires or referees , which again has potential parallels with current and possible future practices in scientific refereeing , but we will not pursue this here . )
it is argued that such a referee can maintain , and ensure , a high quality , fair and fast game for the benefit of both players and spectators .
surely we ask the same of a scientific referee : assurance of quality  in terms of the merit , originality and substantive contribution of the scientific content ; fairness  in terms of equitable treatment for all authors ; consistency  in terms of reasonable , useful feedback to authors ; and timeliness  a fast turnaround of reviews .
these are the very qualities that wasserman ( 2012 ) laments are lacking in the current process .
_  he did nt touch him , ref ! " _
the previous section seems to make the usual argument that refereeing is a necessary evil .
we believe on the contrary that it is a necessary good .
yes , certain referees are annoying , or even aggressive or too dismissive about one s work .
of course , like others , we can tell horror stories about referees completely missing the point or even being outright dishonest . in the past decades
, we indeed have all accumulated good and bad experiences about the ( human and hence imperfect ! ) refereeing process . as authors of dozens or hundreds of peer - reviewed publications , we have however benefited immensely from the unpaid labour of referees ( while at the same time having spent countless hours serving as referees , associate editors , and sometimes editors ) .    at times , we ve been annoyed at having to jump through hoops  and some of us prefer writing books and blogs , where in both cases out our audience is the reader not the referee  but more often than not the suggestions are helpful .
for example , gelman s ( @xcite ) most successful article of the past decade was his paper on prior distributions for hierarchical variance parameters .
this article originated as an example in an appendix of a book co - authored by gelman ( _ bayesian data analysis _ ) , and then was solicited as an article by the editor of the then - new journal _
bayesian analysis_. the referees were brutal ( maybe unfairly so , considering the positive reaction the paper had had in the years since publication ) and the paper could only be published in the journal in the form of a discussion of another article on a related topic . however , as a consequence of this revision process , gelman was motivated to add a whole new section that made the research much more general and interesting .
it is thanks to the referees that the author put in the work to make the paper what it was .
a similar experience occurred with the paper of gelman and rubin ( 1992 ) .
another extreme example experienced by chopin is that of a referee who was adamant about rejecting a paper on grounds that the authors believed were quite unreasonable , but in the third revision the referee did point out a mistake in a sampling algorithm . since publishing a wrong paper is much more damaging in the long run than being rejected by a given journal , this turned out to be most useful .    at the other end of the spectrum
are sloppy referees who form a strong opinion based on a cursory read , along with their particular priors about the topic in question .
the result , especially for competitive journals , is often a rejection based on vague and unrelated comments , which also contributes to an incentive structure that favours incremental and conventional work .
alternatively , an `` accept '' decision based on shallow refereeing can allow a poor paper to be published .
often , however , the system corrects itself , with the discrepancy with the other reports or the lack of substance in the review being spotted by associate editors or editors .
( we are not here considering the referees who follow the unethical practice of rejecting a paper and rushing to write one of their own based on the same idea , which has happened to us only rarely . )
we also believe that our papers are preemptively improved by refereeing , in that we mostly write better papers because we know they will be critically evaluated by colleagues prior to publication .
we go the extra mile , chase typos , think more carefully about real examples , and so on , before submitting , because we do not want to give a negative referee this additional and objective leverage we can ourselves perceive .
some authors seem to be outraged when referees demand particular modifications , because they consider that , as authors , they should have an inalienable right to decide the exact form of their texts .
but we are not artists ( who actually rarely obtain this right anyway ) , we are scientists , and science ( in particular hard science , especially mathematics ) works better through consensus on the validity and correctness of the proposed research .
_ `` the trouble with referees is that they know the rules , but they do not know the game . ''
_  bill shankly .    recognising that scientific review processes have been evolving forever , our current view of the traditional model is that editors send submitted manuscripts to selected reviewers for comment , and then make a decision based on these comments and their own judgement
. the issues of concern in such a simple system arise from the arbitrary and often narrow selection of reviewers , the generous and often unreasonable time allowed for response , the general and often unhelpful guidelines for comments , the opaque manner in which the final decision is made , and more generally the huge and often wasted investment in time by authors and referees .
in particular , junior scholars can take their refereeing duties very seriously , writing long and careful reports on papers which are of such low quality to not be worth the effort .
we agree with kriegeskorte and deca ( 2012 ) that a better use of reviewers time and effort would be to have many reviews of important papers and only zero or one review for the sorts of minor efforts that fill up our journals .
conversely , even a very specialised result can sometimes be useful ; in this case it might well merit a post - publication review thread by its user community , in the same way that more popular restaurants and hotels will receive more reviews on yelp and tripadvisor , giving consumers more information to make their judgements .
some journals have evolved from this traditional model in order to address these issues .
for example , some no have databases of reviewers from which to more objectively draw subject - specific referees ; others demand short review times ; others have formalised the referee process by instituting a list of questions to be answered or providing careful guidelines about the type of review required ; and a small number have adopted the post - modern ( or in fact pre - traditional ) practice of the editorial board itself making decisions at regular team meetings .
a strong argument against doing away with referees is the problem of sifting through the chaff .
the volume of research documents published everyday is overwhelming and accelerating , perhaps not so much in statistics but certainly in biomedical research and engineering .
there is a maximum amount of time one can dedicate to looking at websites , blogs , twitter accounts , and such . and
blog comments have not always delivered the post - publication quality control that some had hoped .
commenting on a blog ( or , for that matter , a _
plos one _ article ) is not a well respected use of time , and there is also the paradox that a comment on a busy blog might not get noticed amidst all the other comments .
right now there seem to be very few blogs providing a useful communal review function ( and none of these blogs focus on statistics research ) .    even keeping track of new arxiv postings
sometimes gets overwhelming .
wasserman writes , `` if you do nt check arxiv for new papers every day , then you are really missing out , '' but our own experience is that it is almost impossible not to miss out . checking
the list of the newly posted arxiv in the morning indeed takes less than a minute , checking potentially interesting papers takes much longer !
even glancing at the 200 or 300 titles at the end of the month is taking its toll , and more if one starts looking at abstracts and pdf files .    without an organised system of reviews
, why should anyone bother to comment on poor or wrong , but not newsworthy , papers ?
the result could well be a clutter of mediocre results making it difficult for researchers who are not well - connected to navigate the field . we , the authors of the present article , know enough experts in our research areas that we can often get a quick evaluation of unpublished work . but a student at a middling university , or one whose advisor is not an expert on statistical computation , or a researcher in biology ( say ) who wants to use the latest computational methods , will not generally have the resources that we have in our social network .
the review process does not completely level the playing field
nothing could , given institutional disparities of resources  but it comes closer to equalising the information available to differently - equipped teams .    given the amount of chaff and the connected tendency to choke on it , filtering will be done  somehow or another .
getting rid of referees and journals in favour of repositories like ( the great ) arxiv would force us to rely on other and less well - defined sources for ranking , selecting , and eliminating papers . again
this would be subject to arbitrariness , subjectivity , bias , variation , randomness , peer pressure , and so on .
in addition , having no prior quality control makes reading a new paper a tremendous chore as one would have to check the references as well , leading to a type of infinite regress , while forcing one to rely on reputation and peer opinions .
in fact , one may wonder if it is really possible to go that far in reducing the impact of peer reviewing .
for many of us , so much depends on our publication record ( including jobs , promotions , grants , even salaries in certain institutions ) , that very few would be bold enough to stop sending papers to peer - reviewed journals from their own initiative . getting rid of peer - reviewed publications would make sense only if the vast majority of scientists in a given field would agree to do all at once . and
, since it is not only individuals but also scientific fields that compete for grant money , one could argue that a simultaneous move from _ all fields _ would be required to ditch peer reviewing , which is of course even less likely .
thus , despite the appeal of chucking the journals and starting over , we think a purely uncontrolled system would be even more unethical than what we currently have , and may be exactly what we would like to avoid in science and academia .
if our profession did start from scratch , we are sure that new institutions would arise to serve the filtering and reviewing functions , but we would prefer to make that switch more smoothly . in the following sections , we make two proposals that constitute a middle ground between what we have now and what wasserman is advocating .
the first is a further evolutionary step in the review process , as described above .
the second is a more radical change .
_ `` the trouble with referees is that they just do nt care which side wins . ''
_  tom canterbury .    in a world where everything ( or nearly everything ) is published  that is , the world we are currently approaching  how can the scientific community sift through the mass of results ?
it should be possible to use the effort that is currently going into the peer review system in a more efficient manner . while writing dozens of long careful referee reports per year , we realize the futility and effort of creating mini - articles for such a small audience ( the author and the journal editor ) .
it makes much more sense to switch to blogging about important papers or general points to reach a much wider audience . and to keep reviews short and to the point ( and available to the readers of the article in question at some point ; see below ) .
this notion seems to be met with reluctance by many of our interlocutors , for whom the secrecy of the reviewing process and the anonymity of the reviewer appear like sacrosanct principles . to wit ,
we note the emphasis placed on this secrecy in a recent update of the _ annals of statistics _ guidelines .    post - publication peer review could be done in different ways , most simply by adding a comment thread to each arxiv article ( with the caveat of being possibly not read ) , but more formal approaches are possible .
kriegeskorte ( 2009 , 2011 ) recommends  peer - to - peer editing : authors ask a senior scientist to edit the paper ; editor chooses 3 reviewers and asks them to openly review the paper ; editor is named on the paper . "
another , perhaps complementary , approach would be for groups of scholars and academic societies to manage a filtering service .
for example , instead of the american statistical association running _ jasa _ , _ jabes _ , _ jcgs _ , _ technometrics _ , etc . , and maintaining a separate editorial staff for each of those journals ( representing a huge amount of possibly overlapping and hence redundant volunteer service on the part of editors and referees ) , it could support filtering services .
the editors of each filter would be expected to scan the literature and handle submissions ( which in this case would be pointers to articles already published on the web ) .
the editorial boards would have the responsibility to come up with monthly ( say ) recommended reading material .
this would require some work , but less than the existing job of producing a journal .
the main concern we see would be to keep the editors focused on solid research rather than getting tabloid - like , but the latter seems less likely if the process involves simply flagging articles rather than formally and exclusively publishing them .
the flagging could even be multidimensional , with some papers tagged as potentially exciting but speculative , and others labelled as solid contributions within an existing paradigm .    instead of a simple thumbs - up or down , reviewers would have the task of situating each new paper within the literature .
as journal editors and frequent referees ourselves , we would appreciate the opportunity to prepare reviews that are directed outward to the potential users of the published articles rather than inward toward the author and editor .
we suspect that a key step in getting post - publication peer review to work is to transfer the efforts that _ would have gone into refereeing _ into filtering .
it would be difficult to start up a filter all on its own without the free labour that comes from referees ( who are in turn motivated by a sense of obligation and scientific community ) .
the hours in the day are limited , and we foresee a challenge in instilling the same sense of duty for filtering and post - publication review as is now present in the journal review process .
_  wheredya get ya ticket , ref  a cereal box ? ! " _    just as it s useful to ask why sport referees do nt always get it right , we could ask the same of our own reviewers . what s broken in our system ? as with sport , there is a constant proliferation of new arenas of training and competition and an exponential growth in the community of participants , which has great potential benefits for science but is daunting for reviewers .
however , unlike sport , we do nt grow or train our reviewers . unlike the training in refereeing that we may have received as a young person , learning to blow a whistle beside an old hand and watching how others did it , our training in scientific reviewing consisted of three words : `` here ,
review this . '' instead of the floodlights and open scrutiny
to which we subject our sports referees , scientific reviews are conducted by fluorescent light behind closed doors . perhaps it s time we came out .    as scientists
, we continually strive to improve , to find new solutions .
publishing our results is our lifeblood , so if we do nt like what we have at present , we should change it . however , as scientists , we should review the present system before discarding it altogether .
the referee process is already evolving ; so we could ask what we could we do to accelerate this .    it would help to have open , ongoing peer learning .
sports referees receive continuing training ; many professional societies demand this ; we should as well .
and we are not solely talking correspondence courses ; we mean the real thing , with floodlights and open scrutiny .
this leads to perhaps our most dramatic suggestion : a reviewer commons , namely a ( virtual ) repository for the placement of scientific reviews , open to all .
the advantages of such a commons are many .
it would encourage high quality , fair and useful reviews .
it would facilitate acknowledgement of reviewer contributions , benefiting both the journals and authors ( since reviews can be referenced in the manuscript ) and the reviewers ( since reviews can be referenced in a c.v .  and accessed by peers ) .
reviewers would then write not only for the authors but also for the readers , turning their comments and suggestions into a valuable discussion at the end of the reviewing process , to be added to their publication list as well . furthermore , as well as improving quality , this notion of a commons might also help to reduce the workload of reviewers and editors .
for example , until the current practice of not requiring authors to declare prior submissions of articles is revised , access to previous reviews might help to mitigate replication of effort by reviewers in dealing with manuscripts doing the rounds of journals and counting on the geometric distribution to achieve acceptance , sometime , somewhere .
we are not the first to argue that revealing the names of referees , not only to the authors , but also to the public , would deter referees from being complacent or un - constructively negative .
indeed , it may bring more explicit recognition in the scientific sense to referees and to their role in publishing better research , possibly all the way to referees reports becoming a valued part of their own publication record , as is already the case for referees for _ hydrology and earth system sciences_.    a related concern is the increasing focus of some journals on headline - grabbing articles .
this can lead to evaluation of articles on the basis of their popularity rather than their science . as discussed above , this is against the principles that we laid down for good refereeing practice .
psychologist sanjay srivastava ( 2011 ) identifies the problem :    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ `` as long as a journal pursues a strategy of publishing ` wow ' studies , it will inevitably contain more unreplicable findings and unsupportable conclusions than equally rigorous but more boring journals .
ground - breaking will always be higher - risk . and definitive will be the territory of journals that publish meta - analyses and reviews . ''
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _    naturally , not all journals are guilty of this charge , and some journals achieve a very strong balance of good story and high quality science .
moreover , switching to a publish - on - the - arxiv - model will not alone solve the problem : it would still be the most dramatic claims that get the most attention
. post - publication peer review could help , though .
again , parallels can be drawn between the concerns expressed above and similar concerns about trends in professional sport : increasing corporate control of the game , decreasing public access to the top games ( for example due to restrictive television rights ) , perception of players as commodities , a focus on spectator entertainment instead of quality of the game , and so on . while these struggles are widely acknowledged and debated , by
far the majority of the discussion is about improving the system rather than abandoning it altogether .
_  three cheers for the ref ! " _    one cheer for quality , two for fairness , three for excellence .
just as backyard players aspire to higher levels of play , ( true ) scientists _ want _ to be reviewed .
we want our work to be high quality and accepted by our peers , and we accept refereeing  and journals  as part of this evaluation excellence .
this does not mean that we must accept poor practice , in terms of quality or ethics , among referees or publishers . nor does it mean that having found such faults , we should abolish the system .
indeed , for the self - same reasons of ethics and quality , it is likely that even if we did away with scientific refereeing , if we opted instead for a web - free - for - all , sooner rather than later a system for identifying excellence and equity would emerge .
so instead of evicting , let s try evolving .
like any good complex system , improvements such as the establishment of a commons or of society supported post - publication peer review might exhibit similar self - organisation whereby a more satisfactory process of scientific review evolves of its own accord  or then again , it might equally implode .
finally , we have not addressed the problems of lack of replicability raised by empirical researchers in social science and medicine and which also arise in applied and computational statistics ( where , for example , a published simulation study can be based on a complex tangle of code that can sometimes not be reproduced by the original researchers , let alone outsiders ) . just as biomedical journals are moving toward registration of protocols and data , statistics researchers
might soon be expected to produce replicable papers with r / matlab / c code , data , and random seeds for all simulations .
perhaps replication will be part of the screening and vetting process : the aggregators that replace traditional journals can note a paper as having its replication materials or not , and perhaps the top aggregators ( for example , the replacements for _ jasa _ and _ jrss _ ) will require replicability to get their stamp of approval .
we thank in advance ( ! ) the referees of this article for their useful comments , the journal editors for accepting the manuscript , and the publisher for staying in business in order to publish our work .
the first ( nc ) and last authors ( cpr ) research is partly supported by the agence nationale de la recherche ( anr , 212 , rue de bercy 75012 paris ) through the 20082012 grant anr-08-blan-0237 `` bigmc . ''
the second author ( ag ) thanks the u.s . national science foundation for partial support of this work . the third author ( km ) acknowledges the generosity of ceremade and crest as hosts while this article was written .
kriegeskorte , n. 2012 .
open evaluation ( oe ) : post - publication peer review and rating .
parliamentary session 2010 - 2011 . written evidence ( pr 14 ) 28 february 2011 http://www.publications.parliament.uk/pa/cm201011/cmselect/cmsctech/writev/856/m14.htm