host - associated microbial communities are known to be of tremendous importance for host fitness , improving nutrient uptake , training the immune system , and resisting invasion by pathogens ( see , for example , brestoff & artis ,  2013 ; kamada et al . ,  2013 ; fredricks ( ed . ) ,  2013 ) .
our understanding of these communities , however , remains remarkably poor .
the origin , maintenance , and importance of community diversity ( fierer et al . ,  2011 ) , the factors determining community stability and resilience ( shade et al . ,  2012 ) , and the mechanisms of community assembly ( costello et al . ,  2012 ) are only some of the questions driving this rapidly expanding field .
although most microorganisms can not be cultured in a laboratory setting , advances in genome - sequencing technology now allow organisms to be probed in their natural environments .
in particular , the 16s rrna tag - sequencing approach identifies community members using fragments of dna from the hypervariable regions of the ribosomal 16s gene .
the development of this technique and the decreasing cost of high - throughput sequencing have prompted a large number of tag - sequencing experiments , including such large - scale efforts as the human microbiome project or the earth microbiome project .
the amount of collected data is growing exponentially .
however , our ability to interpret this data still has important limitations .
the _ de facto _ standard approach to 16s data analysis begins by clustering reads by sequence similarity into `` operational taxonomic units '' ( otus ) ; see fig .
1a ( quince et al . ,  2009 ;
kunin et al . ,  2010 ; huse et al . ,
a variety of clustering techniques have been developed and are widely used in popular software tools or packages ( hunt et al .
,  2008 ; schloss et al . ,  2009 ; edgar ,  2010 ; huang et al . ,  2010 ; edgar et al . ,  2011 ; quince et al . ,  2011 ; schloss et al . ,  2011 ; sul et al . ,  2011 ; caporaso et al . ,  2012 ; zheng et al . ,  2012 ; morgan et al . ,  2013 ; youngblut et al . ,
2013 ) . despite significant progress in the development of such software , all clustering - based approaches suffer from a major shortcoming ( prosser et al .
,  2007 ; hamady & knight ,  2009 ; schloss & westcott ,  2011 ) .
although an otu is a useful concept for coarse - graining sequencing data , its definition is not biologically motivated , but as its name acknowledges is purely operational .
sequences assigned to a particular otu are generally presumed to be close phylogenetic relatives and therefore likely to derive from ecologically similar bacterial subpopulations . however , the assumption that 16s sequence similarity is a good proxy for ecological similarity is notoriously problematic ( prosser et al .
,  2007 ; preheim et al . ,
moreover , otu assignments are not definitive but depend on both the clustering algorithm and the random seed chosen ( schloss & westcott ,  2011 ) .
several approaches have been proposed to improve the resolution of 16s data analysis beyond the standard 97%-similarity otus .
denoising algorithms exploit the predictable structure of certain error types to attempt to reassign or eliminate noisy reads ( huse et al . ,  2010 ; quince et al . ,  2011 ; rosen et al . ,
these algorithms are widely used for identifying low - abundance ( `` rare '' ) species against a noisy background , often with the aim of improving estimates of ecological diversity .
these objectives , however , remain very challenging due to issues that no denoiser can fully address .
any error model is necessarily approximate , and no denoising algorithm can deal with errors that are not adequately described by its error model ; when calling low - abundance species this issue becomes particularly problematic .
an alternative approach termed distribution - based clustering ( dbc ; preheim et al . ,
2013 ) aims to circumvent the limitations of conventional denoisers by using cross - sample comparisons , i.e.  supplementing sequence information by ecological information ( distribution of abundance across multiple biological samples ) .
however , dbc as an otu clustering algorithm also has important limitations : for low - count sequences , cross - sample comparisons necessarily become unreliable , and the execution time is prohibitively long even for moderately - sized datasets .    here , we build on the above methods to address a distinct question . rather than trying to further improve the existing approaches to otu clustering and rare species identification , we combine error - model based denoising and systematic cross - sample comparisons to resolve the fine ( sub - otu ) structure of moderate - to - high abundance community members in 16s illumina data .
importantly , our method does not rely on clustering similar sequences together . in this regard ,
our method is similar to oligotyping ( eren et al . ,  2013 ) , but our approach does not require manual supervision and applies to an entire community rather than an isolated otu .
using published data from a longitudinal study where the tongue community of two human individuals was sampled almost daily for several months ( caporaso et al . ,  2011 )
, we demonstrate that sequence similarity is a very poor predictor of ecological similarity , which we quantify for two bacteria as the correlation of their abundance time traces ( `` dynamical similarity '' ) .
thus , most clustering - based approaches would erroneously group together bacterial subpopulations of high ecological diversity for this data set .
however , a comparative analysis of the tongue communities of the two individuals also shows that when a pair of 16s tags is observed in both individuals , the dynamical similarity of the pair as measured independently in the two individuals is highly correlated .
this correlation falls off substantially when sequences differing by 1 nt out of 130 are compared . in other words ,
the exact sequence of the 16s tag carried by a bacterial subpopulation is predictive of its ecology , while even 99.2% similarity between tags of different subpopulations is generally not predictive of dynamical similarity , as defined above .
our results lend support to the recent idea that even a purely 16s - based study can provide insight into functional relatedness of community members ( cf .
picrust , langille et al . ,
2013 ) , while also exhibiting and beginning to quantify the limitations of such methods .
we demonstrate the applicability of our approach to a broad range of dataset types ( host - associated longitudinal ; environmental cross - sectional ; mock community ) , providing examples when highly similar sequences were found to exhibit ecologically significant distinctions .
finally , we discuss how the single - nucleotide sub - otu resolution of our method can provide new insights into factors shaping community assembly .
we used the raw data from a published long - term longitudinal sampling from four body sites ( gut : feces , right and left palm , and tongue ) of one male and one female individual ( caporaso et al . ,
2011 ) . in this study , the hypervariable region v4 of the bacterial 16s rrna gene was amplified and sequenced with illumina ga - iix . for details on collection and sequencing see the original reference ( caporaso et al . ,  2011 ) .
quality - filtered data published with that study is available at mg - rast:4457768.3 - 4459735.3 and is sufficient to reproduce our results using provided analysis scripts ( see supplementary information ( si ) ) . however , to investigate the performance of our filtering approach at different quality filtering settings , for this work we used the demultiplexed , but not quality - filtered fastq data , kindly provided to us by the study authors .
we split this data into per - sample fastq files using a custom matlab script ( mathworks , inc . ) and subjected it to minimal quality filtering using usearch v.7.0.1090 ( edgar ,  2010 ) , truncating reads at phred quality score 2 ( other thresholds were also evaluated ; see fig .
s4 ) , trimming to a fixed length of 130 nt and eliminating reads with ambiguous characters ( n ) .
in addition , we removed reads with expected number of base call errors exceeding 1 ( maxee parameter in usearch ) .
this criterion only eliminated 1% of trimmed reads .
notably , our approach does not rely on assumptions about a maximum number of errors in a read .
finally , to facilitate cross - sample comparisons , we compiled a library of all 1.4 m unique reads ever observed and a global table listing the abundances of each sequence across samples .
this was done using dereplication capabilities of usearch and a custom perl script ( * mergeseqs.pl * ) .
this script and others referenced in * bold * below are freely available at https://github.com/hepcat72/cff .
finally , the abundance table was normalized to @xmath0 total reads per sample , to correct for varying sample size .
read quality varied across lanes , so the number of reads after quality filtering was highest in a subset of tongue and fecal samples . in this work , we focused primarily on the tongue samples , as these come closest to probing the internal dynamics of a community living in a well - defined location on the body ; however , the analysis of fecal samples supports the same conclusions and is presented in fig .
tongue samples were distributed over two lanes .
the lane 6 samples from the male subject from day 65 onwards ( 314 consecutive samples covering a period of 355 days , @xmath1 reads in quality - filtered samples before normalization ) had approximately 4-fold more reads than those from the female subject and from days 1 - 64 of the male subject ( all on lane 5 ) .
consequently , the analysis below uses the data from the male subject from day 65 onwards , and , for the comparative analysis of the two individuals , also the 135 samples collected from the female subject .
the early samples from the male subject ( days 1 - 64 ) are only used for illustrative purposes ( fig .
3d ) .    to demonstrate the broad applicability of our method we also employed other published data ( figs .
s7 and s11 ) ; the data is described in the corresponding legends .
clustering can be a useful strategy to coarse - grain 16s data while also reducing noise , but if sequencing noise is low enough , such coarse - graining may not be necessary . at low noise , each community member
is predominantly represented by the same 16s sequence , surrounded by a cloud of low - abundance error sequences with the structure of the cloud determined by reproducible error rates .
prior work has described such error clouds in the data ( quince et al . ,  2009 ; edgar ,  2013 ) , and the assumption that high - abundance sequences are likely to be error - free is used in several rank - based denoising and chimera - checking algorithms ( slp , perseus , uchime de novo , uparse , abundantotu ) .
the treatment of reads that are very similar to high - abundance sequences is different across existing algorithms .
for example , slp ( huse et al . ,  2010 ) would consider any read differing by a single nucleotide from a higher - abundance sequence ( its `` direct neighbor '' in sequence space ) as an error .
however , some of these reads may actually represent true community members ( preheim et al . ,  2013 ) .
a more nuanced treatment can accept a sequence as likely to be real if its observed abundance is highly unlikely to have arisen in error , given some assumptions about error rates .
this idea is at the foundation of error - model based denoising .
it was used in ampliconnoise ( quince et al . ,  2011 ) , and its recent implementation in dada ( rosen et al . ,  2012 ) makes dada , to our knowledge , the best denoiser currently available .
however , no error model is perfect , and for all denoisers , errors not explicitly described by their model are labeled as true sequences .
thus a denoising algorithm alone is insufficient for achieving sub - otu resolution : if two close sequences that would fall within a single otu are both identified as `` probably real '' , one of these could still be an error . in the context of a single sample , confidently resolving close sequences as independently real requires a different experimental technique ( faith et al . ,  2013 ) or a complete , high - quality reference database of all bacteria in the sample , which in practice is available only for mock communities .
it is possible to resolve this problem in the framework of standard 16s experiments through a comparison of multiple samples , either longitudinal or cross - sectional ( preheim et al . ,  2013 ) . as an example , fig .
1b shows the abundances of the 10 highest - abundance direct neighbors of the overall top sequence of the tongue community , seq .
# 1 , for a representative set of 100 consecutive samples .
we see that three specific direct neighbors are strongly and consistently overrepresented compared to the other neighboring sequences and , more importantly , exhibit a dynamical behavior of their own ( consider , for example , the 3rd most abundant neighbor ) .
this has a clear interpretation ( fig .
1c ) : these three sequences must belong to other , fairly abundant bacterial subpopulations , possibly related to seq .
# 1 , but distinct and with their own dynamics .    to achieve sub - otu resolution ,
we adopt precisely this strategy , namely a cross - sample correlation analysis of individually denoised samples .
which denoiser should we use ?
dada would be an excellent option ; however , its estimated execution time on the tongue dataset used here is @xmath2 sec ( see si ) .
this is largely due to its exact treatment of probabilities , critically important for the processing of sequences with an abundance of just a few counts .
however , for such sequences the imperfections of the error model become non - negligible and can not be controlled , since cross - sample comparisons are interpretable only for sequences with sufficient abundance .
we therefore designed a new , simplified denoiser .
our algorithm , described below , takes two orders of magnitude less time to execute , yet for sequences of moderate abundance considered here achieves performance equal to dada , as demonstrated using mock community data ( sup .
table s2 ) .      for 16s data obtained using the illumina platform , the main sources of errors are pcr substitutions , pcr chimeras , and substitution errors due to illumina base call errors . of these ,
the substitution errors are responsible for generating the largest number of unique sequences ( fig .
s2 ; see also edgar ,  2013 ) and have the most predictable structure : their rates can be estimated directly from the data .
to do so , we considered the error clouds around the top 10 sequences by overall abundance ( in all tongue samples combined ) . assuming that most of these sequences are in fact errors
, we determined the rates of specific one - nucleotide substitutions ( * errorrates.pl * with z - score threshold of 2 ; see si ) .
these inferred rates were consistent across error clouds observed in the data ( fig .
s3 ) , with the average error rate of only 0.10% per nucleotide ( sup .
table 1 ; compare with quince et al .
,  2011 , table 2 ) .
we then used these error rates to predict the expected abundance of any given sequence if its presence were entirely due to independently generated sequencing errors of its more abundant neighbors ( the `` null model '' ; fig .
s5 ; * nzeros.pl * ) .
sequences whose abundance exceeded a threshold of 10 counts and the null - model prediction by at least 10-fold ( very conservative filtering parameters ) , were marked as `` candidates '' ; their presence can not be explained as an error within a substitution - only error model ( * getcandidates.pl * ) .
candidate sequences include true biological 16s sequences , but also sequences that arose through a different type of error , most notably pcr chimeras .
chimeric sequences were identified using uchime denovo ( edgar ,  2011 ) on the pooled data from all samples .
most such sequences were already eliminated by the abundance threshold requirement : if we relax the abundance threshold to 2 ( excluding singletons only ) , we find that the chimeras detected by uchime , when present in a sample , have abundance under 10 counts in 95% of cases .
however , chimeras of highly abundant parents reproducibly occur at higher abundances ( haas et al ,  2011 ) and are filtered at this step .
candidate sequences that remained after filtering chimeras were labeled `` real '' .
our highly conservative filtering criteria allow us to assume that this list contains only true biological sequences , i.e.  that there are no false positives ( cf .
table s2 ) , except possibly those due to some exceptionally frequent errors not described by our error model ( see si ) .
this stringency comes at the expense of low - abundance false negatives ( true biological sequences labeled as `` possible noise '' ) .
our strategy is to retain all sequences marked `` real '' in 2 or more samples ( out of 507 ; * getreals.pl * ) .
this makes our denoiser specifically adapted to multi - sample analysis : in each sample , only high - confidence detections are identified , which is very fast , and then a liberal criterion applied across samples retains all sequences that ever generated a high - confidence detection , except sample singletons .
in particular , we stress that our detection threshold of 10 counts is not equivalent to removing all sequences with abundance below 10 ; the only sequences excluded from consideration are those than never rise to 10 counts in the entire set of 504 tongue samples , or do so only once . for such sequences ,
the measured counts are dominated by detection and counting noise .    in the interest of speed , and
to ensure the robustness of reported sequence - abundance values with respect to the details of the error model , we did not attempt to remap noisy reads to their most probable source .
our approach relies on the accuracy of measurement of relative abundances of true sequences .
the error remapping process modifies sequence counts in a way that depends on the assumptions of the error model , distorting the relative abundance values whenever neighboring sequences are incorrectly classified as `` reals '' or `` errors '' .
in contrast , discarding noisy reads leaves the relative abundances intact , as long as the probability of making zero errors is approximately constant across all sequences .
this assumption is much weaker than adopting a particular error model .
we estimate the zero - error probability at @xmath3% ( see si ) ; in other words , discarding noisy reads leads only to a @xmath4% loss of sequencing depth .
if read remapping is desired , the analysis described below can be applied to dada denoiser output .    since non - identical reads are never clustered together , ours is a single - nucleotide resolution approach .
the complete workflow of cluster - free filtering is outlined in fig .
s6 and detailed in the si .
the code is freely available at https://github.com/hepcat72/cff .
the starting point for our analysis is a global sequence abundance table listing the abundances of each unique 16s sequence across samples .
we retained the 307 sequences that passed the multi - sample filtering algorithm described in methods , and thus putatively belong to bacteria present in the population at least part of the time .
we denote these sequences by their overall abundance rank : seq .
# 1 ,  # 2 , etc . in this list ,
184 pairs of sequences were direct neighbors in sequence space ( hamming distance 1 ) .
these pairs had 99.2% sequence similarity but were resolved by our criteria as independently present in the community .
the population of bacteria sharing the exact same sequenced fragment of the 16s gene ( at 100% identity ) is the smallest taxonomic unit resolvable by 16s analysis . for notational convenience
, throughout this work we call it the `` subpopulation '' identified by a sequence .      in the standard approach to tag - sequencing data , it is assumed that sequence similarity of 16s hypervariable regions can be used as a proxy for phylogenetic , and therefore ecological relatedness .
our new filtering method , applied to time - series data , allows us to bypass this assumption and assess ecological relatedness independently , based on the similarity of time traces , since each distinct subpopulation will respond in its own way to variation in environmental conditions ( youngblut et al . ,   2013 ) , causing the abundance time traces to be more or less correlated ( or possibly anticorrelated ; see fig .
2a - c illustrates this by showing time traces ( normalized counts versus observation day ) for three examples of sequence pairs .
we find that sequences differing by as little as 1 nucleotide ( 99.2% similarity ) can be ecologically distinct as evidenced by their very different time series ( fig .
2a ) ; see also vandewalle et al .
,  2012 . for comparison , fig .
2b shows another pair of sequences , also with 99.2% sequence similarity but whose abundance time traces appear indistinguishable .
the remarkable correlation between these two traces provides an internal control and demonstrates that the much lower correlation of traces in fig .
2a can not be explained by measurement error but reflects a true ecological difference .
note that the abundances of the two sequences shown in fig .
2b are not equal , but occur with a highly stable ratio .
this could reflect a stable difference in abundance of the bacteria they represent , but is more likely caused by differential amplification efficiency of these sequences by the pca primers ( turnbaugh et al .
,  2010 ; klindworth et al . ,
2013 ) and/or a different number of genomic 16s copies per cell ( tourova ,  2003 ) .
panels a - b show that sequence similarity need not imply ecological similarity .
finally , fig .
2c illustrates that the converse is also true : sequences exhibiting identical time - dependence may have as little as 81% sequence identity .    to quantify the generality of these examples
, it is useful to define a measure of the ecological similarity of the bacterial subpopulations represented by two sequences . a natural candidate metric is the pearson correlation of the measured abundance traces .
note , however , that the maximum correlation one can expect between the time traces of two sequences depends on their abundance : for low - abundance sequences poisson sampling noise becomes non - negligible and sets an upper bound on the correlation coefficient .
we therefore define the `` dynamical similarity '' of two traces as the pearson correlation of their abundance , normalized by their maximum possible correlation @xmath5 , computed as the correlation of the higher - abundance time trace with a poisson - downsampled version of itself ( see si ) . for sequence distance
, we use the hamming distance between sequences after pairwise alignment ( see si ) . with these definitions ,
we can present a 2d histogram of dynamical similarity vs. distance in sequence space for all sequence pairs constructed from the top 200 real sequences ( fig .
as expected , most sequence pairs exhibit no significant dynamical similarity and are also far apart in sequence space , but a subset of closely similar sequences appears to display some degree of anticorrelation between the two measures .
zooming in on this region ( fig .
2e ) makes this anticorrelation more apparent ; however , even when restricted to the subset shown in fig .
2e , the correlation coefficient remains weak ( @xmath6 ) .
sequences separated by up to 6 - 7 nt ( 95% sequence similarity ) tend to be dynamically similar , the effect increasing for smaller distances , but this general trend is very loose and is not a reliable predictor of similarity for any particular pair .
this result was not unexpected , and is frequently used in arguments against over - reliance on the 16s gene sequence ( see , for example , prosser et al . ,  2007 ) , in favor of methods providing functional information , such as shotgun metagenomics .
the novelty of fig .
2e lies in the fact that it was obtained entirely within the framework of 16s tag sequencing methodology .
as explained in the previous section , 16s tags with low dynamical similarity clearly derive from distinct bacterial subpopulations , even if the sequences are themselves highly similar .
we now consider pairs of sequences with highly correlated time traces such as observed in fig .  2b ,
c. such correlated pairs could derive from the same bacterial cells ( as multiple genomic copies of the 16s gene , or as exceptionally common pcr errors not included in our model ) .
alternatively , they could derive from distinct bacterial subpopulations that either occupy the same ecological niche or engage in a strong obligate symbiosis .
such pairs are thus of significant ecological interest , provided it can be shown that the sequences actually derive from different bacterial cells . in this section ,
we demonstrate that cross - sample correlation analysis can , in some cases , successfully make this subtle distinction between same - cell or different - cell sources .    to draw this distinction ,
we make use of the following observation . the abundance ratio of two sequences that derive from the same bacterium is set by some sample - independent parameter ( e.g.  involving differential amplification efficiency , 16s copy number , and/or pcr error rate ) ; therefore , any fluctuation in their abundance ratio is due to measurement noise , and must be uncorrelated between samples . any statistically significant time ( or location ; see si ) correlation of abundance ratio fluctuations , e.g.  in consecutive ( or proximate ) samples , is therefore strong evidence that the two sequences are at least partially contributed by physically distinct subpopulations .
for this approach to succeed , the dynamics of individual subpopulations must be slow enough to allow correlations between consecutive samples to be observed .
we therefore began by computing , for each of the top 100 sequences , the autocorrelation function @xmath7 , defined as the correlation between abundance fluctuations in samples separated by @xmath8 time points , and normalized so that @xmath9 ( for simplicity , we treat samples as though they were equally spaced in time , which is approximately correct ; the mean separation between samples was 1.1 days ) .
the environment experienced by tongue microorganisms changes frequently , and one might have expected that daily sampling would probe the space of possible community states , but provide little information about community dynamics as these would occur on a faster time scale .
surprisingly , we found the time dependence of most sequences in the top 100 to have a significant autocorrelation despite the relatively low sampling rate ( fig .
although conditions on the tongue make fast abundance changes possible , as evidenced by the large , rapid fluctuations in fig .
2a - c , we found the correlation time for the top 100 sequences to be surprisingly long , typically 2 - 4 days but often longer ( fig .
3b ) , sometimes exceeding a month ( fig .
these multi - day autocorrelations make it plausible that for physically distinct subpopulations , the fluctuations of their abundances relative to each other could be slow enough to be detectable even if their ecology is similar .
consider two sequences @xmath10 and @xmath11 whose abundance time traces are highly correlated .
denote by @xmath12 , @xmath13 the two traces renormalized to the same mean for best overlap , as in fig .
2b , c , and let @xmath14 be their fractional difference in a given sample ( a quantity more robust to noise than the nave abundance ratio ) : @xmath15    if @xmath16 reflects abundances of two distinct subpopulations , then @xmath14 can be expected to exhibit an autocorrelation on par with that observed for the individual sequences .
intuitively , if on day 1 , subpopulation @xmath10 is , say , 10% more abundant than @xmath11 , and the dynamics of both are slow , then @xmath10 is likely to maintain its lead on day 2 .
in contrast , if the two sequences are genomic variants contained within the same bacterium , then any difference between @xmath12 and @xmath13 must be due to measurement noise , and @xmath14 will be uncorrelated between samples .
we therefore introduce the _ persistence of difference _ @xmath17 as the 1-day autocorrelation coefficient of @xmath14 : @xmath18 where angular brackets denote averaging over time .
@xmath17 characterizes the persistence of abundance fluctuations of two sequences relative to each other . for
sequences arising from the same cells , @xmath17 must vanish .
any pair of sequences exhibiting a statistically significant @xmath17 must be contributed , at least in part , by two physically distinct bacterial subpopulations .
note that the absolute abundance of a sequence may change dramatically between days ( e.g.  more favorable conditions can cause both subpopulations to proliferate quickly ) , but the normalization of @xmath14 makes @xmath17 insensitive to such overall correlated behavior .    summarizing the above
, we have the following expectation for @xmath17 : for a randomly chosen pair of sequences , with insignificant dynamical similarity , @xmath17 should be significantly non - zero ( due to the slow dynamics of the individual subpopulations ; see si ) , and form a unimodal distribution consistent with the null model of unrelated subpopulations .
in contrast , pairs displaying high dynamical similarity come in two types , and the persistence of difference @xmath17 should display a bimodal distribution : pairs of sequences found within the same bacterial cell will have vanishing or insignificant @xmath17 , while pairs belonging to distinct subpopulations will likely exhibit a persistence of difference comparable with the null model prediction .
this is precisely what we observe .
3c shows , for all sequence pairs constructed from the top 100 sequences , a scatter plot of their persistence of difference @xmath17 versus dynamical similarity as defined previously ( the normalized pearson correlation of their abundances ) .
the mean and standard deviations of the distribution predicted by the null model ( unrelated subpopulations ) are indicated by the green ellipse , and were computed directly from the data by reversing in all pairs the time order for one of the sequences .
the mean and standard deviations of the actual data are indicated by the red cross .
we find , as expected , that the @xmath17 score of dynamically dissimilar sequence pairs is unimodal and consistent with the null - model prediction .
in contrast , the @xmath17 score of dynamically similar pairs exhibits the predicted bimodality ( right side of the plot ) , with a subset exhibiting weak or negligible persistence of difference ( bottom right ) .
as explained above , we interpret these low-@xmath17 pairs as corresponding to genomic 16s variants found within a single bacterium .
letters a - c identify pairs shown on fig .
note that the strong persistence of difference identifies the pair `` b '' as being contributed , at least in part , by distinct bacterial cells , despite 99.2% sequence similarity and an almost perfect correlation of abundances ( fig .
conversely , the low-@xmath17 pair `` c '' ( with only 81% sequence similarity ) likely corresponds to an example of two dissimilar 16s genes contained within a single bacterium . note the enrichment of pairs with high sequence similarity among the dynamically similar pairs , as indicated by the color code ( compare with fig .
remarkably , in the case of pair `` b '' , the conclusion of distinct bacterial subpopulations drawn from fig .
3c can be confirmed directly .
panel d shows the time traces of this pair for days 1 - 64 ( normalization as in fig .
due to the relatively poor sequencing depth in these early samples , they were not included in fig .
the clear separation observed prior to day 40 provides an independent confirmation of our conclusion .
we stress that these data were not used in the analysis presented in fig .  3c , but the sensitivity of the autocorrelation method was sufficient to identify these sequences as deriving from physically distinct cells based solely on the data shown in fig .
the autocorrelation function of the fractional difference @xmath14 for this pair is shown in fig .
we have verified that the persistence of difference for this pair does not change significantly if any window of 100 consecutive samples is used instead of the full time series ( data not shown ) .      figs .  2a , s7 , s10 , and s11 provide examples of some fine features that standard otu - based methods would fail to detect , but which become accessible with cluster - free filtering .
we now ask whether such cases are the exception or the rule . for a given sequence similarity threshold ,
we can define , for each of the most abundant sequences , its would - be otu , namely the ensemble @xmath19 of all `` real '' sequences within the chosen similarity threshold .
we construct the time trace of the abundance of this otu as the sum of the abundances of all its members .
we can now ask : how representative is this time trace of the true behavior of the member sequences ? let @xmath20 be the correlation coefficients between time traces of individual members and the otu itself , normalized to the maximum expected correlation as before .
we define unweighted and weighted otu quality scores @xmath21 and @xmath22 as , respectively , the simple average of @xmath20 , and an average weighted by the abundance of the member : @xmath23    here @xmath24 is the number of subpopulations in the otu and @xmath25 is the average abundance of member @xmath26 .
the weighted quality score @xmath22 is always larger , because the most abundant sequence dominates the sum and so is better correlated with the otu trace .
thus @xmath22 tells us how representative the otu is of its most abundant member . the unweighted quality score @xmath21 tells us how diverse is the group of subpopulations lumped together into an otu .
if the sequences grouped into an otu are all dynamically identical ( are poisson - resampled versions of each other at different abundances ) , both quality scores will be close to 1 . if the otu is dominated by one subpopulation , with other members dynamically different but very low in abundance
, we will have @xmath27 , but @xmath28 .
finally , if the otu contains several dynamically distinct subpopulations at comparable abundances , both quality scores will be low .
the average quality scores for otus assembled around the top 5 sequences are presented in fig .  4 as a function of sequence similarity threshold .
the relatively high weighted quality score @xmath22 means that an otu time trace is , on average , fairly representative of its most abundant member .
the unweighted score @xmath21 is , however , dramatically lower , indicating that the otus group together sequences from subpopulations with high dynamical diversity .
these quality scores rely on abundance time - trace correlations , which become contaminated with noise for low - abundance sequences . for the purposes of fig .  4 , to apply these definitions conservatively , we therefore restricted our attention only to high - abundance members of the otu , considering only sequences from the top 200 by overall abundance .
further , our cluster - free filtering method also has finite resolution , as the sequences we analyze are only 130nt long and may derive from distinct 16s genes , implying some unresolved diversity .
this limited resolution leads to an artificial inflation of otu quality scores as the similarity threshold approaches 100% .
for both these reasons the true quality scores of otus are likely even lower ( see si ) .      the fact that tag sequence similarity within the 16s gene is only loosely correlated with dynamical similarity ( fig .
2e ) was not unexpected ( see , for example , prosser et al . ,  2007 and references therein ) .
at a neutral mutation rate of order 10 - 9 per base pair per generation ( ochman ,  2003 ) , an average difference of a single nucleotide out of 100 would already require divergence for millions of generations .
a more precise estimate of divergence time should take into account the possibility of horizontal gene transfer , whose rate in an ecologically relevant setting is hard to assess .
however , it is clear that , generically , two bacteria that differ by even 1 nt in a particular hypervariable region of the 16s gene likely diverged a long time ago .
these bacteria are likely to also differ elsewhere in their 16s gene , and to carry even more significant differences in functional parts of their genome .
in contrast , what if we consider two bacteria whose sequenced portions of their 16s genes are identical ?
since the length of the sequenced fragment is small ( typically @xmath29 nt ) and the mutation rate is low , these bacteria could still have diverged a very long time ago ( lukjancenko et al . ,  2010 ) . however , depending on circumstances , the actual time since the last common ancestor may be much shorter .
for example , consider two communities that frequently exchange members .
if two bacteria drawn from two such communities are 100% identical in their 16s tags , a likely explanation for this identity is a recent exchange event , in which case the entire genomes of these bacteria may be close to identical .
we conclude that in the presence of strain exchange between communities , exact sequence identity and near - identity may have fundamentally different implications .
the study of caporaso et al . sampled the tongue microbiota of two cohabiting individuals ( rob knight , personal communication ) , and so strain exchange is likely to be a highly significant factor ( song et al . ,  2013 ) .
we hypothesized , therefore , that these communities would share some non - negligible number of subpopulations at 100% sequence identity , and that these common subpopulations might have similar ecology in both communities .
we began by identifying the fraction of common 16s sequences in the list of the top n for each individual ( at 100% sequence identity ) . based on our strain exchange hypothesis
, we expected to find some matches , but were still surprised to find this fraction to be as high as 75% ( fig .
such a high proportion of perfect matches provides strong evidence that the identical sequences found in these two communities most likely diverged from a common ancestor more recently than any pair of close , but non - identical sequences within the same community .
the same conclusion is supported by the analysis of fecal samples from the two individuals ( fig .
we then considered the 73 sequences that were found among the top 100 of both individuals and asked whether the behavior of these subpopulations was predominantly shaped by their presumed common origin ( causing them to be similar ) or by local adaptation ( causing them to diverge while leaving the 16s region intact ; see lukjancenko et al . ,  2010 ) . to this end , for each pair of sequences @xmath30 drawn from this list , we measured their dynamical similarity independently in the two datasets ; @xmath31 for the male and @xmath32 for the female .
if the effect of local adaptation were dominant , then the exactness of a match of 16s sequences would not carry much information : the ecologies and genomes would be no more similar between 100%-identical partners in the two communities than between any other sequences within the same bacterial `` species '' ( otu ) ; this scenario is implicitly assumed by taxonomy - based methods .
alternatively , if the ecology were determined primarily by the shared recent ancestor , then identical 16s tag sequences in the two communities would correspond to bacterial subpopulations with almost identical genomes . in this scenario ,
provided local adaptation did not modify the ecology of a subpopulation significantly , @xmath31 and @xmath32 should be strongly correlated , and unlike the first scenario , this correlation would be noticeably degraded for any less than 100% sequence identity .
the latter is indeed what we observe ( fig .
5b demonstrates that subpopulations identified by the exact same 16s tags in the two individuals are dynamically similar ; see also fig .  s11d and s12 . to obtain fig .
5c , we constructed an `` inexact pairing '' of sequences between individuals , whereupon each sequence from the top 100 in the female individual was matched to the highest - abundance sequence from the top 100 in the male individual that differed from it by exactly 1 nucleotide , when such a match existed .
this matching corresponds to 99.2% sequence identity , yet already substantially degrades the correlation between @xmath31 and @xmath32 ( fig .
we conclude that 100% identity of tag sequences has qualitatively different implications from even 99.2% near - identity .
in this work , we have demonstrated that cross - sample correlation analysis of denoised 16s data can be exploited to achieve sub - otu resolution .
the cluster - free filtering approach we presented reliably identified up to 20 distinct subpopulations within standard 97% similarity otus , and a comparative analysis of oral communities of two cohabiting individuals demonstrates that most such subpopulations are shared between the two communities .
furthermore , subpopulations identified by the exact same 16s tags in the two individuals are dynamically similar , whereas even a single nucleotide mismatch is enough to degrade this similarity .
overall , our analysis shows that coarse - graining sequence data into otus is not essential for ecological applications of 16s tag sequencing methodology .
our approach combines two novelties .
first and foremost , we do not cluster similar sequences together .
regrettably , in the literature the term `` clustering '' has multiple meanings .
most denoising algorithms aim to assign erroneous reads to their most likely source , to make the abundance estimates of true sequences more accurate .
the same term `` clustering '' is used both for this read remapping and for merging multiple true sequences into a single otu . however , these two practices are fundamentally different .
read remapping constitutes data denoising ; as such , it is always advantageous , can be done in a principled way , and can be evaluated against an objective standard of performance . adding it to our approach would likely somewhat improve the results . in contrast , otu clustering is a form of data coarse - graining , and the optimal degree of coarse - graining is necessarily application - dependent .
importantly , for some applications it may not be necessary or desirable . when studying coarse features of community composition and dynamics , e.g.  comparing communities across habitats ( costello et al
. ,  2009 , huttenhower et al .  2012 ) coarse - graining is appropriate .
for example , metrics of community comparison such as unifrac ( lozupone & knight ,  2005 ) are widely used precisely because , by construction , they are not sensitive to otu sub - structure . however , when studying subtle differences between broadly similar communities , e.g.  samples from similar habitats or repeated sampling of the same habitat , the sub - otu structure becomes a valuable source of insight .
this is the intended application for our approach .
although we focused on longitudinal illumina data , the denoising algorithm we developed does not assume short read length or low error rate and is directly applicable to a wide range of dataset types ( see examples in fig .
s7 and s11 ) , provided the error structure is consistent across samples ( preheim et al . ,
we expect our approach to be useful for investigating the structure and dynamics of discrete community subtypes such as those observed in the vaginal community ( huttenhower et al .
2012 ) .
our second novelty is to exploit the quantitative advantage offered by multi - sample ( time - course or cross - sectional ) data .
since the copy number of the 16s gene carried by a bacterium is typically unknown ( tourova ,  2003 ) , and the pcr amplification bias among different 16s fragments can sometimes reach orders of magnitude ( turnbaugh et al .
,  2010 ; klindworth et al . ,
2013 ) , the 16s data from a single sample carries very little quantitative information about community composition .
in contrast , the ratios of sequence abundance are highly informative and can be measured very precisely , as demonstrated in fig .
recently , time - course data collection has been gaining in popularity , as it was recognized that such experiments can offer valuable insight into community dynamics ( shade et al . ,  2013 and references therein ) .
however , another major advantage of such datasets , namely that changes in sequence abundance ratios can be measured much more accurately than absolute abundances , is only beginning to be explored . for us , time - series data provides a context where sub - otu resolution acquires its full power .
specifically , we have shown that cross - sample comparisons enable us to decouple sequence similarity from dynamical similarity while remaining fully within the framework of 16s tag sequencing .
high - quality reference databases can complement our approach to facilitate paralog identification .
the basic methodology described here should also be extendable to other marker genes .
the new approach described in this work is not a replacement for otu clustering ; it discards low - abundance sequences and so is unsuitable for studies of population - level alpha- or beta - diversity .
however , the novel statistical and computational techniques we present allow full utilization of the quantitative information carried by sequences with a moderate - to - high abundance . this has promising applications for the study of factors affecting community assembly .
as discussed above , sub - otu resolution can provide insight into the prevalence of strain exchange between communities , invasion / extinction dynamics of otu subpopulations , and the time scale of ecological divergence relative to sequence divergence .
in addition , the dynamics of individual - specific subpopulations could help characterize the role of host genetics or the host immune system on shaping the community , particularly in the context of highly controlled experiments with germ - free animals .
we thank w bialek , af bitbol , cp broedersz , ds fisher , r knight and members of his lab , sa levin , y meir , sw pacala , sp preheim and mj rosen for helpful discussions , g caporaso for providing the data , and r edgar for usearch support .
this work was partially supported under the darpa biochronicity program , grant d12ap00025 , and national science foundation grants phy-1305525 and ccf-0939370 .
99 biblabel#1 @bibitem#1@bibitem#1-    brestoff jr , artis d ( 2013 ) .
commensal bacteria at the interface of host metabolism and the immune system .
_ nat immunol _
* 14 * : 676 - 684 .
caporaso jg , lauber cl , costello ek , berg - lyons d , gonzalez a , stombaugh j et al ( 2011 ) .
moving pictures of the human microbiome . _
genome biology _ * 12 * : r50 .
caporaso jg , lauber cl , walters wa , berg - lyons d , huntley j , fierer n et al ( 2012 ) .
ultra - high - throughput microbial community analysis on the illumina hiseq and miseq platforms .
_ isme j _ * 6 * : 1621 - 1624 .
costello ek , lauber cl , hamady m , fierer n , gordon ji , knight r ( 2009 ) .
bacterial community variation in human body habitats across space and time .
_ science _ * 326 * : 1694 - 1697 .
costello ek , stagaman k , dethlefsen l , bohannan bjm , relman da ( 2012 ) .
the application of ecological theory toward an understanding of the human microbiome .
_ science _ * 336 * : 1255 - 1262 .
desantis tz , hugenholtz p , larsen n , rojas m , brodie el , keller k et al ( 2006 ) .
greengenes , a chimera - checked 16s rrna gene database and workbench compatible with arb .
_ appl environ microbiol _ * 72 * : 5069 - 5072 .
edgar rc ( 2004 ) .
muscle : multiple sequence alignment with high accuracy and high throughput .
_ nucleic acids res .
_ * 32 * : 1792 - 1797 edgar rc ( 2010 ) .
search and clustering orders of magnitude faster than blast .
_ bioinformatics _ * 26 * : 2460 - 2461 .
edgar rc , haas bj , clemente jc , quince c , knight r ( 2011 ) .
uchime improves sensitivity and speed of chimera detection .
_ bioinformatics _ * 27 * : 2194 - 2200 .
eren am , maignien l , sul wj , murphy lg , grim sl , morrison hg et al ( 2013 ) . oligotyping : differentiating between closely related microbial taxa using 16s rrna gene data . _
methods in ecology and evolution _ * 4 * : 11111119 .
faith jj , guruge jl , charbonneau m , subramanian s , seedorf h , goodman al et al ( 2013 ) .
the long - term stability of the human gut microbiota .
_ science _ * 341 * : 1237439 .
fierer n , lennon jt ( 2011 ) . the generation and maintenance of diversity in microbial communities .
_ am j bot _ * 98 * : 439 - 448 .
fredricks dn ( 2013 ) .
_ the human microbiota : how microbial communities affect health and disease . _ wiley - blackwell : hoboken , n.j .
haas bj , gevers d , earl am , feldgarden m , ward dv , giannoukos g et al ( 2011 ) .
chimeric 16s rrna sequence formation and detection in sanger and 454-pyrosequenced pcr amplicons .
_ genome res _ * 3 * : 494 - 504 .
hamady m , knight r ( 2009 ) .
microbial community profiling for human microbiome projects : tools , techniques , and challenges . _ genome res _ * 19 * : 1141 - 1152 .
huang y , niu bf , gao y , fu lm , li wz ( 2010 ) .
cd - hit suite : a web server for clustering and comparing biological sequences . _ bioinformatics _ * 26 * : 680 - 682 .
hunt de , david la , gevers d , preheim sp , alm ej , polz mf ( 2008 ) .
resource partitioning and sympatric differentiation among closely related bacterioplankton .
_ science _ * 320 * : 1081 - 1085 .
huse sm , welch dm , morrison hg , sogin ml ( 2010 ) .
ironing out the wrinkles in the rare biosphere through improved otu clustering .
_ environ microbiol _ * 12 * : 1889 - 1898 .
huttenhower c , gevers d , knight r , abubucker s , badger jh , chinwalla at et al ( 2012 ) .
structure , function and diversity of the healthy human microbiome .
_ nature _ * 486 * : 207 - 214 .
kamada n , chen gy , inohara n , nunez g ( 2013 ) .
control of pathogens and pathobionts by the gut microbiota .
_ nat immunol _
* 14 * : 685 - 690 .
klindworth a , pruesse e , schweer t , peplies j , quast c , horn m et al ( 2013 ) .
evaluation of general 16s ribosomal rna gene pcr primers for classical and next - generation sequencing - based diversity studies .
_ nucleic acids res _ * 41*. kunin v , engelbrektson a , ochman h , hugenholtz p ( 2010 ) .
wrinkles in the rare biosphere : pyrosequencing errors can lead to artificial inflation of diversity estimates . _ environ microbiol _ * 12 * : 118 - 123 .
langille mgi , zaneveld j , caporaso jg , mcdonald d , knights d , reyes ja et al .
( 2013 ) predictive functional profiling of microbial communities using 16s rrna marker gene sequences .
_ nature biotechnol _ * 31 * : 814821 .
lozupone c , knight r ( 2005 ) .
unifrac : a new phylogenetic method for comparing microbial communities .
_ appl environ microbiol _ * 71 * : 8228 - 8235 .
lukjancenko o , wassenaar tm , ussery dw ( 2010 ) .
comparison of 61 sequenced escherichia coli genomes .
_ microbial ecology _ * 60 * : 708 - 720 .
morgan mj , chariton aa , hartley dm , court ln , hardy cm ( 2013 ) .
improved inference of taxonomic richness from environmental dna .
_ plos one _ * 8 * : e71974 .
ochman h ( 2003 ) .
neutral mutations and neutral substitutions in bacterial genomes .
_ mol biol evol _ * 20 * : 2091 - 2096 .
preheim sp , perrotta ar , martin - platero am , gupta a , alm ej ( 2013 ) .
distribution - based clustering : using ecology to refine the operational taxonomic unit .
_ appl environ microbiol _ * 79 * : 6593 - 6603 .
prosser ji , bohannan bjm , curtis tp , ellis rj , firestone mk , freckleton rp et al ( 2007 ) .
essay  the role of ecological theory in microbial ecology . _
nature reviews microbiology _ * 5 * : 384 - 392
. quince c , lanzen a , curtis tp , davenport rj , hall n ,
head i m et al ( 2009 ) .
accurate determination of microbial diversity from 454 pyrosequencing data .
_ nat methods _ * 6 * : 639-u627 .
quince c , lanzen a , davenport rj , turnbaugh pj ( 2011 ) . removing noise from pyrosequenced amplicons .
_ bmc bioinformatics _ * 12 * : 38 .
rosen mj , callahan bj , fisher ds , holmes sp ( 2012 ) .
denoising pcr - amplified metagenome data .
_ bmc bioinformatics _ * 13 * : 283 .
schloss pd , westcott sl , ryabin t , hall jr , hartmann m , hollister eb et al ( 2009 ) .
introducing mothur : open - source , platform - independent , community - supported software for describing and comparing microbial communities .
_ appl environ microbiol _ * 75 * : 7537 - 7541 .
schloss pd , gevers d , westcott sl ( 2011 ) .
reducing the effects of pcr amplification and sequencing artifacts on 16s rrna - based studies .
_ plos one _ * 6*. schloss pd , westcott sl ( 2011 ) . assessing and improving methods used in operational taxonomic unit - based approaches for 16s rrna gene sequence analysis .
_ appl environ microbiol _ * 77 * : 3219 - 3226 .
shade a , peter h , allison sd , baho d , berga m , buergmann h et al ( 2012 ) . fundamentals of microbial community resistance and resilience . _ frontiers in microbiology _ * 3*. shade a , caporaso jg , handelsman j , knight r , fierer n ( 2013 ) . a meta - analysis of changes in bacterial and archaeal communities with time .
_ isme j _ * 7 * : 1493 - 1506 .
song sj , lauber c , costello ek , lozupone ca , humphrey g , berg - lyons d et al ( 2013 ) .
cohabiting family members share microbiota with one another and with their dogs
. _ elife _ * 2*. sul wj , cole jr , jesus ed , wang q , farris rj , fish ja et al ( 2011 ) .
bacterial community comparisons by taxonomy - supervised analysis independent of sequence alignment and clustering .
_ proc natl acad sci _ * 108 * : 14637 - 14642 .
tourova tp ( 2003 ) .
copy number of ribosomal operons in prokaryotes and its effect on phylogenetic analyses .
_ microbiology _ * 72 * : 389 - 402 .
turnbaugh pj , quince c , faith jj , mchardy ac , yatsunenko t , niazi f et al ( 2010 ) .
organismal , genetic , and transcriptional variation in the deeply sequenced gut microbiomes of identical twins .
_ proc natl acad sci _ * 107 * : 7503 - 7508 .
vandewalle jl , goetz gw , huse sm , morrison hg , sogin ml , hoffmann rg et al ( 2012 ) .
acinetobacter , aeromonas and trichococcus populations dominate the microbial community within urban sewer infrastructure .
_ environ microbiol _ * 14 * : 2538 - 2552 .
youngblut nd , shade a , read js , mcmahon kd , whitaker rj ( 2013 ) .
lineage - specific responses of microbial communities to environmental change .
_ appl environ microbiol _ * 79 * : 39 - 47 . zhang z , schwartz s , wagner l , miller w ( 2000 ) . a greedy algorithm for aligning dna sequences , _ j comput biol _ * 7 * : 203 - 214 .
zheng zj , kramer s , schmidt b ( 2012 ) .
dysc : software for greedy clustering of 16s rrna reads .
_ bioinformatics _ * 28 * : 2182 - 2183 .
in this section , we illustrate the idea of error - model - based denoising ( see also the introduction in rosen et al . , 2012 ) and give a detailed description of the simple denoiser we designed for this work .
we then describe the workflow of an open source software package we created to implement this denoiser , and compare its performance on mock community data with dada ( rosen et al . , 2012 ) .
finally , to illustrate that our cluster - free filtering approach is not restricted to longitudinal illumina data , we provide an example of its application to a very different dataset , specifically 454 sequencing data from a cross - sectional environmental sampling performed by preheim et al . , 2013 .
clustering can be a useful strategy for filtering noise by coarse - graining data .
however , such coarse - graining may not be a necessity : if the noise level is low , as suggested by known estimates of pcr and sequencing error rates ( see , for example , quince et al .
2011 ) , then we can avoid clustering , since we expect each community member to be predominantly represented by the same 16s sequences .
we begin by illustrating this idea using the tongue microbiome data of caporaso et al . since the tongue community is relatively stable ( costello et al . , 2009 )
, the low - noise scenario would predict that certain specific sequences should consistently dominate in each sample . alternatively ,
if the noise were high , then the high - abundance community members would be represented by clouds of similar reads , none of which would clearly dominate .    to show that the data of caporaso et al .
supports the first ( low - noise ) scenario , we identified the top 5 sequences by overall abundance .
these sequences were strongly different ( fig .
[ fig : rank ] , inset ) , corresponding for the most part to bacteria from different phyla : in decreasing order of abundance , these were _ neisseria sp .
_ ( phylum _ proteobacteria _ , class _ betaproteobacteria _ ) , _ haemophilus sp . _ ( phylum _ proteobacteria _ , class _ gammaproteobacteria _ ) , _ fusobacterium sp . _ ( phylum _ fusobacteria _ ) , _ streptococcus sp . _ ( phylum _ firmicutes _ ) , and _ prevotella sp . _
( phylum _ bacteroidetes _ ) .
( taxonomy assigned by a blast search ( blastn 2.2.22 , matrix @xmath33 , gap / extenstion penalty @xmath34 ) against greengenes database ; desantis et al . , 2006 .
all 5 sequences had a match with 100% identity over 100% of sequence length . )
the sample - by - sample rank of these overall top 5 sequences was consistently in the top 10 .
we stress that the goal of fig .
[ fig : rank ] is not to characterize the temporal stability of community composition ( previously characterized , for example , in costello et al . , 2009 ) ;
rather , it serves to show that the community members that correspond to these highest - abundance tags are consistently represented by the same 130nt sequence ( at 100% identity ) across all samples .
in other words , despite the presence of noise in the data , 100% sequence identity is not an unreasonable criterion : the error rate is low enough that the error - free sequence dominates over the `` error cloud '' of its variants ( edgar , 2013 ) .
this key observation is the foundation of the approach described in this work .      to estimate
the rates of substitution errors observed in data after quality filtering , we used the `` error clouds '' around the high - abundance sequences in the dataset .
since all sequences were trimmed to a length of 130nt , each `` mother '' sequence has 390 direct neighbors in sequence space ( @xmath35 ) .
for very high - abundance sequences such as seq .
# 1 , all 390 neighbors were observed in at least one sample of the time series .
the time series of their abundances , normalized to the abundance of seq .
# 1 , is shown in fig .
[ fig : errorcloud ] . for this figure ,
the neighbors were ordered by the type of substitution that differentiates them from the mother sequence , and , within these categories , by the position of the differing nucleotide along the sequence .
we see that , with a few exceptions ( most notably the three neighbors also shown in fig .
1b ) , the abundance of a given neighbor is a constant fraction of the abundance of the mother sequence .
this is precisely what we expect for neighbors that arise as pcr or sequencing errors of the mother sequence , and the abundance ratio is then the probability of that particular error .
we see that the error rate is set primarily by the type of substitution , and does not exhibit significant dependence on the position along the sequence . for long reads ,
we would likely have seen an increase in error rates towards the end of the sequence , but our sequences are only 130nt long , well within the capabilities of accurate base - calling of the illumina platform .
we can therefore assign probabilities to substitution errors based solely on the substitution type ( which nucleotide was replaced by which other ) , independent of the position along the read .    to determine these probabilities , we first identify the neighbors that are outliers in their substitution category ; they likely correspond to true biological sequences physically present in the community , rather than sequencing errors .
outlier exclusion is done based on z - scores , i.e. for each sequence we compare its raw cumulative abundance over all samples to the mean in its substitution category , and normalize by the standard deviation in the category . a strong
outlier differing from the mother sequence by a nucleotide substitution at location @xmath24 will skew the error rate estimation at that location : some substitution type will appear to be unusually frequent .
therefore , we exclude nucleotide locations that correspond to the strongest outliers , those for which the z - score exceeds some threshold .
the remaining locations are then used to estimate the error rates : for each of these locations , we count the number of times a particular substitution occurred , as well as the number of times the nucleotide was recorded correctly .
after appropriate normalization , these counts give us the probability of each type of the error .
[ fig : errorrates ] shows the inferred substitution rates for error clouds around the top 50 sequences by abundance , using minimally quality - filtered data processed as described in the methods ( phred score cutoff 2 , z - score threshold 2 ) .
we find that the rates of different substitutions can differ dramatically ( up to  50-fold ) , but our estimates are highly reproducible ( note the log scale on the y axis ) , with variability predictably increasing if lower - abundance error clouds are used . table  s1 lists the error rates estimated from the error clouds of the top 10 sequences ( mean @xmath36 standard deviation ) .
note that , in principle , this effective error probability includes both the base - call errors of the illumina sequencer and the single - nucleotide substitution errors occurring during pcr .
however , the approximate symmetry between rates of a substitution and its reverse - complement partner ( e.g. @xmath37 ) , and a clear bias towards transitions as opposed to transversions , suggests that the observed substitutions are dominated by pcr errors ( compare with quince et al . , 2011 , table 2 ) .
inferring error rates directly from the data offers multiple strong advantages . specifying the error rate as an external parameter ( e.g. , morgan et al . , 2013 )
necessarily requires resorting to a conservative global upper bound . different pcr conditions and different sequencing machines will have different error rates ( for example , compare fig .
[ fig : qcparams ] and fig .
[ fig : dbc]a ) .
further , substitutions vary strongly in probability : in our case , using a single upper bound on error rates would have over - estimated the probability of certain error types by up to 50-fold , reducing our ability to resolve close sequences . in other words , measuring substitution rates directly from the data both reduces the number of algorithm parameters and improves performance .
.substitution error rates per nucleotide , multiplied by 1000 , as measured from the `` error clouds '' of the top 10 sequences by abundance .
error bars are standard deviations across the 10 estimates . [ cols="<,^,^,^,^,^",options="header " , ]      the approach described in this work does not explicitly rely on the longitudinal nature of the sampling .
most of our analysis can be readily applied to any multi - sample datasets , e.g. a cross - sectional sampling or a location series , provided samples were collected and processed in a similar way so that the error structure can be assumed to be similar .
further , and despite the caveats we described , our method can be applied even to data collected using the 454 sequencing platform . to illustrate the broad applicability of our approach , we used data from a cross - sectional environmental sampling conducted by preheim et al .
( sra accession number from srp029470 ) .
lake water microbiota were sampled at depths ranging from 0 m ( surface ) to 22 m with 1-meter depth intervals .
the authors used this data to illustrate their sequence clustering algorithm ( dbc ) that also relies on cross - sample comparisons to distinguish between closely related otus ; for details , see the original reference ( preheim et al . , 2013 ) .
they report their algorithm worked best with stringent quality filtering whereupon sequences were trimmed to just 76 nt , and any reads containing bases with phred quality scores at or below 16 were discarded .
this filtering retained 7.78 m total sequences ( 120k unique ) .
since our approach includes data denoising , we could use much more liberal quality score filtering and retain more data ( usearch maxee of 1 and truncating at phred quality score 2 ) . to compare runtime of our algorithm and dbc
, we increased the read truncation length so as to keep the same total number of sequences .
this set the quality - filtered sequence length to @xmath38 nt , 20% longer than used by the authors ( 7.98 m sequences , 300k unique ; ` tcsh run_cff_on_fastq.tcsh 91 analysisresults " preheimdata/*.fastq " ` . ) .
[ fig : dbc]a shows the substitution error rates inferred from the data at both sets of quality filtering parameters .
note that these rates are significantly lower than those of fig .
[ fig : qcparams ] ( the scales of the two plots are identical ) , exhibit a very weak transition / transversion bias , and are more sensitive to phred score quality filtering than what we have seen with caporaso et al . data ( fig .  [ fig : qcparams ] ) .
this seems to indicate that the protocol used by preheim et al .
generates significantly fewer pcr substitution errors .
this dependence of error rates on the experimental protocol highlights the advantage of being able to estimate error rates for a given dataset directly from the data , without the need for a separate calibration .
[ fig : dbc]bc provide examples of sequences differing by a single nucleotide exhibiting ecologically significant distinctions , as identified by our method in this environmental dataset ; compare with fig .  2a , fig .
[ fig : birthdeath ] and fig .
[ fig : feces]ab .
sequence abundance is shown as a function of depth ( each sample was independently normalized to @xmath39 total quality - filtered reads per sample , to correct for varying sample size ) .
the dbc method of preheim et al .
is also capable of identifying otus differing by a single nucleotide ( compare fig .
[ fig : dbc]bc to fig .
5b in the original reference ) ; however , our analysis achieved higher resolution by retaining longer reads and took only 13 min single - core processor time ( intel xeon cpu 2.83ghz ) , compared to 8 hours analysis time the authors report for parallelized dbc running on a cluster with 60 - 100 processes executing simultaneously .
in fact , the true runtime difference is even greater , since the complexity of both algorithms scales with the number of _ unique _ sequences rather than total reads . for @xmath40 as used by the authors of dbc , our algorithm completes in only 210 seconds .
we stress , however , that dbc and cluster - free filtering seek to achieve different goals and are not directly comparable .
dbc is an otu clustering algorithm , whereas the goal of cluster - free filtering is to identify sub - otu structure of moderate - to - high - abundance community members .
however , to our knowledge dbc is the only existing tool that exploits cross - sample comparisons to inform the interpretation of sequencing data , and the performance comparison above serves to illustrate the drastically different computational cost of the two approaches .
we have described a method that employs cross - sample comparisons to achieve sub - otu resolution .
the analysis presented in the main text uses data from a study with an uncommonly large number of samples ; in contrast , the previous section demonstrates that our method can be usefully applied to a dataset with only 22 datapoints .
what is the minimum number of samples required by our method ?
the answer is that the number of samples determines the resolution that can be achieved ; more samples will always allow higher resolution , but coarser differences can be resolved with just a few .
for example , just 2 samples ( say , 0 m and 10 m ) would have been enough to resolve the two subpopulations presented on fig .
[ fig : dbc]c .
by contrast , the difference between depth traces of seq .
# 10 and seq .
# 49 ( fig .
[ fig : dbc]b ) is less pronounced and more samples are required . finally , resolving the sequences in fig .
2b would not have been possible with fewer than @xmath41100 samples . for high - abundance sequences where the complex struture of noise in the counts can be neglected
, this tradeoff can be formally quantified using the jensen - shannon divergence as a measure of distance between abundance distributions of two sequences across samples ; for details , see preheim et al . , 2013 .
the maximum degree to which time traces of two sequences can be correlated is a function of their abundance : for low - abundance sequences the poisson sampling noise becomes non - negligible and sets an upper bound for the best achievable correlation coefficient .
consequently , to define a correlation as strong or weak , any measured correlation coefficient should be compared to this abundance - dependent quantity rather than to 1 .
let @xmath42 be the true abundance time trace of some bacterial strain ( in units of cells , rather than sequence counts ) .
imagine that two sequences in the dataset were measuring the abundance of this exact same strain , but with different amplification efficiencies @xmath43 and @xmath44 ( let @xmath45 ) .
neglecting all sources of noise other than the poisson counting noise , the abundance traces of these two sequences can be modeled by    @xmath46,\ ] ] where @xmath47 $ ] denotes adding poisson noise .
since poisson noise is unavoidable , the correlation coefficient between these two traces sets an upper bound for the correlation between @xmath48 and any other trace @xmath49 with the same mean abundance as @xmath50 .
this maximum correlation depends on the shape of the trace @xmath42 and amplification efficiencies @xmath43 , @xmath44 , and can be expressed as follows :    @xmath51=\operatorname{corr}\left(\operatorname{poiss}[\lambda_1 n(t)],\operatorname{poiss}[\lambda_2/\lambda_1
\ast \lambda_1 n(t)]\right).\ ] ]    and therefore , in terms of measurable quantities only :    @xmath52 \approx \operatorname{corr}\left(\operatorname{poiss}[n_1(t)],\operatorname{poiss}[\langle n_2\rangle/\langle n_1\rangle \ast n_1(t)]\right).\ ] ]    here @xmath53 denotes the average abundance , and we use the higher - abundance trace of the pair as the best estimate of the shape of the true abundance @xmath42 .
the maximum correlation coefficient depends on the shape of the trace @xmath48 and on the mean abundance of the trace we compare it to ; the lower the mean abundance is , the stronger the effect of poisson noise and the lower the @xmath54 .
in practice , for a pair of traces @xmath48 , @xmath50 , we compute their best expected correlation as follows :    1 .
take the more abundant trace @xmath48 2 .
construct a renormalized trace @xmath55 3 .
poisson - resample both of these 10 times : denote these @xmath56 , @xmath57 , @xmath58 .
4 .   compute the 100 correlation coefficients between all pairs @xmath59 .
set @xmath60=\langle c_{ij}\rangle$ ] .
the shape of the function @xmath60 $ ] is illustrated on fig .
[ fig : maxcorr ] .
we use the blast definition , i.e. the ratio of the number of mismatches to the total number of columns after pairwise realignment , and multiply this ratio by the length of the sequence ( 130 nt ) . for close sequences that differ by a few substitution errors the alignment is trivial , and this normalization corresponds to the hamming distance between sequences , in nt .
we define the autocorrelation time @xmath61 of a sequence as the time shift @xmath8 at which the autocorrelation function @xmath7 falls below the threshold of statistical significance . for reasons discussed above
, the notions of strong ( significant ) or weak ( insignificant ) correlation of sequence time traces are abundance - dependent .
therefore , instead of using a fixed threshold value for all sequences , we proceed in the following way . for a given sequence ,
we first compute its root - mean - square autocorrelation coefficient for time shifts between 70 and 100 samples :    @xmath62    if we assume that all autocorrelation observed at such large time shifts is entirely due to noise , then @xmath63 provides a natural scale for statistical significance .
we conservatively define the significance threshold at twice the magnitude of @xmath63 .
note that @xmath63 provides an upper bound on a statistically significant correlation value .
if some dynamical processes in the population are slow enough that they contribute to the autocorrelation function even at such large time shifts ( _ cf .
[ fig : birthdeath ] ) , this will increase @xmath63 and cause us to underestimate the true autocorrelation time .
this means that assuming @xmath63 was entirely due to noise is a safe approximation to make : if it does not hold , it can only strengthen our conclusion that the sequence abundance time traces exhibit multi - day autocorrelations .
[ fig : birthdeath]ab shows examples of sequences exhibiting steady change in abundance for more than a month . in both cases ,
the slow - changing sequence is 99.2% similar to a very high - abundance community member and could not have been resolved by traditional otu - based methods .
note the sharp jump in panel b at day 182 of the sequence representing the invading subpopulation ( red ) to an abundance value close to the equilibrium established after day 210 .
it is intriguing to speculate that this trace may document spatial invasion of a subpopulation already established elsewhere on the tongue , a region accidentally sampled on day 182 .
to distinguish between 16s tags coming from distinct subpopulations or from physically the same bacterial cells , we introduced a quantity we called the _ persistence of difference _ @xmath17 .
for this , we first defined the fractional difference @xmath14 between two time traces renormalized to the same mean @xmath16 :    @xmath15    we then defined the persistence of difference @xmath17 as the 1-day autocorrelation coefficient of @xmath14 . if @xmath10 and @xmath11 are two genomic variants contained within the same bacterium , then any difference between @xmath12 and @xmath13 must be due to measurement noise , and @xmath17 must vanish .
if , however , @xmath16 reflect abundances of two distinct subpopulations , then @xmath14 can be expected to exhibit some degree of autocorrelation due to the slow dynamics observed for most individual sequences .
we gave an intuitive argument for this in the main text . here ,
to gain some extra intuition about the null model for @xmath17 , we calculate it explicitly in the simplest case when the two traces @xmath64 are independent and can be approximated by a stationary , weakly fluctuating process :    @xmath65    here @xmath66 have zero mean , unit variance and are uncorrelated .
assuming @xmath67 , we can write :    @xmath68    and therefore , making use of the independence assumption ,    @xmath69    here @xmath70 are the one - day autocorrelation coefficients of the fluctuations of the two individual sequences .
the independence approximation made above is clearly not valid for the dynamics of most community members .
for this reason , for the purposes of fig .  3c , the null - model prediction was constructed directly from the data , by reversing in all pairs the time order for one of the sequences prior to the calculation of @xmath17 .
this removes any real correlations of the traces while preserving autocorrelation and other properties of the traces such as their fluctuation spectrum . nevertheless , the calculation above is useful as it explains why the null - model expectation for @xmath17 is non - zero when both sequences have slow internal dynamics .
note that a sequence with an exceptionally long intrinsic time scale ( as shown in fig .
[ fig : birthdeath]ab ) will have a large @xmath17 score when paired with any other sequence .
these two sequences were therefore excluded from fig .
none of the cross - sample comparison methodology described in this work is limited to time series data .
the `` persistence of difference '' argument accompanying fig .
4 is no exception ; however , it does rely on two additional assumptions , namely that the composition of samples varies smoothly with some parameter labeling the samples , and that the sampling frequency is sufficiently high to allow correlations of fluctuations to be observed between consecutive samples . for the longitudinal data series of caporaso et al .
this parameter was time ; for a location series one can expect community composition to vary smoothly in space , and the same argument can be applied . in other words ,
the use of `` persistence of difference '' @xmath17 need not be limited strictly to longitudinal datasets .
however , autocorrelation - based analysis is particularly sensitive to the number of samples ( see section  [ sec : numberofsamples ] ) . determining whether @xmath17 can be a useful concept for studying the spatial heterogeneity of populations requires further investigation .
as described in the main text , for the purposes of fig .  4 , when calculating otu quality scores , we restricted our attention only to high - abundance members of the otu , considering only sequences from the top 200 by overall abundance . since most of the diversity is contributed by low - abundance species ( huttenhower et al . , 2012 ) ,
4 underestimates the true diversity of an otu .
including lower - abundance otu members makes otu quality scores drop continuously as new otu members are added ; however , it also becomes increasingly hard to separate dynamical diversity from the effects of noise .
consequently , in fig .  4 we report our most conservative estimate of within - otu diversity , where we use only the highest - abundance members out of all those resolved by cluster - free filtering ( there was an average of @xmath71 resolved sequences within a 97% otu , and only @xmath72 per otu were used for fig .  4 ) .
in addition , otu quality scores were calculated under the assumption that each sequence represents a separate subpopulation .
sequences that in fact derive from the same bacteria ( 16s paralogs or errors not in our model ) appear in the defining equation as independent , dynamically identical subpopulations , increasing the apparent otu quality score .
this is another reason why the true quality scores of otus are likely even lower than reported in fig .
to confirm our conclusions from the analysis of tongue microbiome data presented in the main text , we repeated our analysis using fecal samples of the two individuals , collected in the same study ( caporaso et al . , 2011 ) .
there were 374 samples , 243 from the male subject and 131 from the female , with @xmath73 reads per sample .
we normalized the observed abundances to @xmath74 total reads in each sample to correct for varying sample size . as before , we labeled sequences in order of decreasing overall abundance ( pooling samples from both individuals ) : seq  # 1f , seq  # 2f , etc . , where `` f '' reflects that we are now dealing with fecal samples rather than the tongue .
again , we find that sequences differing by as little as a single nucleotide can exhibit ecologically significant differences in their dynamics .
the most striking example is that the dominating sequence in individual `` male 3 '' differs from the dominating sequence in `` female 4 '' by a single nucleotide , and virtually no cross - contamination is observed ( fig/  [ fig : otu]a ) .
both these sequences map to _ bacteroides sp . _ in greengenes ( desantis et al . , 2006 ) .
another example is presented in panel b. finally , we repeat the cross - individual analysis presented , for tongue samples , in fig .  5ab .
we find that the two gut communities as probed by the fecal samples also share a large fraction of sequences at 100% identity .
this once again supports the scenario whereby the communities exchange members with non - negligible frequency , although less so than the tongue samples .
the observation of panel a is therefore unlikely to represent the effect of dispersal limitation , suggesting instead a functional difference between the representatives of _ bacteroides sp . _ established in the two individuals or a resistance to invasion . finally , we find that the dynamical similarity of shared sequences , when measured independently in the two individuals , is clearly correlated , just as it was for the tongue communities ( fig .
5b ) . with the number of shared sequences being lower for fecal samples than for tongue samples ,
the statistics were insufficient to compare dynamical similarity of `` intentionally mismatched '' sequences as in fig .
5c .      the same analysis as in fig .
5b can be performed for shared 97% otus rather than shared sequences ( at 100% identity ) .
we constructed otus using closed - reference otu picking as implemented in qiime , matching sequences at 97% sequence similarity against the greengenes database .
[ fig : otu ] shows the scatter plot of the dynamical similarity between pairs of common otus , as measured independently in the two individuals , for 78 common otus ( those shared within the top 100 ) .
note , however , that most otus are dominated by a single high - abundance sequence ( as evidenced by the high weighted quality score on fig .  4 ) , and most of these dominating sequences are shared across the two communities ( fig .
for these reasons , the plot shown here is very similar to fig .  5b , but only because the within - otu diversity is masked by dominating subpopulations .
0.2 in -0.2 in 1em    desantis tz , hugenholtz p , larsen n , rojas m , brodie el , keller k et al ( 2006 ) .
greengenes , a chimera - checked 16s rrna gene database and workbench compatible with arb .
_ appl environ microbiol _ * 72 * : 50695072 .