this article proposes an approach to inferring the causal impact of a market intervention , such as a new product launch or the onset of an advertising campaign .
our method generalises the widely used difference - in - differences approach to the time - series setting by explicitly modelling the counterfactual of a time series observed both before and after the intervention .
it improves on existing methods in two respects : it provides a fully bayesian time - series estimate for the effect ; and it uses model averaging to construct the most appropriate synthetic control for modelling the counterfactual .
the ` causalimpact ` r package provides an implementation of our approach ( http://google.github.io/causalimpact/ ) .    inferring
the impact of market interventions is an important and timely problem . partly because of recent interest in big data ,
many firms have begun to understand that a competitive advantage can be had by systematically using impact measures to inform strategic decision making .
an example is the use of `` a@xmath0b experiments '' to identify the most effective market treatments for the purpose of allocating resources [ @xcite ] .    here , we focus on measuring the impact of a discrete marketing event , such as the release of a new product , the introduction of a new feature , or the beginning or end of an advertising campaign , with the aim of measuring the event s impact on a response metric of interest ( e.g. , sales ) .
the causal impact of a treatment is the difference between the observed value of the response and the ( unobserved ) value that would have been obtained under the alternative treatment , that is , the effect of treatment on the treated [ @xcite , @xcite ( @xcite ) ] . in the present setting the response variable is a time series , so the causal effect of interest is the difference between the observed series and the series that would have been observed had the intervention not taken place .
a powerful approach to constructing the counterfactual is based on the idea of combining a set of candidate predictor variables into a single `` synthetic control '' [ @xcite ] .
broadly speaking , there are three sources of information available for constructing an adequate synthetic control .
the first is the time - series behaviour of the response itself , prior to the intervention .
the second is the behaviour of other time series that were predictive of the target series prior to the intervention .
such control series can be based , for example , on the same product in a different region that did not receive the intervention or on a metric that reflects activity in the industry as a whole . in practice , there are often many such series available , and the challenge is to pick the relevant subset to use as contemporaneous controls .
this selection is done on the _ pre - treatment _ portion of potential controls ; but their value for predicting the counterfactual lies in their _ post - treatment _ behaviour .
as long as the control series received no intervention themselves , it is often reasonable to assume the relationship between the treatment and the control series that existed prior to the intervention to continue afterwards .
thus , a plausible estimate of the counterfactual time series can be computed up to the point in time where the relationship between treatment and controls can no longer be assumed to be stationary , for example , because one of the controls received treatment itself . in a bayesian framework ,
a third source of information for inferring the counterfactual is the available prior knowledge about the model parameters , as elicited , for example , by previous studies .    ) with an intervention beginning in january 2014 .
two other markets ( @xmath1 , @xmath2 ) were not subject to the intervention and allow us to construct a synthetic control [ cf .
@xcite ] . inverting the state - space model described in the main text
yields a prediction of what would have happened in @xmath3 had the intervention not taken place ( posterior predictive expectation of the counterfactual with pointwise 95% posterior probability intervals ) .
the difference between observed data and counterfactual predictions is the inferred causal impact of the intervention . here
, predictions accurately reflect the true ( gamma - shaped ) impact . a key characteristic of the inferred impact series is the progressive widening of the posterior intervals ( shaded area ) .
this effect emerges naturally from the model structure and agrees with the intuition that predictions should become increasingly uncertain as we look further and further into the ( retrospective ) future .
another way of visualizing posterior inferences is by means of a cumulative impact plot .
it shows , for each day , the summed effect up to that day . here
, the 95% credible interval of the cumulative impact crosses the zero - line about five months after the intervention , at which point we would no longer declare a significant overall effect . ]
[ fig : intro : illustration ]    we combine the three preceding sources of information using a state - space time - series model , where one component of state is a linear regression on the contemporaneous predictors . the framework of our model allows us to choose from among a large set of potential controls by placing a spike - and - slab prior on the set of regression coefficients and by allowing the model to average over the set of controls [ @xcite ] .
we then compute the posterior distribution of the counterfactual time series given the value of the target series in the pre - intervention period , along with the values of the controls in the post - intervention period .
subtracting the predicted from the observed response during the post - intervention period gives a semiparametric bayesian posterior distribution for the causal effect ( figure  [ fig : intro : illustration ] ) .      as with other domains , causal inference in marketing requires subtlety .
marketing data are often observational and rarely follow the ideal of a randomised design .
they typically exhibit a low signal - to - noise ratio .
they are subject to multiple seasonal variations , and they are often confounded by the effects of unobserved variables and their interactions [ for recent examples , see @xcite , @xcite ( @xcite ) ] .
rigorous causal inferences can be obtained through randomised experiments , which are often implemented in the form of geo experiments .
many market interventions , however , fail to satisfy the requirements of such approaches .
for instance , advertising campaigns are frequently launched across multiple channels , online and offline , which precludes measurement of individual exposure .
campaigns are often targeted at an entire country , and one country only , which prohibits the use of geographic controls within that country .
likewise , a campaign might be launched in several countries but at different points in time .
thus , while a large control group may be available , the treatment group often consists of no more than one region or a few regions with considerable heterogeneity among them .
a standard approach to causal inference in such settings is based on a linear model of the observed outcomes in the treatment and control group before and after the intervention .
one can then estimate the difference between ( i )  the pre - post difference in the treatment group and ( ii )  the pre - post difference in the control group .
the assumption underlying such _ difference - in - differences _ ( dd ) designs is that the level of the control group provides an adequate proxy for the level that would have been observed in the treatment group in the absence of treatment [ see @xcite ] .
dd designs have been limited in three ways .
first , dd is traditionally based on a static regression model that assumes i.i.d .
data despite the fact that the design has a temporal component .
when fit to serially correlated data , static models yield overoptimistic inferences with too narrow uncertainty intervals [ see also @xcite , @xcite ( @xcite ) , @xcite ] .
second , most dd analyses only consider two time points : before and after the intervention . in practice
, the manner in which an effect evolves over time , especially its onset and decay structure , is often a key question .
third , when dd analyses _ are _ based on time series , previous studies have imposed restrictions on the way in which a synthetic control is constructed from a set of predictor variables , which is something we wish to avoid . for example
, one strategy [ @xcite ] has been to choose a convex combination @xmath4 of @xmath5 predictor time series in such a way that a vector of pre - treatment variables ( not time series ) @xmath1 characterising the treated unit before the intervention is matched most closely by the combination of pre - treatment variables @xmath6 of the control units w.r.t . a vector of importance weights @xmath7 .
these weights are themselves determined in such a way that the combination of pre - treatment outcome time series of the control units most closely matches the pre - treatment outcome time series of the treated unit .
such a scheme relies on the availability of interpretable characteristics ( e.g. , growth predictors ) , and it precludes nonconvex combinations of controls when constructing the weight vector @xmath8 .
we prefer to select a combination of control series without reference to external characteristics and purely in terms of how well they explain the pre - treatment outcome time series of the treated unit ( while automatically balancing goodness of fit and model complexity through the use of regularizing priors ) .
another idea [ @xcite ] has been to use classical variable - selection methods ( such as the lasso ) to find a sparse set of predictors .
this approach , however , ignores posterior uncertainty about both which predictors to use and their coefficients
.    the limitations of dd schemes can be addressed by using state - space models , coupled with highly flexible regression components , to explain the temporal evolution of an observed outcome .
state - space models distinguish between a state equation that describes the transition of a set of latent variables from one time point to the next and an observation equation that specifies how a given system state translates into measurements .
this distinction makes them extremely flexible and powerful [ see @xcite for a discussion in the context of marketing research ] .
the approach described in this paper inherits three main characteristics from the state - space paradigm .
first , it allows us to flexibly accommodate different kinds of assumptions about the latent state and emission processes underlying the observed data , including local trends and seasonality .
second , we use a fully bayesian approach to inferring the temporal evolution of counterfactual activity and incremental impact .
one advantage of this is the flexibility with which posterior inferences can be summarised .
third , we use a regression component that precludes a rigid commitment to a particular set of controls by integrating out our posterior uncertainty about the influence of each predictor as well as our uncertainty about which predictors to include in the first place , which avoids overfitting .
the remainder of this paper is organised as follows .
section  [ sec : theory ] describes the proposed model , its design variations , the choice of diffuse empirical priors on hyperparameters , and a stochastic algorithm for posterior inference based on markov chain monte carlo ( mcmc ) .
section  [ sec : apps : synthetic ] demonstrates important features of the model using simulated data , followed by an application in section  [ sec : apps : empirical ] to an advertising campaign run by one of google s advertisers .
section  [ sec : disc ] puts our approach into context and discusses its scope of application .
structural time - series models are state - space models for time - series data .
they can be defined in terms of a pair of equations @xmath9 where @xmath10 and @xmath11 are independent of all other unknowns .
equation  ( [ eq : observation ] ) is the _ observation equation _ ; it links the observed data @xmath12 to a latent @xmath13-dimensional state vector @xmath14 .
equation  ( [ eq : transition ] ) is the _ state equation _ ; it governs the evolution of the state vector @xmath14 through time . in the present paper
, @xmath12 is a scalar observation , @xmath15 is a @xmath13-dimensional output vector , @xmath16 is a @xmath17 transition matrix , @xmath18 is a @xmath19 control matrix , @xmath20 is a scalar observation error with noise variance @xmath21 , and @xmath22 is a @xmath23-dimensional system error with a @xmath24 state - diffusion matrix @xmath25 , where @xmath26 . writing the error structure of equation  ( [ eq : transition ] ) as @xmath27 allows us to incorporate state components of less than full rank ; a model for seasonality will be the most important example .
structural time - series models are useful in practice because they are flexible and modular .
they are flexible in the sense that a very large class of models , including all arima models , can be written in the state - space form given by ( [ eq : observation ] ) and ( [ eq : transition ] ) .
they are modular in the sense that the latent state as well as the associated model matrices @xmath28 , and @xmath25 can be assembled from a library of component sub - models to capture important features of the data .
there are several widely used state - component models for capturing the trend , seasonality or effects of holidays .
a common approach is to assume the errors of different state - component models to be independent ( i.e. , @xmath25 is block - diagonal ) .
the vector @xmath14 can then be formed by concatenating the individual state components , while @xmath16 and @xmath18 become block - diagonal matrices .
the most important state component for the applications considered in this paper is a regression component that allows us to obtain counterfactual predictions by constructing a synthetic control based on a combination of markets that were not treated .
observed responses from such markets are important because they allow us to explain variance components in the treated market that are not readily captured by more generic seasonal sub - models .
this approach assumes that covariates are unaffected by the effects of treatment .
for example , an advertising campaign run in the united states might spill over to canada or the united kingdom .
when assuming the absence of spill - over effects , the use of such indirectly affected markets as controls would lead to pessimistic inferences , that is , the effect of the campaign would be underestimated [ cf .
@xcite ] .
the first component of our model is a local linear trend , defined by the pair of equations @xmath29 \label{eq : local - linear - trend } \\[-8pt ] \nonumber \delta_{t+1 } & = & \delta_{t } + \eta_{\delta , t},\end{aligned}\ ] ] where @xmath30 and @xmath31 .
the @xmath32 component is the value of the trend at time @xmath33 .
the @xmath34 component is the expected increase in @xmath35 between times @xmath33 and @xmath36 , so it can be thought of as the _ slope _ at time @xmath33 .
the local linear trend model is a popular choice for modelling trends because it quickly adapts to local variation , which is desirable when making short - term predictions .
this degree of flexibility may not be desired when making longer - term predictions , as such predictions often come with implausibly wide uncertainty intervals .
there is a generalisation of the local linear trend model where the slope exhibits stationarity instead of obeying a random walk .
this model can be written as @xmath37 \label{eq : generalized - local - linear - trend } \\[-8pt ] \nonumber \delta_{t+1 } & = & d + \rho(\delta_{t } - d ) + \eta_{\delta , t},\end{aligned}\ ] ] where the two components of @xmath38 are independent . in this model ,
the slope of the time trend exhibits @xmath39 variation around a long - term slope of @xmath40 .
the parameter @xmath41 represents the learning rate at which the local trend is updated .
thus , the model balances short - term information with information from the distant past .
there are several commonly used state - component models to capture seasonality .
the most frequently used model in the time domain is @xmath42 where @xmath43 represents the number of seasons and @xmath44 denotes their joint contribution to the observed response @xmath12 .
the state in this model consists of the @xmath45 most recent seasonal effects , but the error term is a scalar , so the evolution equation for this state model is less than full rank .
the mean of @xmath46 is such that the total seasonal effect is zero when summed over @xmath43 seasons .
for example , if we set @xmath47 to capture four seasons per year , the mean of the _ winter _ coefficient will be @xmath48 .
the part of the transition matrix @xmath16 representing the seasonal model is an @xmath49 matrix with @xmath50 s along the top row , 1 s along the subdiagonal and 0 s elsewhere .
the preceding seasonal model can be generalised to allow for multiple seasonal components with different periods . when modelling daily data , for example , we might wish to allow for an @xmath51 day - of - week effect , as well as an @xmath52 weekly annual cycle .
the latter can be handled by setting @xmath53 , with zero variance on the error term , when @xmath33 is not the start of a new week , and setting @xmath16 to the usual seasonal transition matrix , with nonzero error variance , when @xmath33 is the start of a new week .
control time series that received no treatment are critical to our method for obtaining accurate counterfactual predictions since they account for variance components that are shared by the series , including , in particular , the effects of other unobserved causes otherwise unaccounted for by the model .
a natural way of including control series in the model is through a linear regression .
its coefficients can be static or time - varying .    a _ static _
regression can be written in state - space form by setting @xmath54 and @xmath55 .
one advantage of working in a fully bayesian treatment is that we do not need to commit to a fixed set of covariates .
the spike - and - slab prior described in section  [ sec : posterior - sampling ] allows us to integrate out our posterior uncertainty about which covariates to include and how strongly they should influence our predictions , which avoids overfitting .
all covariates are assumed to be contemporaneous ; the present model does not infer on a potential lag between treated and untreated time series .
a known lag , however , can be easily incorporated by shifting the corresponding regressor in time .
an alternative to the above is a regression component with _ dynamic _ regression coefficients to account for time - varying relationships [ e.g. , @xcite ] .
given covariates @xmath56 , this introduces the dynamic regression component @xmath57 \label{eq : dynamic - regression } \\[-8pt ] \nonumber \beta_{j , t+1 } & = & \beta_{j , t } + \eta_{\beta , j , t},\end{aligned}\ ] ] where @xmath58 . here , @xmath59 is the coefficient for the @xmath60th control series and @xmath61 is the standard deviation of its associated random walk .
we can write the dynamic regression component in state - space form by setting @xmath62 and @xmath63 and by setting the corresponding part of the transition matrix to @xmath64 , with @xmath65 .
structural time - series models allow us to examine the time series at hand and flexibly choose appropriate components for trend , seasonality , and either static or dynamic regression for the controls . the presence or absence of seasonality ,
for example , will usually be obvious by inspection .
a more subtle question is whether to choose static or dynamic regression coefficients .
when the relationship between controls and treated unit has been stable in the past , static coefficients are an attractive option .
this is because a spike - and - slab prior can be implemented efficiently within a forward - filtering , backward - sampling framework .
this makes it possible to quickly identify a sparse set of covariates even from tens or hundreds of potential variables [ @xcite ] .
local variability in the treated time series is captured by the dynamic local level or dynamic linear trend component .
covariate stability is typically high when the available covariates are close in nature to the treated metric .
the empirical analyses presented in this paper , for example , will be based on a static regression component ( section  [ sec : apps : empirical ] ) .
this choice provides a reasonable compromise between capturing local behaviour and accounting for regression effects .
an alternative would be to use dynamic regression coefficients , as we do , for instance , in our analyses of simulated data ( section  [ sec : apps : synthetic ] ) .
dynamic coefficients are useful when the linear relationship between treated metrics and controls is believed to change over time .
there are a number of ways of reducing the computational burden of dealing with a potentially large number of dynamic coefficients .
one option is to resort to dynamic latent factors , where one uses @xmath66 with @xmath67 and uses @xmath68 instead of @xmath69 as part of @xmath15 in ( [ eq : observation ] ) , coupled with an ar - type model for @xmath68 itself .
another option is latent thresholding regression , where one uses a dynamic version of the spike - and - slab prior as in @xcite .
the state - component models are assembled independently , with each component providing an additive contribution to @xmath12 .
figure  [ fig : theory : model ] illustrates this process assuming a local linear trend paired with a static regression component .
is modelled as the result of a latent state plus gaussian observation noise with error standard deviation @xmath70 .
the state @xmath71 includes a local level @xmath32 , a local linear trend @xmath34 , and a set of contemporaneous covariates @xmath69 , scaled by regression coefficients @xmath72 .
state components are assumed to evolve according to independent gaussian random walks with fixed standard deviations @xmath73 and @xmath74 ( conditional - dependence arrows shown for the first time point only ) . the model includes empirical priors on these parameters and the initial states . in an alternative formulation ,
the regression coefficients @xmath75 are themselves subject to random - walk diffusion ( see main text ) . of principal interest is the posterior predictive density over the unobserved counterfactual responses @xmath76 .
subtracting these from the actual observed data @xmath77 yields a probability density over the temporal evolution of causal impact . ]
let @xmath78 generically denote the set of all model parameters and let @xmath79 denote the full state sequence .
we adopt a bayesian approach to inference by specifying a prior distribution @xmath80 on the model parameters as well as a distribution @xmath81 on the initial state values .
we may then sample from @xmath82 using mcmc .
most of the models in section  [ sec : components - state ] depend solely on a small set of variance parameters that govern the diffusion of the individual state components . a typical prior distribution for such a variance is @xmath83 where @xmath84 is the gamma distribution with expectation @xmath85 .
the prior parameters can be interpreted as a prior sum of squares @xmath86 , so that @xmath87 is a prior estimate of  @xmath88 , and @xmath89 is the weight , in units of prior sample size , assigned to the prior estimate .
we often have a weak default prior belief that the incremental errors in the state process are small , which we can formalise by choosing small values of @xmath89 ( e.g. , 1 ) and small values of @xmath87 .
the notion of `` small '' means different things in different models ; for the seasonal and local linear trend models our default priors are @xmath90 , where @xmath91 is the sample variance of the target series .
scaling by the sample variance is a minor violation of the bayesian paradigm , but it is an effective means of choosing a reasonable scale for the prior .
it is similar to the popular technique of scaling the data prior to analysis , but we prefer to do the scaling in the prior so we can model the data on its original scale .
when faced with many potential controls , we prefer letting the model choose an appropriate set .
this can be achieved by placing a spike - and - slab prior over coefficients [ @xcite ( @xcite ) , @xcite ] .
a spike - and - slab prior combines point mass at zero ( the `` spike '' ) , for an unknown subset of zero coefficients , with a weakly informative distribution on the complementary set of nonzero coefficients ( the `` slab '' ) .
contrary to what its name might suggest , the `` slab '' is usually not completely flat , but rather a gaussian with a large variance .
let @xmath92 , where @xmath93 if @xmath94 and @xmath95 otherwise .
let @xmath96 denote the nonzero elements of the vector @xmath97 and let @xmath98 denote the rows and columns of @xmath99 corresponding to nonzero entries in @xmath100 .
we can then factorise the spike - and - slab prior as @xmath101 the spike portion of ( [ eq : spike - slab - general ] ) can be an arbitrary distribution over @xmath102 in principle ; the most common choice in practice is a product of independent bernoulli distributions , @xmath103 where @xmath104 is the prior probability of regressor @xmath60 being included in the model .
values for @xmath104 can be elicited by asking about the _ expected model size _
@xmath105 , and then setting all @xmath106 .
an alternative is to use a more specific set of values  @xmath104 .
in particular , one might choose to set certain @xmath104 to either  1 or  0 to force the corresponding variables into or out of the model . generally , framing the prior in terms of expected model size has the advantage that the model can adapt to growing numbers of predictor variables without having to switch to a hierarchical prior [ @xcite ] .
for the `` slab '' portion of the prior we use a conjugate normal - inverse gamma distribution , @xmath107 the vector @xmath108 in equation  ( [ eq : slab - normal ] ) encodes our prior expectation about the value of each element of @xmath75 . in practice , we usually set @xmath109 .
the prior parameters in equation  ( [ eq : slab - variance ] ) can be elicited by asking about the expected @xmath110 $ ] as well as the number of observations worth of weight @xmath111 the prior estimate should be given .
then @xmath112 .    the final prior parameter in ( [ eq : slab - normal ] ) is @xmath99 , which , up to a scaling factor , is the prior precision over @xmath75 in the full model , with all variables included .
the total information in the covariates is @xmath113 , and so @xmath114 is the average information in a single observation .
zellner s @xmath115-prior [ @xcite ] sets @xmath116 , so that @xmath115 can be interpreted as @xmath115 observations worth of information .
zellner s prior becomes improper when @xmath113 is not positive definite ; we therefore ensure propriety by averaging @xmath113 with its diagonal , @xmath117 with default values of @xmath118 and @xmath119 .
overall , this prior specification provides a broadly useful default while providing considerable flexibility in those cases where more specific prior information is available .
posterior inference in our model can be broken down into three pieces .
first , we simulate draws of the model parameters @xmath120 and the state vector @xmath121 given the observed data @xmath122 in the training period .
second , we use the posterior simulations to simulate from the posterior predictive distribution @xmath123 given the observed pre - intervention activity @xmath124 .
third , we use the posterior predictive samples to compute the posterior distribution of the pointwise impact @xmath125 for each @xmath126 .
we use the same samples to obtain the posterior distribution of cumulative impact .
we use a gibbs sampler to simulate a sequence @xmath127 from a markov chain whose stationary distribution is @xmath128 . the sampler alternates between a _ data - augmentation _ step that simulates from @xmath129 and a _ parameter - simulation _ step that simulates from @xmath130 .
the data - augmentation step uses the posterior simulation algorithm from @xcite , providing an improvement over the earlier forward - filtering , backward - sampling algorithms by @xcite , @xcite , and @xcite . in brief , because @xmath131 is jointly multivariate normal , the variance of @xmath132 does not depend on @xmath124 .
we can therefore simulate @xmath133 and subtract @xmath134 to obtain zero - mean noise with the correct variance .
adding @xmath135 restores the correct mean , which completes the draw .
the required expectations can be computed using the kalman filter and a _ fast mean smoother _ described in detail by @xcite .
the result is a direct simulation from @xmath136 in an algorithm that is linear in the total ( pre- and post - intervention ) number of time points ( @xmath137 ) and quadratic in the dimension of the state space ( @xmath13 ) .    given the draw of the state , the parameter draw is straightforward for all state components other than the static regression coefficients @xmath138 .
all state components that exclusively depend on variance parameters can translate their draws back to error terms @xmath22 and accumulate sums of squares of @xmath38 , and , because of conjugacy with equation  ( [ eq : typical - variance ] ) , the posterior distribution will remain gamma distributed .
the draw of the static regression coefficients @xmath138 proceeds as follows . for each @xmath139 in the pre - intervention period ,
let @xmath140 denote @xmath12 with the contributions from the other state components subtracted away , and let @xmath141 .
the challenge is to simulate from @xmath142 .
because of conjugacy , we can integrate out @xmath138 and @xmath143 and be left with @xmath144 where @xmath145 is an unknown normalizing constant .
the sufficient statistics in equation  ( [ eq : inclusion - indicator - posterior ] ) are @xmath146 to sample from ( [ eq : inclusion - indicator - posterior ] ) , we use a gibbs sampler that draws each @xmath147 given all other @xmath148 .
each full - conditional is easy to evaluate because @xmath147 can only assume two possible values .
it should be noted that the dimension of all matrices in ( [ eq : inclusion - indicator - posterior ] ) is @xmath149 , which is small if the model is truly sparse .
there are many matrices to manipulate , but because each is small , the overall algorithm is fast .
once the draw of @xmath150 is complete , we sample directly from @xmath151 using standard conjugate formulae . for an alternative that may be even more computationally efficient ,
see @xcite .
while the posterior over model parameters and states @xmath152 can be of interest in its own right , causal impact analyses are primarily concerned with the posterior incremental effect , @xmath153 as shown by its indices , the density in equation  ( [ eqn : theory : postpred ] ) is defined precisely for that portion of the time series which is unobserved : the counterfactual market response @xmath154 that would have been observed in the treated market , after the intervention , in the absence of treatment .
it is also worth emphasising that the density is conditional on the observed data ( as well as the priors ) and only on these , that is , on activity in the treatment market before the beginning of the intervention as well as activity in all control markets both before and during the intervention .
the density is _ not _ conditioned on parameter estimates or the inclusion or exclusion of covariates with static regression coefficients , all of which have been integrated out .
thus , through bayesian model averaging , we commit neither to any particular set of covariates , which helps avoid an arbitrary selection , nor to point estimates of their coefficients , which prevents overfitting .    the posterior predictive density in ( [ eqn : theory : postpred ] )
is defined as a coherent ( joint ) distribution over all counterfactual data points , rather than as a collection of pointwise univariate distributions .
this ensures that we correctly propagate the serial structure determined on pre - intervention data to the trajectory of counterfactuals .
this is crucial , in particular , when forming summary statistics , such as the cumulative effect of the intervention on the treatment market .
posterior inference was implemented in c@xmath155 with an r interface .
given a typically - sized data set with @xmath156 time points , @xmath157 covariates , and @xmath158 iterations ( see section  [ sec : apps : empirical ] for an example ) , this implementation takes less than 30  seconds to complete on a standard computer , enabling near - interactive analyses .
samples from the posterior predictive distribution over counterfactual activity can be readily used to obtain samples from the posterior causal effect , that is , the quantity we are typically interested in . for each draw @xmath159 and for each time point @xmath160
, we set @xmath161 yielding samples from the approximate posterior predictive density of the effect attributed to the intervention .
in addition to its pointwise impact , we often wish to understand the cumulative effect of an intervention over time .
one of the main advantages of a sampling approach to posterior inference is the flexibility and ease with which such derived inferences can be obtained . reusing the impact samples obtained in ( [ eqn : theory : causal_samples ] ) ,
we compute for each draw @xmath159 @xmath162 the preceding _ cumulative sum _ of causal increments is a useful quantity when @xmath163 represents a _ flow _ quantity , measured over an interval of time ( e.g. , a day ) , such as the number of searches , sign - ups , sales , additional installs or new users .
it becomes uninterpretable when @xmath163 represents a _
quantity , usefully defined only for a point in time , such as the total number of clients , users or subscribers . in this case
we might instead choose , for each @xmath159 , to draw a sample of the posterior _ running average _ effect following the intervention , @xmath164 unlike the cumulative effect in ( [ eqn : cumlift ] ) , the running average is always interpretable , regardless of whether it refers to a flow or a stock .
however , it is more context - dependent on the length of the post - intervention period under consideration . in particular , under the assumption of a true impact that grows quickly at first and then declines to zero , the cumulative impact approaches its true total value ( in expectation ) as we increase the counterfactual forecasting period , whereas the average impact will eventually approach zero ( while , in contrast , the probability intervals diverge in both cases , leading to more and more uncertain inferences as the forecasting period increases ) .
to study the characteristics of our approach , we analysed simulated ( i.e. , computer - generated ) data across a series of independent simulations .
generated time series started on 1  january 2013 and ended on 30  june 2014 , with a perturbation beginning on 1  january 2014 .
the data were simulated using a dynamic regression component with two covariates whose coefficients evolved according to independent random walks , @xmath165 , initialised at @xmath166 .
the covariates themselves were simple sinusoids with wavelengths of 90  days and 360  days , respectively .
the latent state underlying the observed data was generated using a local level that evolved according to a random walk , @xmath167 , initialised at @xmath168 .
independent observation noise was sampled using @xmath169 . in summary ,
observations @xmath12 were generated using @xmath170 to simulate the effect of advertising , the post - intervention portion of the preceding series was multiplied by @xmath171 , where @xmath172 ( not to be confused with @xmath173 ) represented the true effect size specifying the ( uniform ) relative lift during the campaign period .
an example is shown in figure  [ fig : apps : synthetic : inference](a ) .     prior .  interval coverage . using an effect size of 10%
, the plot shows the proportion of simulations in which the pointwise central 95% credible interval contained the true impact , as a function of campaign duration .
intervals should contain ground truth in 95% of simulations , however much uncertainty its predictions may be associated with .
error bars represent 95% credible intervals . ]      to study the properties of our model , we began by considering under what circumstances we successfully detected a causal effect , that is , the statistical power or sensitivity of our approach .
a related property is the probability of _ not _ detecting an absent impact , that is , specificity .
we repeatedly generated data , as described above , under different true effect sizes .
we then computed the posterior predictive distribution over the counterfactuals and recorded whether or not we would have concluded a causal effect .
for each of the effect sizes 0% , 0.1% , 1% , 10% and 100% , a total of @xmath174 simulations were run .
this number was chosen simply on the grounds that it provided reasonably tight intervals around the reported summary statistics without requiring excessive amounts of computation . in each simulation , we concluded that a causal effect was present if and only if the central 95% posterior probability interval of the cumulative effect excluded zero .
the model used throughout this section comprised two structural blocks .
the first one was a local level component .
we placed an inverse - gamma prior on its diffusion variance with a prior estimate of @xmath175 and a prior sample size @xmath176 .
the second structural block was a dynamic regression component .
we placed a gamma prior with prior expectation @xmath177 on the diffusion variance of both regression coefficients . by construction
, the outcome variable did not exhibit any local trends or seasonality other than the variation conveyed through the covariates .
this obviated the need to include an explicit local linear trend or seasonality component in the model .
in a first analysis , we considered the empirical proportion of simulations in which a causal effect had been detected .
when taking into account only those simulations where the true effect size was greater than zero , these empirical proportions provide estimates of the sensitivity of the model w.r.t . the process by which the data were generated .
conversely , those simulations where the campaign had no effect yield an estimate of the model s specificity . in this way
, we obtained the power curve shown in figure  [ fig : apps : synthetic : inference](b ) .
the curve shows that , in data such as these , a  market perturbation leading to a lift no larger than 1% is missed in about 90% of cases .
by contrast , a perturbation that lifts market activity by 25% is correctly detected as such in most cases .    in a second analysis , we assessed the coverage properties of the posterior probability intervals obtained through our model .
it is desirable to use a diffuse prior on the local level component such that central 95% intervals contain ground truth in about 95% of the simulations .
this coverage frequency should hold regardless of the length of the campaign period .
in other words , a longer campaign should lead to posterior intervals that are appropriately widened to retain the same coverage probability as the narrower intervals obtained for shorter campaigns .
this was approximately the case throughout the simulated campaign [ figure  [ fig : apps : synthetic : inference](c ) ] .      to study the accuracy of the point estimates supported by our approach , we repeated the preceding simulations with a fixed effect size of 10% while varying the length of the campaign . when given a quadratic loss function ,
the loss - minimizing point estimate is the posterior expectation of the predictive density over counterfactuals .
thus , for each generated data set @xmath178 , we computed the expected causal effect for each time point , @xmath179 to quantify the discrepancy between estimated and true impact , we calculated the absolute percentage estimation error , @xmath180 this yielded an empirical distribution of absolute percentage estimation errors [ figure  [ fig : apps : synthetic : acc](a ) , blue ] , showing that impact estimates become less and less accurate as the forecasting period increases . this is because , under the local linear trend model in ( [ eq : local - linear - trend ] ) , the true counterfactual activity becomes more and more likely to deviate from its expected trajectory .    2 s.e.m . )
at which predictions become less accurate as the length of the counterfactual forecasting period increases ( blue ) .
the well - behaved decrease in estimation accuracy breaks down when the data are subject to a sudden structural change ( red ) , as simulated for 1  april 2014 .
illustration of a structural break .
the plot shows one example of the time series underlying the red curve in . on 1  april 2014 ,
the standard deviation of the generating random walk of the local level was tripled , causing the rapid decline in estimation accuracy seen in the red curve in . ]
it is worth emphasising that all preceding results are based on the assumption that the model structure remains intact throughout the modelling period . in other words , even though the model is built around the idea of multiple ( nonstationary ) components ( i.e. , a time - varying local trend and , potentially , time - varying regression coefficients ) , this structure itself remains unchanged .
if the model structure does change , estimation accuracy may suffer .
we studied the impact of a changing model structure in a second simulation in which we repeated the procedure above in such a way that 90  days after the beginning of the campaign the standard deviation of the random walk governing the evolution of the regression coefficient was tripled ( now 0.03 instead of 0.01 ) . as a result , the observed data began to diverge much more quickly than before .
accordingly , estimations became considerably less reliable [ figure  [ fig : apps : synthetic : acc](a ) , red ] .
an example of the underlying data is shown in figure  [ fig : apps : synthetic : acc](b ) .
the preceding simulations highlight the importance of a model that is sufficiently flexible to account for phenomena typically encountered in seasonal empirical data .
this rules out entirely static models in particular ( such as multiple linear regression ) .
to illustrate the practical utility of our approach , we analysed an advertising campaign run by one of google s advertisers in the united states .
in particular , we inferred the campaign s causal effect on the number of times a user was directed to the advertiser s website from the google search results page .
we provide a brief overview of the underlying data below [ see @xcite for additional details ] .
the campaign analysed here was based on product - related ads to be displayed alongside google s search results for specific keywords .
ads went live for a period of 6  consecutive weeks and were geo - targeted to a randomised set of 95 out of 190 designated market areas ( dmas ) .
the most salient observable characteristic of dmas is offline sales . to produce balance in this characteristic , dmas were first rank - ordered by sales volume .
pairs of regions were then randomly assigned to treatment / control .
dmas provide units that can be easily supplied with distinct offerings , although this fine - grained split was not a requirement for the model .
in fact , we carried out the analysis as if only one treatment region had been available ( formed by summing all treated dmas ) .
this allowed us to evaluate whether our approach would yield the same results as more conventional treatment - control comparisons would have done .
the outcome variable analysed here was search - related visits to the advertiser s website , consisting of organic clicks ( i.e. , clicks on a search result ) and paid clicks ( i.e. , clicks on an ad next to the search results , for which the advertiser was charged ) . since paid clicks were zero before the campaign , one might wonder why we could not simply count the number of paid clicks after the campaign had started .
the reason is that paid clicks tend to cannibalise some organic clicks . since we were interested in the net effect , we worked with the total number of clicks .
the first building block of the model used for the analyses in this section was a local level component .
for the inverse - gamma prior on its diffusion variance we used a prior estimate of @xmath181 and a prior sample size @xmath176 .
the second structural block was a static regression component .
we used a spike - and - slab prior with an expected model size of @xmath182 , an expected explained variance of @xmath183 and 50 prior @xmath184 .
we deliberately kept the model as simple as this .
since the covariates came from a randomised experiment , we expected them to already account for any additional local linear trends and seasonal variation in the response variable . if one suspects that a more complex model might be more appropriate , one could optimise model design through bayesian model selection . here
, we focus instead on comparing different sets of covariates , which is critical in counterfactual analyses regardless of the particular model structure used .
model estimation was carried out using 10,000 mcmc samples .
we began by applying the above model to infer the causal effect of the campaign on the time series of clicks in the treated regions .
given that a set of unaffected regions was available in this analysis , the best possible set of controls was given by the untreated dmas themselves ( see below for a comparison with a purely observational alternative ) .
as shown in figure  [ fig : apps : empirical : forecast : original](a ) , the model provided an excellent fit on the pre - campaign trajectory of clicks ( including a spike in `` week @xmath185 '' and a dip at the end of `` week @xmath50 '' ) .
following the onset of the campaign , observations quickly began to diverge from counterfactual predictions : the actual number of clicks was consistently higher than what would have been expected in the absence of the campaign .
the curves did not reconvene until one week after the end of the campaign .
subtracting observed from predicted data , as we did in figure  [ fig : apps : empirical : forecast : original](b ) , resulted in a posterior estimate of the incremental lift caused by the campaign .
it peaked after about three weeks into the campaign , and faded away after about one week after the end of the campaign .
thus , as shown in figure  [ fig : apps : empirical : forecast : original](c ) , the campaign led to a sustained cumulative increase in total clicks ( as opposed to a mere shift of future clicks into the present or a pure cannibalization of organic clicks by paid clicks ) . specifically , the overall effect amounted to 88,400 additional clicks in the targeted regions ( posterior expectation ; rounded to three significant digits ) , that is , an increase of @xmath186 , with a central 95% credible interval of @xmath187 $ ] .    to validate this estimate
, we returned to the original experimental data , on which a conventional treatment - control comparison had been carried out using a two - stage linear model [ @xcite ] .
this analysis had led to an estimated lift of 84,700 clicks , with a 95% confidence interval for the relative expected lift of @xmath188 $ ] .
thus , with a deviation of less than 5% , the counterfactual approach had led to almost precisely the same estimate as the randomised evaluation , except for its wider intervals .
the latter is expected , given that our intervals represent prediction intervals , not confidence intervals . moreover , in addition to an interval for the sum over all time points , our approach yields a full time series of pointwise intervals , which allows analysts to examine the characteristics of the temporal evolution of attributable impact .
the posterior predictive intervals in figure  [ fig : apps : empirical : forecast : original](b ) widen more slowly than in the illustrative example in figure  [ fig : intro : illustration ] .
this is because the large number of controls available in this data set offers a much higher pre - campaign predictive strength than in the simulated data in figure  [ fig : intro : illustration ] .
this is not unexpected , given that controls came from a randomised experiment , and we will see that this also holds for a subsequent analysis ( see below ) that is based on yet another data source for predictors .
a consequence of this is that there is little variation left to be captured by the random - walk component of the model .
a reassuring finding is that the estimated counterfactual time series in figure  [ fig : apps : empirical : forecast : original](a ) eventually almost exactly rejoins the observed series , only a few days after the end of the intervention .
an important characteristic of counterfactual - forecasting approaches is that they do not require a setting in which a set of controls , selected at random , was exempt from the campaign .
we therefore repeated the preceding analysis in the following way : we discarded the data from all control regions and , instead , used searches for keywords related to the advertiser s industry , grouped into a handful of verticals , as covariates . in the absence of a dedicated set of control regions , such industry - related time series can be very powerful controls , as they capture not only seasonal variations but also market - specific trends and events ( though not necessarily advertiser - specific trends ) .
a major strength of the controls chosen here is that time series on web searches are publicly available through google trends ( http://www.google.com/trends/ ) .
this makes the approach applicable to virtually any kind of intervention . at the same time , the industry as a whole is unlikely to be moved by a single actor s activities .
this precludes a positive bias in estimating the effect of the campaign that would arise if a covariate was negatively affected by the campaign .
as shown in figure  [ fig : apps : empirical : forecast : observational ] , we found a cumulative lift of 85,900 clicks ( posterior expectation ) , or @xmath189 , with a @xmath190 $ ] interval .
in other words , the analysis replicated almost perfectly the original analysis that had access to a randomised set of controls .
one feature in the response variable which this second analysis failed to account for was a spike in clicks in the second week before the campaign onset ; this spike appeared both in treated and untreated regions and appears to be specific to this advertiser .
in addition , the series of point - wise impact [ figure  [ fig : apps : empirical : forecast : observational](b ) ] is slightly more volatile than in the original analysis ( figure  [ fig : apps : empirical : forecast : original ] ) . on the other hand , the overall point estimate of 85,900 , in this case ,
was even closer to the randomised - design baseline ( 84,700 ; deviation ca .
1% ) than in our first analysis ( 88,400 ; deviation ca .
4% ) . in summary ,
the counterfactual approach effectively obviated the need for the original randomised experiment .
using purely observational variables led to the same substantive conclusions .    ) that had been based on a randomised design . ]      to go one step further still , we analysed clicks in those regions that had been exempt from the advertising campaign .
if the effect of the campaign was truly specific to treated regions , there should be no effect in the controls . to test this , we inferred the causal effect of the campaign on _ unaffected _ regions , which should _ not _ lead to a significant finding . in analogy with our second analysis
, we discarded clicks in the treated regions and used searches for keywords related to the advertiser s industry as controls .    .
time series of clicks to the advertiser s website .
pointwise ( daily ) incremental impact of the campaign on clicks .
cumulative impact of the campaign on clicks . ]
as summarised in figure  [ fig : apps : empirical : forecast : null ] , no significant effect was found in unaffected regions , as expected .
specifically , we obtained an overall nonsignificant lift of @xmath191 in clicks with a central 95% credible interval of @xmath192 $ ] .    in summary ,
the empirical data considered in this section showed : ( i )  a clear effect of advertising on treated regions when using randomised control regions to form the regression component , replicating previous treatment - control comparisons ( figure  [ fig : apps : empirical : forecast : original ] ) ; ( ii )  notably , an equivalent finding when discarding control regions and instead using observational searches for keywords related to the advertiser s industry as covariates ( figure  [ fig : apps : empirical : forecast : observational ] ) ; ( iii )  reassuringly , the absence of an effect of advertising on regions that were not targeted ( figure  [ fig : apps : empirical : forecast : null ] ) .
the increasing interest in evaluating the incremental impact of market interventions has been reflected by a growing literature on applied causal inference . with the present paper we are hoping to contribute to this literature by proposing a bayesian state - space model for obtaining a counterfactual prediction of market activity .
we discuss the main features of this model below .
in contrast to most previous schemes , the approach described here is fully bayesian , with regularizing or empirical priors for all hyperparameters .
posterior inference gives rise to complete - data ( smoothing ) predictions that are only conditioned on past data in the treatment market and both past and present data in the control markets .
thus , our model embraces a dynamic evolution of states and , optionally , coefficients ( departing from classical linear regression models with a fixed number of static regressors ) , and enables us to flexibly summarise posterior inferences .
because closed - form posteriors for our model do not exist , we suggest a stochastic approximation to inference using mcmc . one convenient consequence of this is that we can reuse the samples from the posterior to obtain credible intervals for all summary statistics of interest .
such statistics include , for example , the average absolute and relative effect caused by the intervention as well as its cumulative effect .
posterior inference was implemented in c@xmath193 and r and , for all empirical data sets presented in section  [ sec : apps : empirical ] , took less than 30  seconds on a standard linux machine .
if the computational burden of sampling - based inference ever became prohibitive , one option would be to replace it by a variational bayesian approximation [ see , e.g. , @xcite ] .
another way of using the proposed model is for power analyses . in particular ,
given past time series of market activity , we can define a point in the past to represent a hypothetical intervention and apply the model in the usual fashion . as a result
, we obtain a measure of uncertainty about the response in the treated market after the beginning of the hypothetical intervention .
this provides an estimate of what incremental effect would have been required to be outside of the 95% central interval of what would have happened in the absence of treatment .
the model presented here subsumes several simpler models which , in consequence , lack important characteristics , but which may serve as alternatives should the full model appear too complex for the data at hand .
one example is classical multiple linear regression . in principle , classical regression models go beyond difference - in - differences schemes in that they account for the full counterfactual trajectory .
however , they are not suited for predicting stochastic processes beyond a few steps .
this is because ordinary least - squares estimators disregard serial autocorrelation ; the static model structure does not allow for temporal variation in the coefficients ; and predictions ignore our posterior uncertainty about the parameters .
put differently : classical multiple linear regression is a special case of the state - space model described here in which ( i )  the gaussian random walk of the local level has zero variance ; ( ii )  there is no local linear trend ; ( iii )  regression coefficients are static rather than time - varying ; ( iv )  ordinary least squares estimators are used which disregard posterior uncertainty about the parameters and may easily overfit the data .
another special case of the counterfactual approach discussed in this paper is given by synthetic control estimators that are restricted to the class of convex combinations of predictor variables and do not include time - series effects such as trends and seasonality [ @xcite ] .
relaxing this restriction means we can utilise predictors regardless of their scale , even if they are negatively correlated with the outcome series of the treated unit .
other special cases include autoregressive ( ar ) and moving - average ( ma ) models .
these models define autocorrelation among observations rather than latent states , thus precluding the ability to distinguish between state noise and observation noise [ @xcite ] .    in the scenarios we consider ,
advertising is a planned perturbation of the market .
this generally makes it easier to obtain plausible causal inferences than in genuinely _
observational _ studies in which the experimenter had no control about treatment [ see discussions in @xcite , robinson , mcnulty and krasno ( @xcite ) , @xcite ] . the principal problem in observational studies is endogeneity : the possibility that the observed outcome might not be the result of the treatment but of other omitted , endogenous variables . in principle , propensity scores can be used to correct for the selection bias that arises when the treatment effect is correlated with the likelihood of being treated [ @xcite ] . however , the propensity - score approach requires that exposure can be measured at the individual level , and it , too , does not guarantee valid inferences , for example , in the presence of a specific type of selection bias recently termed `` activity bias '' [ @xcite ] .
counterfactual modelling approaches avoid these issues when it can be assumed that the treatment market was chosen at random .
overall , we expect inferences on the causal impact of designed market interventions to play an increasingly prominent role in providing quantitative accounts of return on investment [ @xcite ] .
this is because marketing resources , specifically , can only be allocated to whichever campaign elements jointly provide the greatest return on ad spend ( roas ) if we understand the causal effects of spend on sales , product adoption or user engagement . at the same time
, our approach could be used for many other applications involving causal inference .
examples include problems found in economics , epidemiology , biology or the political and social sciences . with the release of the causalimpact r package we hope to provide a simple framework serving all of these areas .
structural time - series models are being used in an increasing number of applications at google , and we anticipate that they will prove equally useful in many analysis efforts elsewhere .
the authors wish to thank jon vaver for sharing the empirical data analysed in this paper .