Parameter Recovery Deep Dive
A parameter recovery exercise is an incredibly important exercise for validating the internal consistency of a causal inference model. A parameter refers to a variable in the model whose value we infer using data. These parameters form the internal machinery of the model, governing key components of the model such as each channel’s time-shifted impact, a context variable’s effect, or the latent baseline (non-marketing) effect that drives the KPI over time.
Before we jump into how to interpret the parameter recovery information in Recast, it’s good to get a general understanding of what parameter recovery is and why it matters.
In short, the goal of a parameter recovery exercise is to simulate data where we know the true relationships between all of the variables, and then we want to put our model to the test to see if it gets the right answer when we run the simulated data through the model. We have more detailed discussion of the purpose of these checks at the following links:
Overview of parameter recovery
Doing the work of simulating parameters, generating data, running the model, and then comparing the modeled results to the simulated parameters is a lot of work, so Recast has built tools to do this for you. When you launch a “parameter recovery” run via the Recast “Launch Run” page, under the hood Recast is going to:
For each parameter in the model, choose a random value drawn from the prior you set
Use those parameter values to generate data for
The dependent variable / KPI variable as well as
Any lower-funnel channels
Recast will then discard the original KPI and lower-funnel channel data, and replace them with the simulated values from step 2. The resulting synthetic dataset will include:
actual marketing spend,
Actual contextual variables, and
simulated KPI and lower-funnel channel data.
Recast will launch a “model run” using the data from step 3 and the priors you set.
Once the run has completed, Recast will compare the parameters estimated via the model in step 4 with the parameters generated in step 1.
This information is summarized in the “parameter recovery dashboard”
When interpreting the parameter recovery dashboard, it’s critical to keep in mind that we are looking at the results of a real model run on simulated data.
So if you look at the results and say “wow the intercept is way too high, that doesn’t make sense for my business” then you’re interpreting the results incorrectly! We should expect that the results we see in the parameter recovery don’t necessarily match business reality because all we care about checking is if the modeled results match the simulated parameters!
So for example when we look at the following graph, the black line is the simulated value for the intercept. That is to say – it’s completely made up! It was randomly generated from within the prior (yellow region) so there’s no reason to spend any time trying to interpret the black line.
What we care about is the blue shaded region and the blue line. This area represents the model’s estimates and here we can see that the model does a pretty good job of following the actuals, although in the last hundred days of data the model does miss a bit.
We can review these same plots for all of the different parameters in the model, which can help give us a sense for how often we can expect the model to be “wrong” once we start using it with actual data to make real world decisions.
One example of this has to do with time shifts. In general, media mix models don’t work with super-long time shifts because as the time-shift gets longer the amount of signal associated with any given change to marketing budgets is reduced. This parameter recovery exercise can help us understand how much of a problem that is! If our model is struggling to recover ROIs with the time-shift priors we have, we know that with the amount of spend variation in the data and the assumed time-shifts that the model’s results simply won’t be reliable when we try to use it on real (not simulated) data.