Reading Professor Art Owen’s experimentation class notes today (recommended by the author of gave me a new understanding of experimentation and attribution, by comparing and contrasting them together. And why attribution is hard, if not impossible, to solve.
Experimentation (randomized controlled experiments)and attribution both make causality inferences.
Experimentation makes inferences about the effect of causes: if we reduce the steps needed to check out on our eCommerce site, the purchase rate will increase. We could investigate by creating two versions of the checkout process, randomly assign to users, and compare the results between two groups.
Attribution, on the other hand, make inferences about the causes of effects: now we see the purchase rate has increased, we look backward for the cause(s). You can immediately see why it’s nearly impossible to solve. There could be so many reasons: campaigns, new designs, seasons, etc. Many we don’t have data on, and realistically, many unknown (e.g. word of mouth).
Even if we could nail down the complete 10 potential causes. Some could change the purchase rate on their own while others only have an impact when paired with other causes. And even when all 2¹⁰ causal possibilities are known without error, attribution remains hard to compute.
Related to this post:
