Causal Experiments Don’t Give a Holistic Picture of Incrementality
Couple of days ago, I came across a interesting post in which the following claims were made.
1) In case of MMM, Contribution ≠ Incrementality
2) MMM and Causal Experiments should hone in on ‘one truth’
3) Experiments provide a narrow but stronger causal identification.
I wouldn’t have written this post, but the OP blocked me and deleted my comments for politely disagreeing and pointing out errors in their LLM generated post and LLM generated rebuttals.
So let me provide my POV on the above.
📌 Contribution ≠ Incrementality
MMM is implicitly causal and the biggest incrementality test one can do. Therefore in case of MMM, Contribution = Incrementality.
In case of MMM, contribution naturally is incrementality because the base answers the question of brand equity or organic sales.
The rest of the line items (media/marketing variables) is what incrementality is.
(See the detailed post in comments.)
📌 Incrementality by MMM ≠ Incrementality by Causal Methods
As mentioned above, the incrementality of MMM answers the question “what would be my organic sales due to brand equity or brand goodwill, if I turn off all my marketing / media spends”.
The ‘incrementality’ answered by causal experiment is not holistic. It just answers the question ‘what would have happened, if I didn’t spend on that particular channel’.
This is more of a ‘counterfactual’ answer rather than an incrementality answer.
Causal methods don’t prove ‘true’ incrementality. They just validate the ‘effect’ at play.
For incrementality (the one defined in marketing), MMM is your best bet.
📌 MMM and Causal Experiments won’t point to the same truth.
MMM and Causal experiments won’t point to the same ‘truth’. They measure different things. Causal Experiments give you estimands like ATE, ATT whereas MMM gives you the ‘conditional mean’ i.e. what is the effect of your one channel given that all other channels are at play in the market. The effect of any one channel is ‘conditioned’ on other channels.
Causal experiments predominantly are uni-variable in nature and doesn’t account for other channel’s effect explicitly. The moment they start accounting for it, they become similar to MMM sans the adstock
transformations.
📌 Experiments provide a narrow but stronger causal identification.
No, not all experiments are causal. And even when they are, they are no match to the larger causal identification of MMM.
(See the detailed post in comments).
P.S: I am not against the use of LLMs. LLMs are powerful, but one must have strong foundational knowledge about the subject at hand to discern whether the LLM is spewing garbage or providing accurate answers. LLMs can often play both sides, they don’t have any subject conviction. It is for this reason, we at Aryma Labs built our own RAG based MMMGPT trained on our own resources and having strong guard rails on what is generated or retrieved.
