Blogs

The Prestige-ification of Marketing Measurement – Are you watching closely? Few weeks ago I read a post titled “why is Bayesian MMM so popular”. There are so many articles like that which hype Bayesian methods as though they are a silver bullet. But do you really want to know why Bayesian methods are popular in Marketing Measurement and perhaps in other fields? It is not for the right reasons. One particular quote from the famous movie ‘Prestige’ sums up the reason “Now you’re looking for the secret, but you won’t find it, because of course you’re not really looking. You don’t really want to work it out. You want to

Good Quasi Causal Models Should have high R Squared Value A connection forwarded yet another article from PYMC (Benjamin Vincent) asking my take. Seems like this person too has blocked me. So much for PYMC being open source but not open to counter arguments 😅 The article is titled ‘Goodness of fit is not the objective in causal first projects’. I disagree. Bayesians generally don’t have a straightforward notion of ‘Goodness of fit (GOF)’. They rely on Posterior Predictive Checks (again in-sample like R²) or Bayesian R². 📌 Traditional R² ≠ Bayesian R² Traditional R² answers how much variance in Y (DV/KPI) is explained by the model’s fitted values. Hence

One of the signs that you should not trust a method or its evangelists is that they actively block you just for disagreeing with them (even in the most polite ways). I will never trust Bayesian methods or Bayesians simply for the reason that they simply don’t have stomach for a good debate or a counter argument. The moment they feel they are losing the argument they block you. I know Dr. Juan Orduz might be held in great regard in Bayesian circles, but it is not a sign of a good academic or statistician to block someone just for posing a counter argument. I had written the below for

Prior Predictive Check – Checking your Assumptions against your Assumptions !! A Prior Predictive Check (PPC) is often presented in Bayesian circles as a kind of “sanity test before modeling”. But what it actually does and what people think it does are two very different things. 📌 What is a Prior Predictive Check? Before seeing any data: ▪️You specify priors for your parameters (say in MMM parlance they would be ROI numbers or contribution numbers etc.) ▪️You then Sample parameters only from the priors and push those sampled parameters through your model to simulate outcomes. So far so good. But here comes the catch – how do you validate that

Humans Don’t Think in Probabilities So Bayesians all over the world are celebrating that Elon Musk himself has endorsed “Bayes Theorem or Bayesian Thinking.” But sorry Bayesians, as great an entrepreneur Musk is, he like all humans can’t be right about everything. And lately he has been wrong about a lot of things. I won’t get into politics here and will restrict myself to what I know best – statistics. Humans don’t think in probabilities. We think we do because it makes us look smarter than we actually are. Thinking in probabilities means thinking in ranges, not point estimates. There was a recent example where Uber experimented with a price

90% of the clients already know their top marketing / media channels with out MMM. So why MMM? One Interesting fact that we observed in all the MMM projects that we have delivered so far (almost 350+ Models) is that – Almost 90% of our clients had figured out what their top drivers of the KPI were !! It is characterized by high spends on this channel repeatedly. I am sure other vendors and would have noticed the same too. So the obvious question becomes: If clients already know their top performing channels, why do they need MMM? Ans: Because knowing ‘what works’ is very different from knowing ‘how much

How we reduced MMM’s In-sample MAPE by 4.6 percentage points through RBF technique. Recently we replaced an existing MMM vendor for a prominent client. One of the complaints the client had with the earlier vendor was that – they were not able to capture key events and promotional events accurately. The client had clear data on which weeks they ran the promotional events and which weeks they conducted the key events. However despite the data, the earlier vendor could not capture it comprehensively. We dug into their code and found that they had used the archaic ‘dummy encoding’ to capture the events. 📌 The problem with Dummy Variable (one hot) Encoding

Why the MMM – Experimentation – MTA Triangle is kinda wrong Around a decade ago, I was in between jobs and took on a NLP consulting gig. The task was to develop a Topic Modeling system. It is here that I first learnt about Dirichlet distribution. It is a beautiful concept. During a recent client pitch meeting, we walked through our familiar MMM-Experimentation – Causality triangulation slide and something clicked. Our triangulation diagram reminded me of the Dirichlet allocation. And after thinking about it for days, I realized something uncomfortable. Almost every vendor (including us) draws this as a perfect equilateral triangle. And this might be conceptually wrong !! 📌

How in-sample MAPE can signal a decline in media effectiveness In a well built Marketing Mix Model (MMM) or (any time-series model), parameter identification and predictive accuracy usually improve as more data are added. Temporality wise, an MMM is at its most accurate in the most recent time period than in the beginning. We also published a research paper on the same (link in comments). This is also why for budget optimization one considers the latest 12 months or 6 months data. But there are cases where the model performance (in-sample MAPE) may start to deteriorate in the most latest time periods. Now what gives? When in-sample MAPE starts deteriorating in

Why “Test vs Total” Geo Testing is Misleading and Not Causal. We onboarded a client early this month and we were shocked to find what their earlier experimentation vendor had implemented for them. I always knew SCM experiments are no good and similarly most SCM vendors always try to pull a wool over the eyes. Here is what happened: The client was experimenting running Meta and Google Ads in 10 states in USA. The client was approached by this experimentation vendor and was even given free 6 months trial of their so called sophisticated SCM tool. The client wanted to carry out causal experiments and asked the vendor if they can