Bayesian MMM’s Token and Context Window Tax
Everyone debates Bayesian vs Frequentist MMM on statistical grounds. Now this debate has moved to a new turf – AI.
Last week I wrote about Claude code and Jevons Paradox in MMM. I think regardless of all the warnings, people will use Claude or similar LLMs to write MMM code.
This brings us to the question- Bayesian MMM or Frequentist MMM?
At least from a statistical point of view it has been clearly proven that Frequentist MMM is the superior one. Every Bayesian MMM zeroes in on a point estimate eventually anyways (see link to post in comments).
📌 What are Tokens and Context Window
Tokens:
You can think of tokens as the units of information an LLM (like Claude) reads and processes.
A token can be a word, part of a word, or just a symbol. Code, numbers, logs, comments all count as tokens.
In MMM context:
When you work with MMM in an LLM, tokens include: Model code (R/Python), coefficients, diagnostics, your prompts + model responses.
Example:
(beta_tv = 0.12) consumes multiple tokens
A full Bayesian trace summary could consume hundreds to thousands of tokens !!
Context Window:
I am simplifying but the context window is the total amount of tokens the model can “remember” at one time. It is like a working short term memory
Again in MMM, it includes everything in the conversation:
Your earlier prompts, Model code, Debugging steps, Outputs, Explanations etc.
If you exceed the context window, Older context may get dropped. Hence it is always better to have a huge Context window.
📌 The Bayesian MMM’s Token and Context Window Tax
A lot of people believe the interpretation is easier in Bayesian framework. Sure, but that interpretation rarely means real life utility.
Because of ‘Interpretation is easier’, people often overlooked the complexity of specifying a Bayesian Model.
But not anymore.
In the era of LLM’s where the currencies are Tokens and Context Window, one can clearly see that the Bayesian complexity costs us.
Bayesian MMMs end up costing 2-3x more tokens and context window than Frequentist MMMs.
📌 Why do I call this a tax?
Well it is a tax for choosing complexity over simplicity.
It is a tax for avoiding modeling scrutiny (“How did you choose the prior? Can you justify it”?) and hiding behind a false scientific aura.
It is a tax for quantifying uncertainty where none really existed.
You can choose whether to pay this tax. Tax could be subsidized. But the Tax never goes away. Somebody pays for it.
As someone who has built both NLP products and MMM models, I guess I have a good vantage point to guide.
So be it Statistical or AI front, Frequentist Methods are a more prudent option. I know a ton of good statisticians who can help you get the intuition of Frequentism. I guess that would be far less a cost than paying more for complexity.
LLMs will lead more MMM models but as a paradox the need for genuine MMM vendors will only grow.
