The USS Scorpion Story: A Bayesian Myth That Refuses to Die
I have seen a lot of debates on Frequentist vs Bayesian. Bayesians often bring up the USS Scorpion as a success story of Bayesian method application in real world.
For context: USS Scorpion- a US nuclear submarine, imploded in the Atlantic in 1968, triggering an urgent search for its wreckage.
📌 The Exaggeration of Bayesian Success
The success of Bayesian methods is often exaggerated and mythologized in the case of the USS Scorpion. Firstly, the search area was already significantly narrowed down using hydroacoustic signals.
This happened before any meaningful application of Bayesian methods.
In a way, the hardest part of the problem i.e. “where to look” was largely solved by physics and signal intelligence !!
📌 The Bayesian application outcome
Dr. Kristin Lennox in her famous talk (link in comments 43.40 -49.00) illustrates how the Bayesian method was applied.
The Bayesian methods were used to do 4 things primarily.
– Combine multiple hypotheses
– Assign probabilities across the search space
– Identify a “highest probability region” and search
– Update probabilities
Sounds like a very scientific way to go about things. But guess where the submarine was found? It was found right outside the ‘highest probability region’.
📌 Discretization of probability space the issue?
Dr. Kristin Lennox puts the blame on discretization of probability distribution for ‘not finding the submarine’ quickly enough.
But discretization or not, the concept of a “highest probability region” still exists. Whether drawn as a box or a smooth contour, the model still implied: “search here first” and yet the actual outcome fell outside that implied priority zone. To me this is a failure of Bayesian Method.
Even after many years we still have not answered two fundamental questions.
– Can we really encode our subjective beliefs into a probability distribution?
– How good are we in doing that?
📌 The Bayesian MMM parallel
Even in MMM (Marketing Mix modeling), Bayesian methods steal a lot of credit. Much of the success / failure of real attribution identification depends on the prior definition stage. Meta ROI range 4.3 – 5.4, TV ROI range 2.3 – 4.3 etc.
The parameter search space is already narrowed down by the client much like acoustic signals narrowed the search space for the USS scorpion search.
In Bayesian MMM, ROI ranges come from (or are approved by) clients. So when attribution looks right (seldom the case with Bayesian methods but still), the credit should go to them.
The clients unknowingly pay the vendor while they did the heaviest work 😅.
May be clients should set different pricing plans – one where they supply the prior ROI and contribution numbers and one where the vendor comes with the number on their own🤔.
Before celebrating any Bayesian success story, pls go through the intricate details. Most often you will find it taking up more credit than it actually deserved.
