How Train – Test Validation Quietly Changes the Philosophy of MMM
How Train – Test Validation Quietly Changes the Philosophy of MMM Yesterday, I interviewed a candidate for an MMM role from a rival MMM company. Initially, I was genuinely impressed. They spoke about a fairly rigorous frequentist setup and mentioned several checks beyond just R squared to evaluate model quality. That was refreshing to hear because MMM quality cannot be judged by R squared alone. But then the candidate mentioned that they perform the exact same checks on the test dataset as well !! I was a bit taken aback. I further asked what they would do if the prediction accuracy went down in the test data (which almost always