Why you should not calibrate MMM models through experiments

Marketing Mix Modelling (MMM) is a statistical method used to measure the impact of marketing activities and optimize budget allocation across various channels. With a decline of cookie-based tracking and increasing privacy concerns, MMM has become more popular as a privacy-friendly solution for understanding marketing effectiveness. As MMM techniques evolve, there’s been a surge in companies and innovations focused on improving model accuracy. A critical part of this process is calibration—fine-tuning model parameters to match real-world data. This is different from validation, which tests how well a model generalizes to new, unseen data. Calibration ensures the model’s predictions are consistent with observed outcomes, while validation assesses robustness. A common belief is that experiments (e.g., Incrementality tests, Geo Tests) can be used to calibrate MMM models. However, our findings show that relying on experiments for calibration can worsen accuracy, leading to biased insights. Through this paper, our aim is to provide the MMM community with better methods for calibration, helping marketers make more reliable, data-driven decisions.