At Aryma Labs, we increasingly leverage information theoretic methods over correlational ones. ICYMI, link to the article on the same is in resources.
To set the background, let me explain what is AIC and KL Divergence
๐๐ค๐๐ข๐ค๐ ๐ข๐ง๐๐จ๐ซ๐ฆ๐๐ญ๐ข๐จ๐ง ๐๐ซ๐ข๐ญ๐๐ซ๐ข๐จ๐ง (๐๐๐)
AIC = 2k -2ln(L)
where
k is the number of parameters
L is the likelihood
The underlying principle behind usage of AIC is the ‘Information Theory’.
Coming back to AIC, In the above equation we have the likelihood. We try to maximize the likelihood.
It turns out that, maximizing the likelihood is equivalent of minimizing the KL Divergence.
๐๐ก๐๐ญ ๐ข๐ฌ ๐๐ ๐๐ข๐ฏ๐๐ซ๐ ๐๐ง๐๐?
From an information theory point of view, KL divergence tells us how much information we lost due to our approximating of a probability distribution with respect to the true probability distribution.
๐๐ก๐ฒ ๐ฐ๐ ๐๐ก๐จ๐จ๐ฌ๐ ๐ฆ๐จ๐๐๐ฅ๐ฌ ๐ฐ๐ข๐ญ๐ก ๐ฅ๐จ๐ฐ๐๐ฌ๐ญ ๐๐๐
When comparing models, we choose the models with lowest AIC because in turn it means that the KL divergence also would be minimum. Low AIC score means little information loss.
Now you know how KL divergence an AIC are related and why we choose models with low AIC score.
๐๐๐ฎ๐ญ๐ข๐จ๐ง ๐๐๐จ๐ฎ๐ญ ๐๐๐
One of the misconceptions about AIC is that the AIC helps in choosing the best model out of a given set of models.
However, the key word here is ‘Relative’. AIC helps in choosing the ‘best model’ relative to other models.
For example, if you had 5 MMM models (fitted for same response variable) and all 5 are overfitted badly, then AIC will choose the least overfitted model among all models.
AIC will not caution that all your MMM models are poorly fitted. In a way AIC is like a supremum of a set.
๐๐ ๐๐ข๐ฏ๐๐ซ๐ ๐๐ง๐๐ ๐ญ๐จ ๐ ๐๐ฎ๐ ๐ ๐๐ข๐๐ฌ ๐ข๐ง ๐๐จ๐๐๐ฅ
Another interesting way we leverage KL Divergence is to gauge bias in the model. For a problem like MMM, bias in model is always unwanted.
The model could have bias for variety of reasons – misspecification of model, treatment of multicollinearity through regularization etc. We are doing some interesting research using KL divergence to reduce bias in our models. (More on this soon).
P.S: Useful link to papers in resources.
First image credit in resources.
Resources:
Why Aryma Labs does not rely on Correlation alone
https://open.substack.com/pub/arymalabs/p/why-aryma-labs-does-not-rely-on-correlation?r=2p7455&utm_campaign=post&utm_medium=web
Facts and fallacies of AIC :
https://robjhyndman.com/hyndsight/aic/
Image credit:
https://www.npr.org/sections/thetwo-way/2013/11/16/245607276/howd-they-do-that-jean-claude-van-dammes-epic-split
