Good Quasi Causal Models Should have high R Squared Value
A connection forwarded yet another article from PYMC (Benjamin Vincent) asking my take. Seems like this person too has blocked me. So much for PYMC being open source but not open to counter arguments 😅
The article is titled ‘Goodness of fit is not the objective in causal first projects’.
I disagree.
Bayesians generally don’t have a straightforward notion of ‘Goodness of fit (GOF)’. They rely on Posterior Predictive Checks (again in-sample like R²) or Bayesian R².
📌 Traditional R² ≠Bayesian R²
Traditional R² answers how much variance in Y (DV/KPI) is explained by the model’s fitted values.
Hence it is based on:
– A single best-fit estimate of coefficients (β)
– A single predicted value for each observation
– A single number summarizing fit
On the other hand, Bayesian R² answers how much variance is explained by posterior predictive mean.
Here, you don’t get a single value but a distribution of R² values.
Because Bayesians can’t arrive at a single R² (point estimate), it is convenient to dismiss goodness of fit altogether 🙂.
📌 Goodness of fit (R²) should be high for Quasi Causal Models
Many conflate RCTs and Quasi Causal Experiments.
In an RCT, causal identification is secured by design (variables are controlled for with no confounders). Regression and R² only improve estimator efficiency not establish causality.
Also, in RCTs, the estimator is mostly unbiased.
In quasi causal settings, regression is the identification strategy !
Therefore, while low R² does not threaten causal validity in RCTs, it may indicate omitted variable bias (OVB) in observational causal models which is generally untestable (directly).
Assuming the causal model is well specified, a high R² in a quasi causal setup indicates the variables explain variation in the dependent variable better.
Thus, GOF is a necessary (not sufficient) condition for causal identification in quasi causal setups.
Irrelevant variables can inflate R² but the same applies to Bayesian R².
Diagnosing R² inflation is more straightforward in Frequentist setups. In Bayesian models, inflation may arise from variable addition, bad priors, or even multicollinearity (which inflates posteriors – see link in comments).
But to say GOF is not the objective in causal tests is incorrect.
GOF is an objective in quasi causal tests.
How else would we know if the variables explain changes in KPI?
GOF is about retrodiction not prediction.
In Quasi Causal set up – retrodiction of past values is in itself a good sign of causal structure identification.
