In the morning I woke up deciding that I had absolutely nothing new or interesting to say about model comparison (which I cast as a decision-making problem). By the end of the day I was writing furiously on it. I guess I realized that I might as well write down what I know and see if it stands on its own. It probably doesn't. I think the new things I have to add (maybe) are (a) pragmatic considerations about computation and precision and (b) an argument that you can never know your utility precisely (and rarely your priors). My point is that if you don't know the inputs precisely, there is no reason to do a huge computation. This has the consequence that if you are looking to split models that are close
in marginalized likelihood you need to know a lot of unknowables precisely and do a lot of computation. If you are looking to split models that are far apart, you don't need to do anything serious; the difference will be obvious.
ps. If you want to know why you can never know your utility precisely, it is because it necessarily involves long-term considerations. The long term
is the period over which all variables and assumptions can change. So you never have a precise model of it.
This reminds me of a talk Skilling gave where he mentioned what kind of accuracy you might need for log(Z). The usual story is that you need to get down to ~0.1 accuracy because that's about how precise our probability judgments usually are. But in big problems the prior probability of a "close" evidence ratio is really tiny. You'd be very unlucky to get a dataset that was so borderline.
ReplyDelete