Lang and I stopped in to see Phil Gregory at UBC, who has been writing good stuff about Bayesian methods for exoplanet discovery. In the conversation I sharpened up my objections to Bayesian-evidence-based model selection as it is done in practice. It could be done well in principle but that would require properly informed priors. If the priors are uninformative
, small changes in the outskirts of the prior-allowed regions can have enormous effects on the evidence integrals.
2010-12-08
Phil Gregory
Subscribe to:
Post Comments (Atom)
What are some alternatives to the Bayesian Evidence? I've heard of things like BIC and DIC, but aren't those ad hoc solutions?
ReplyDeleteYes, AIC, BIC, and so on are ad-hoc.
ReplyDeleteBut *any* decision method within Bayesianism is ad-hoc, because Bayes tells you how to assign *probabilities* only. A real Bayesian never decides. If decision is necessary, a utility must be specified. So the evidence calculation is only part of the story. But yes, evidence is perfect if the prior is informative. At the same time it is totally wrong if the prior was pulled out of a hat. There are non-Bayesian but also no-free-parameters ways of deciding like cross-validation. My point is not that evidence is wrong, just that it has never been calculated correctly.
I agree that it is very easy to do Bayesian model selection badly. The key is to assign good priors for parameters given the models. One way of improving this is to pay close attention to P(D|M), the predictive distribution for the data before you knew it.
ReplyDelete