Among other discussion topics, Bovy and I debated the usefulness of the fact that if you have N pieces of data about an object, each of which says something about, say, the question of whether that object is a quasar, you can multiply together the N independently calculated likelihoods for quasar and for star. This sounds great, but it only makes sense when the N pieces of data tell you independent things. That is, it is only true if the joint probability of all the data equals the product of the probabilities of each data item separately. This is almost never the case!
In related news, we figured out why Zolotov and I are having trouble computing the Bayes factor for our project: Bayes factors are hard to compute! And they are also very slippery, because a small change to an uninformative prior—one that makes no change whatsoever to the posterior probability distribution—can have a huge impact on the Bayes factor. Once again, I am reminded that model selection is where I part company from the Bayesians. If you don't have proper, informed priors, you can't compute marginalized relative probabilities of qualitatively different models. It is cross-validation and data-prediction only that is reliable in most real situations.
Re: model selection
ReplyDeleteIf the answer depends sensitively on assumptions, I want to know about that, not sweep it under the carpet. I think the only sensible reason to avoid Bayesianity is if the calculations are too hard (which is very common).
Also, most of the "problems" with Bayesian model selection come from the fact that people suck at assigning prior probabilities, not because model selection is somehow fundamentally different from parameter estimation (it isn't).
For example, in the common "one extra parameter" situation, people seem to like assigning delta function + wide uniform distribution as a prior, but if you actually probe their beliefs something like a Cauchy distribution would usually be more appropriate (without the delta function at all).