Today was almost all administration, but I did have a great email conversation with Iain Murray about the Comet Holmes project. I was saying that Lang and I are getting biased answers, and his response was: If your model is good, the posterior PDF pretty-much has to include the right answer. That is, you can be biased, but Bayes never confidently rules out the truth. I have many things to say about that, but the blogosphere doesn't have room for it all! He said: Go hierarchical. I said: Argh! (because he is right).
Bayes always confidently rules out the truth if the hypothesis space is continuous. :-) Then, of course, you need more sophisticated definitions. I contend that logarithmic scoring rules are the general answer to these kinds of questions.
ReplyDeleteP. S. It'd be awesome if you came to MaxEnt. It's not far from New York this year! :-)