I am not sure it counts as "research" but I spent part of the morning touring the future space of the NYU Center for Data Science, currently occupied by Forbes Magazine. The space is excellent, and can be renovated to meet our needs beautifully. The real question is whether we can understand our needs faster than the design schedule.
In the afternoon, Foreman-Mackey and I discussed the difference between frequentist and Bayesian estimates of parameter uncertainty. There are regimes in which they agree, and we couldn't quite agree on what those are. Certainly in the super-restrictive case of Gaussian-shaped likelihood function (Gaussian-shaped in parameter space), and (relatively) uninformative priors, the uncertainty estimates converge. But I think the convergence is more general than this.
"Certainly in the super-restrictive case of Gaussian-shaped likelihood function (Gaussian-shaped in parameter space), and (relatively) uninformative priors, the uncertainty estimates converge. But I think the convergence is more general than this."
ReplyDeleteWhy? I'm fairly sure this is it, but I could be wrong. It's just not as restrictive of a case as you might think, these conditions are extremely common.
"There are regimes in which they agree, and we couldn't quite agree on what those are."
ReplyDeleteWell, you could ask statisticians, who have explored this quite a bit. It's well understood in the setting of parametric models (probably since the 1950s, certainly since the 1970s). Nuisance parameters can complicate things. In nonparametric settings it's an open research question.
The Bernstein-von Mises theorem is the basis for a formal, general statement of the asymptotic correspondence in parametric settings (i.e., where the likelihood is approximately normal). However, there are other settings where there is a correspondence. For example, confidence and credible regions correspond exactly for a Poisson rate when you have the right prior (uniform, I think); Jaynes has a nice discussion of this in his 1976 paper "Confidence intervals vs. Bayesian intervals" (paper 32 here: http://bayes.wustl.edu/etj/node1.html). I dimly recall seeing a more general discussion of this kind of correspondence in exponential families, but I don't recall the details.
The BvM theorem is about approximate matching of credible and confidence regions. The literature on "probability matching priors" is about how to choose a prior to make the correspondence exact or agree to higher order than sqrt(N). Reference priors (and other "objective Bayes" tools) also often give higher order convergence, although they don't explicitly seek this.
The situation in nonparametric settings is much more complicated. Roughly speaking, a nonparametric model lets the "effective" number of parameters grow without bound as N (# of data) grows, and in such settings the likelihood will typically never swamp the prior the way it does in parametric settings. Early results on frequentist behavior of early nonparametric Bayesian methods were discouraging. With some theoretically appealing priors, consistency (appropriate convergence to the truth as N->infinity) was not guaranteed, and there could even be settings where credible regions could have vanishingly small asymptotic coverage. Again roughly speaking, intuition fails in large dimensional spaces, and nonparametric priors that seemed innocuous were in fact putting all their mass in weird places. More recently Subhashis Ghosal, Judith Rousseau, Aad van der Vaart, and others have been finding settings where nonparametric Bayes has appealing frequentist properties (e.g., Gaussian processes with the "right" priors, and mixtures with the "right" priors, can behave well). E.g., Judith has studied nonparametric settings where a generalization of the BvM theorem holds.
A good paper for entering into this literature is a 2004 review by Berger & Bayarri: "The Interplay of Bayesian and Frequentist Analysis" (http://projecteuclid.org/euclid.ss/1089808273). In the 10 years since that paper, there has been a lot of work on nonparametric settings, but I don't know of any recent reviews of this work.