what is a good method?

A day of mainly writing: Alex Malz nearly finished a NASA graduate fellowship proposal; I put comments on pages from Dun Wang's CPM paper; and I closed issues open on my MCMC tutorial. I had a long discussion with Tony Butler-Yeoman (Wellington) and Marcus Frean (Wellington) about our Oddity method for detecting anomalies (like astronomical sources) in imaging. They asked me two very good questions about writing for astronomers: How do you demonstrate to astronomers that this is a useful method that they want to try—with a few good examples or a large-scale statistical test? And how do you write a methods paper in astrophysics?

On the latter, I advised our new methods-paper template, which is this: Introduction, then a full statement of all of the assumptions underlying the method. Then a demonstration that the method is best or very good under those assumptions (using fake data or analytical arguments). Then a demonstration that the method is okay on real data. Then a discussion, in which the assumptions are addressed, one by one: This permits discussion of advantages, disadvantages, limitations, and places where improvements are possible. The key idea of all this is that a good method should be the best possible method under some set of identifiable assumptions. I don't think that's too much to ask of a method (and yet it is not true of most of the things the data-analysis community does these days).