don't cut on significance, cut on value

I blogged yesterday about Dan Foreman-Mackey's new six-planet system. He found it, but then rejected it because the transit of the outer planet induces a significant astrometric offset. We figured out (on the board!) that the offset is consistent with small contamination from a nearby fainter star, and concluded that we should be rejecting stars not on the significance of the astrometric signal, but the absolute amplitude of the signal. This is a re-learn of a general lesson: If you use signficances for selection, errors in your selection depend on the errors in your errors and it makes selection a function of those errors. And so on. Usually (but not always) bad. (Or hard to model well.)

No comments:

Post a Comment