2009-09-14

incremental data

In many of the problems we are interested in, we would like to have a chain or sampling of models that are consistent with all data so far and then, as new data come in, we would like to trim the chain of models that the new data rule out (really disfavor), and then extend the chain with new models that are consistent with everything. I have a strong intuition that we can do this without re-starting from scratch. I discussed this with Lang, and a short-term goal is to convert the exoplanet project over to this mode.

This idea is not unrelated to the fact that as new data come in, you can treat them as an entirely new experiment, for which the prior probability distribution is the posterior probability distribution from the previous data.

1 comment:

  1. In machine learning this setting of data arriving in a stream is called "online" learning (as opposed to, say, "batch" learning).

    ReplyDelete