modeling everything, ever

In a day in which Lang and I spent an inordinate amount of time formatting our Comet 17P/Holmes paper for error-free submission to the AJ (this was hilarious but not exactly enlightening scientifically), one bright spot was having lunch with Rob Fergus, his student Li Wan, and my student Dan Foreman-Mackey to discuss the first steps towards our probabilistic theory of everything. The short-term goal is to create a quantitative model of every pixel of digital or digitized astronomical imaging ever taken from any facility anywhere at any time in any bandpass. Once that's done, we will think about bigger projects. But seriously, we came up with some (still vague) first steps, one of which is for the computer scientists to read some astronomy papers, and for the astronomers to read some computer-science papers.

Very Wide Field Surveys, day 1

Today was the first day of Very Wide Field Surveys in the Light of Astro2010 in Baltimore. The talks were overwhelmingly extragalactic in emphasis, and there were many about not-yet-taken data. Highlights for me included: Jim Gunn (Princeton) making some of Blanton and my work on galaxies in SDSS his prime motivation for his new projects; Martha Haynes (Cornell) showing examples of dark galaxies from ALFALFA that contain substantial HI gas but no stars to the limits of very deep optical imaging; Matt Jarvis (Hertfordshire) explaining that LOFAR and similar surveys record the amplitudes and phases from many antennae, but then delete them after map-making, so certain kinds of reanalyses will be impossible, no matter what; Chris Martin (Caltech) showing one of the eclipsing binaries found by Schiminovich, Lang, and me in the GALEX data; Ned Wright (UCLA) showing an incredibly low-temperature brown dwarf (maybe the lowest ever?) from WISE; Steve Warren (Imperial) showing an embargoed (very) high-redshift quasar from UKIDSS with a beautiful spectrum and praising Daniel Mortlock for his excellent target-selection skills; and a great pair of rapid-fire poster sessions in which each poster at the meeting got one viewgraph and exactly sixty seconds of summary time. Many luminaries are present at the meeting, and many old friends too (and some in both categories simultaneously).

Atlas planning

This past Spring I revived an old project to build some kind of professionally and statistically useful atlas of galaxies from the SDSS data. I spent some time today in the wilderness planning the stages of the project. When I look at the non-triviality of doing a good job on it, I think that even the infrastructure work required to generate the Atlas will produce several publications. First order of business: Get The Tractor running on very (angularly) large galaxies in the SDSS to measure their properties consistently.

eclipse modeling

Schiminovich and I met briefly today with his undergraduate researcher Adam Greenberg (Columbia), who is going to look at doing a better job of fitting the eclipses we have discovered in the GALEX time-domain data. His first order of business is to implement a solid-disk-on-solid-disk eclipse model; then look at better models. We discussed how to parameterize and initialize the model so that simple local optimization should work.

Throughout the day, Phil Marshall, Douglas Applegate (Stanford), and I had a long and detailed discussion about Bayesian evidence (or Bayes factors) as compared to cross-validation in model selection. This is an issue that I have been thinking about a lot and these two helped me sharpen up and modify my thinking substantially. Despite being a Bayesian in practice, I don't believe that most uses of the Bayesian evidence in the literature are correct or justified, mainly because the integral depends so much on aspects of the prior which are (in practice) chosen by the investigator not via an introspective or probabilistic analysis of her or his true prior beliefs but rather more-or-less at random. That is, the prior doesn't really represent prior belief. In this, I am closer to Andrew Gelman (Columbia), who sees the prior as a pragmatic (and testable) regularization of the problem; indeed Gelman and I discussed this point in a separate thread this week.

calibration model

Foreman-Mackey and I spent some time discussing his robust model of photometric calibration for multi-epoch imaging surveys. The basic idea is that every star is drawn from a distribution that is a mixture of variable and non-variable stars, and that every observation of every star is drawn from a distribution that is a mixture of good (inlier) and bad (outlier) measurements. These mixtures permit the model to be most constrained by the most valuable data, but at the expense of many nuisance parameters. It works remarkably well on real data from SDSS Stripe 82, and has great general applicability in the world of time domain astrophysics. Our current goal is to understand how to model variations in calibration parameters in time and angle.

GSOC meet-up, day 3

Today we got OpenID working on Astrometry.net's nascent beta-test web site. This means that users can log in using one of their existing logins from some existing site they trust; we don't have to maintain our own user records or authentication system. I love web 2.0!

For days 4 and 5, team (Chen and Lalimarmo) will be in Princeton, working with Dustin Lang. Good luck team!

GSOC meet-up, day 2

Today was all about database schema for the beta-test (rather than alpha-test) version of Astrometry.net. The idea is to get web-submitted images to insert into a sensible data management system, and then from there to the calibration system and back to the user, with all parts (submissions, images, results, and so on) stored in normal form. By the end of the day, there was a functioning new web system that does a good chunk of what our existing alpha-test site does but with much better underlying technology.

GSOC meet-up, day 1

Astrometry.net's Google Summer of Code interns, Kevin Chen and Carlos Lalimarmo, showed up today to start a one-week code sprint and team meeting, in New York and Princeton. Their first job is to make our web presence look and act a little less 1994, if you know what I'm sayin'. By the end of the day there was a minmal API, but not yet any functional web pages. Because the GSOC is such a competitive program, we have two great coders in Chen and Lalimarmo; their excellence was evident even on day one.

philosophy

On the plane home from La Palma, I read this paper (PDF) by Gelman and Shalizi on the philosophy behind statistics. They, despite being Bayesians, argue that since we don't literally believe as True (with a capital T) any of our models (and I argued the same here), the literal probability interpretation of Bayesian reasoning is flawed. They argue that all the important moments of statistics occur at the points of model checking, model investigation, and the choice of which models to compute in the first place. Gelman and Shalizi's paper was a great post-meeting read; now enough philosophy and back to work!

Astrostatistics and Data Mining, day 5

On the last morning of the summer school, Lupton (Princeton) led (and I assisted in) a lab session on image processing, where the task was to patch missing data in an image using an interpolator based on a Gaussian process. During it, I realized (as Lupton had earlier) that an excellent cosmic-ray detector could be based on modeling an image with a Gaussian process, where the covariance of the process was set locally with the correct covariance expected given the point-spread function (and minimal assumptions about the properties of stars and galaxies in the images). I also realized that Lang and my Tractor project (current version number: vaporware) to model astronomical images could also be used as an interpolator for astronomical images (and a cosmic-ray detector; hell it does everything of course). All that said, interpolation is never necessary; missing data might be ugly, but it is only a problem if you are using algorithms you shouldn't be.

In other parts of the day, Tsalmantza and I worked on finishing our empirical spectral modeling method paper, with a goal of finishing next week.

Astrostatistics and Data Mining, day 4

In the morning, Lupton (Princeton) spoke about imaging data, with a lot of time spent on the (enormous, enormous) value of being band-limited, and how you measure sources with maximum-likelihood techniques. He gave big shout-outs to Lang and my preliminary (and unpublished) results from The Tractor: our project to build a model of all the astronomical imaging in the SDSS, improving on and giving a more transparent and modifiable probabilstic basis to the output of the SDSS software. One of the remarkable aspects of this project, which Lupton noted, is that a very simple model of galaxies and stars does a damned good job of explaining the vast majority of SDSS image pixels, so modeling is not only the best thing you can do, it is also close to saturating the information content in the data.

One thing I realized during his session is that imaging survey data analysis could be much more accurate and precise if we built (hierarchically) priors (maybe even physically motivated ones) on the point-spread function. All the regularizations in use are very heuristic and issues are clearly visible. Another realization I had is that, for faint sources found ab initio in some data, the optimization of the likelihood with respect to position ensures that any unmarginalized flux estimate will be an over-estimate. I have thought about this, years ago, but never delivered. Airplane project? Or maybe I should just offer beer to the first of my loyal readers who demonstrates the magnitude of the effect as a function of signal-to-noise and shows that marginalization corrects it.

In the afternoon I spoke about hierarchical modeling, as demonstrated in our exoplanet distribution modeling projects and our extreme-deconvolution projects. Bailer-Jones (MPIA) objected to my use of the word uninformative to describe certain kinds of priors. I agree; I was using that because it is the jargon of the day. You are always injecting information with your priors; if you can go hierarchical, you inject correct information.

Astrostatistics and Data Mining, day 3

In the morning I wrote a Gaussian processes code to model radial velocity data, just for fun. I am definitely reinventing the wheel, but I am learning a lot. I am using Python classes to cache all the expensive matrix operations; this should make things as fast as they can be without serious engineering.

In the afternoon, Lupton (Princeton) talked about the SDSS and other large surveys. He said that the decisions they made to make the catalog would not all be agreed upon by all users, but they were science-driven, and driven by particular goals. Then, when asked how we could re-make those decisions and re-analyze the data, he essentially said you can't. But he followed that by saying that he wants LSST to be different, with reanalysis possible through smart APIs or equivalent. This meshes nicely with things Anthony Brown said on day 1.

There were a bunch of talks on classifying variables, all using the Random Forest method. I have to learn more about that. A discussion following these talks got a little bit into the issues of generative modeling vs black-box classifying. I far, far prefer the former, of course, because it advances the science (and does a better job, I hope) while performing the classification.

Astrostatistics and Data Mining, day 2

This morning we had the lab session; I had the students write from scratch a cross-validation code for model comparison. The model and optimizer we were using were not so good, so some of the code didn't work very well, but I learned a lot. I hope the students learned something. We tried to figure out if HD104067 has one companion or two. It was a pleasure to have Robert Lupton (Princeton) in the room, as he was tough on my code and also interjected some very useful stuff. Just before the lab session, Aigrain talked about Gaussian processes, showing some absolutely amazing examples from Rasmussen's book.

In the afternoon, there were many excellent talks; too many to mention individually. One highlight was Kitaura (Garching), who is very close to achieving the goal I have of performing properly marginalized measures of cosmological density fields. He might be able to take us away from point estimates of the correlation function! Ascasibar (Madrid) showed a very nice visualization of the positions of galaxies (relative to one another) in spectrum space. He is using some simple tools and finding some nice structure; much of the discussion after his talk was about the possibility of finding sharper tools and maybe even more dramatic results. Matthew Graham (Caltech) gave me a shout-out about not trusting meta-data and only working at the raw pixels (when I can). Christlieb (Heidelberg) showed some astounding spectra from large surveys and discussed how hard it might be to find and understand oddities in the next generation of million-spectra surveys. He included APOGEE in his list of important new surveys, to my pleasure.

There has been a bit of Bayes vs non-Bayes arguing at the meeting, and I have tried not to enter into it. I am a Bayesian, but I am also a pragmatist (and I am also willing to concede that its subjectivity is an issue for many, and not unreasonably). In the long run, we should be optimizing our long-term future discounted free cash flow.

Astrostatistics and Data Mining, day 1

I gave morning lectures at the summer-school part of this meeting on La Palma, on model specification and model choice. I argued that a model is an approximate specification of the probability of the data. I argued for cross-validation, or, when your priors are informative (which, in astronomy, they very rarely are), the Bayesian evidence. Tomorrow we pair-code.

I was followed by Suzanne Aigrain (Oxford), who talked about time-domain methods, especially ones that permit modeling of stochastic processes. She is leading up to Gaussian processes tomorrow, which is highly relevant to my conversations with exoplanet hunters and with Schiminovich about modeling the intensity variations in eclipsing stars.

In the afternoon the workshop part of the meeting started. One highlight for me was that Anthony Brown (Leiden) spent a good part of his talk about Gaia data on Lang and my proposals for a probabilistic catalog. He is committed to having the raw data and the processing pipelines preserved and curated. His message was simple: If you have the best survey and it is going to stay best for many years (decades in the case of Gaia), then re-analysis of your data will be an important capability for the community. Brown was followed by Berry Holl (Lund), who gave an update on his work to make hypothesis testing possible with the full Gaia-catalog covariance matrix, which is far too large to represent even on disk.

Later in the afternoon, the Heidelberg Gaia group showed extremely good results from support vector machines. As my loyal reader knows, I don't like black-box methods like SVM, but they sure do work well.

models and selection

Spent flights to the Canary Islands working on models and model selection for a summer school here, sponsored by the Gaia community. I fit a bunch of models to some radial velocity data supplied by Suzanne Aigrain (Oxford), but then the good models beat the bad models by so many orders of magnitude, model selection is trivial. Debating what examples to use in the coding sessions. For now all I need to do is prepare a few hours of lecture and discussion material.