cross-validation

I gave my MPIA Hauskolloquium on model selection today. I strongly advocated leave-one-out cross-validation. It is very easy, conceptually simple, robust to many kinds of mistakes (about model space and noise amplitude), and employs a "prediction" utility that matches the goals of most scientists. Despite the frequentism of all this, I am still thought-of as a Bayesian around here. I am only a Bayesian when I have to be! That turns out to be frequently.

The marginalized likelihood (marginalized over orbital phase) mentioned yesterday did work well, so Jagannath and I are ready to write our paper. We discussed the introduction for the paper and the speed of the code.

marginalization over orbital phase

Jagannath and I changed our stream-fitting code to marginalize over orbital phase. This is much better (in principle) than optimizing for it at each star. Will it perform well in practice? We wait to see. On a related note, Tsalmantza and I finished the marginalization of our quasar-redshift likelihoods over spectral parameters. Here too it remains to be seen if this improves our redshift predictions. Both of these projects are, in some sense, testing Bayesianism, since frequentists can't marginalize their likelihoods.

At lunch, Christian Schwab, Andreas Quillenbach (LSW), and I discussed the crazy stellar pulsations that can look very much like exoplanets. Our project is to model them and do hypothesis testing.

cross-validation demos

I am giving a talk on model selection on Friday. Today I made a demo that compares AIC, BIC, and leave-one-out cross-validation. All these are frequentist model selection criteria (despite the B in BIC). Of them, cross-validation is by far the best for many reasons: It is based on data prediction, not theory; it is (relatively) insensitive to bad uncertainty estimates; it has an interpretation in terms of a sensible utility. One of the oddities of model selection, not often appreciated, is that no principled frequentist or Bayesian result tells you which model to choose; data analysis just puts probabilities on models. If you want to remain principled you either never choose and just propagate all models (best practice for Bayesians), or else explicitly state your utility function and make the decision that maximizes your expected utility. Any time you choose a model without specifying or calculating utility, you have made a mistake, I think. Perhaps not a serious mistake, but a mistake nonetheless.

anisotropic noise tensors

For both quasar structure functions and Jagannath's project on tidal streams, I am playing around a lot with non-trivial variance tensors to insert into Gaussian probability distribution functions. On the quasars, I am trying to make a very pedagogical introduction to the idea that the structure function—a model of quasar variance—should be cast in terms of Gaussian processes—a set of general models for variance. This was Bovy's good idea; it makes all sorts of new kinds of data analyses possible and all sorts of existing observations more useful. On the streams, we are modeling observational uncertainty (which is simple in distance, transverse angles, radial velocity, and transverse angular velocity but complex in position–velocity space) in six-dimensional phase space, so that our cold-stream data analysis has a realistic description. The short-term goal is to establish a method for fitting streams that is justified in terms of probabilistic inference.

After lunch, Rory Holmes (MPIA) led a discussion of Euclid calibration and observing strategy, because the MPIA is part of a team proposing to operate part of that mission.

fitting a curve to stars

Tsalmantza and I continued with binary quasar inspection. Myers, Hennawi and I discussed Pan-STARRS calibration with Eric Morganson (MPIA). On a related note, at the weekly MPIA Pan-STARRS meeting, Cristina Afonso (MPIA) showed that Pan-STARRS data are very stable and can be calibrated to sub-percent precision when there are many overlapping epochs. Kasper Schmidt (MPIA), Rix, and I discussed quasar variability, and I worked out a few things we can do to model it more flexibly.

Most importantly, however, Jagannath showed up in Heidelberg for a week, and we discussed our project of fitting orbits to streams of stars at lunch with Rix. He made some suggestions—in particular to consider the effect of systematic distance errors—which slightly adjusted our to-do list; we are on track to finish a zeroth order document by the end of the week.

two new binary quasars!

Tsalmantza, Roberto Decarli (MPIA), and I discovered two new binary quasars today. One we found by our fitting, and one by looking for quasars judged to be galaxies by the spectroscopic pipeline (the reason these are sometimes binaries is because the broad lines can be shifted very far from the narrow lines, tricking the pipeline). Given that there are only four binary quasars known previously, this was a pretty good day's work.

priors on spectra

Hennawi nearly convinced me that Tsalmantza and I should be putting priors on our coefficients in spectral space and switch from producing likelihoods to producing marginalized likelihoods or posterior probabilities. I spent the evening writing up the project we are doing together and writing down how we could construct and use priors.

cross-validation

I built a substantial leave-one-out cross-validation demo for my talk on model complexity. The demo is very cool and will be useful when I write up a document about model selection. The talk is in one week.

redshifts and multiple redshifts

Today Tsalmantza and I got results on two projects: We found quasars with potential double-redshifts (that is, double quasars) with our data-driven quasar model, and we determined low-redshift quasar redshifts with great precision, with this same model. In the former project, the idea is to do a two-quasar vs one-quasar hypothesis test for every quasar. In the latter project, the question is whether we can make narrow-line-quality redshift determinations of quasars for which the narrow lines are not visible. Both projects look promising; we ended the day pretty optmistic.

non-parametric fitting

I got working a simple (though not fast) system to fit a highly parameterized (what is called, for some reason, non-parametric) curve to a set of data today, in preparation for a future talk about model complexity. My model has more parameters than data, but it optimizes, and it has a complexity that is continuously variable; the complexity is not the number of parameters.

PHAT

Rix, Julianne Dalcanton (UW), and I spent a morning in the garden discussing the high-level data-analysis plan (or maybe inference plan) for Dalcanton's PHAT project. This project is imaging a large part of Andromeda and the inference is about the ages, masses, and initial mass functions of young stellar clusters. Among other things we discussed the difference between a model that assumes a flat age distribution, and a model that insists on a flat age distribution, and the difficulty of making the latter within Bayesian inference. Indeed, this was what was hard initially about Bovy and my Solar System (or orbital roulette) project.

Dalcanton

I can't say I got much done today, except consult with Lang on his astrometric calibration work on Dalcanton's PHAT project to image M31. At Galaxy Coffee, Dalcanton quickly showed the first data from PHAT, which is absolutely beautiful, and so much better than anything that has come before.

cusp hypothesis testing

I finally wrote down in detail how we can do a hypothesis test for an ultra-faint galaxy between the cusp-in-projected-phase-space and self-gravitating-blob models. Zolotov is working on the execution.

data-analysis consulting

I gave (solicited) data-analysis advice to Rainer Klement (MPIA), Yujin Yang (MPIA), and Christy Tremonti (Wisconsin). I like my role as consultant. Hennawi, Myers, and I spent a long lunch reviewing our long to-do list for this summer. If we even get one fifth done I will be happy.

Groningen

I gave a seminar at Groningen today, where I was hosted by Scott Trager. I gave my Gaia talk, which I fear is way too pessimistic; I need to accentuate the positive.

the Oort problem

After talking to Bovy yesterday about the Oort problem (and extensions, to measure the local gravitational potential gradients in the Galaxy), Rix and I asked Volker Springel (Heidelberg) about the possibility of making mock catalogs and performing method tests in realistic simulations. He was positive, but we have no plans. What I would really like to do is to set up a blind test with a cash prize! But we don't really know how wrong our answers become as our assumptions are violated, and many of the Oort assumptions are wrong in detail. After this conversation, we learned about the Meerkat project, which is an SKA precursor but very ambitious nonetheless.

still fitting a line

I finished the third (I think) draft of my old line-fitting document, taking into account all the great comments I received from blog-readers and other experts. It is in Lang's hands, then it will go to Bovy, and then to the arXiv. I can't wait to be done; it is one of the milestones for the summer.

angle-mixed methods

In the morning with Rix and later in the day with Michael Perryman (ESA, Leiden) I discussed the likely failure of angle-mixed approaches to inferring the Milky Way dynamics from a kinematic snapshot. Rix still believes we should do the Oort problem (measuring the disk potential locally) because that is likely to be close to angle-mixed, at least for some populations. I spent the other parts of the day writing, with the exception of a break in which Roberto Decarli (MPIA) explained to Tsalmantza and me how our spectral modeling could revolutionize the study of black-hole binaries. We are going to pursue that next week.

ULIRG clustering, spectral modeling

After a few days of vacation I arrived at the MPIA for my usual summer stay. Tsalmantza and I specified the scope and content of the paper we intend to complete this summer, and Ben Weiner (Arizona) spoke about the clustering and star formation of redshift-unity galaxies. He showed that neither quasars nor luminous star-forming galaxies are clustered like the massive, red galaxies of which they are supposed to be the progenitors.