wrote like the wind; frequentist vs Bayes on sparsity

My goal this year in Heidelberg is to move forward all writing projects. I didn't really want to start new projects, but of course I can't help myself, hence the previous post. But today I crushed the writing: I wrote four pages in the book that Rix (MPIA) wants me to write, and I got more than halfway done with a Templeton Foundation pre-proposal that I'm thinking about, and I partially wrote up the method of the robust dimensionality reduction that I was working on over the weekend. So it was a good day.

That said, I don't think that the iteratively reweighted least squares implementation that I am using in my dimensionality reduction has a good probabilistic interpretation. That is, it can't be described in terms of a likelihood function. This is related to the fact that frequentist methods that enforce sparsity (like L1 regularization) don't look anything like Bayesian methods that encourage sparsity (like massed priors). I don't know how to present these issues in any paper I try to write.

image denoising; the space of natural images

I got in a bit of research in a mostly-teaching day. I saw the CDS Math-and-Data seminar, which was by Peyman Milanfar (Google) about de-noising models. In particular, he was talking about some of the theory and ideas behind the de-noising that Google uses in its Pixel cameras and related technology. They use methods that are adaptive to the image itself but which don't explicitly learn a library of image priors or patch priors or anything like that from data. (But they do train the models on human reactions to the denoising.)

Milanfar's theoretical results were nice. For example: De-noising is like a gradient step in response to a loss function! That's either obvious or deep. I'll go with deep. And good images (non-noisy natural images) should be fixed points of the de-noising projection (which is in general non-linear). Their methods identify similar parts of the images and use commonality of those parts to inform the nonlinear projections. But he explained all this with very simple notation, which was nice.

After the talk I had a quick conversation with Jonathan Niles-Weed (NYU) about the geometry of the space of natural images. Here's a great argument he gave: Imagine you have two arbitrarily different images, like one of the Death Star (tm) and one of the inside of the seminar room. Are these images connected to one another in the natural-image subspace of image space? That is, is there a continuous transformation from one to the other, every point along which is itself a good natural image?

Well, if I can imagine a continuous tracking shot (movie) of me walking out of the seminar room and into a spaceship and then out of the airlock on a space walk to repair the Death Star (tm), and if every frame in that movie is a good natural image, and everything is continuous, then yes! What a crazy argument. The space of all natural images might be one continuously connected blob. Crazy! I love the way mathematicians think.

Daubechies

Today I had the great honor of meeting Ingrid Daubechies (Duke), who is a pioneering and accomplished mathematician, known for some of the fundamental work on wavelets and representations that have been incredibly important in data. For example, the JPEG standard is based on her wavelets! She gave a talk at the end of the day on teeth. Yes teeth. It turns out that the shapes of tooth surfaces tell you simultaneously about evolution and diet. And she has worked out beautiful ways to first get distances between surfaces. Like metric distances in surface space. And then join those distances up into local manifolds. It could have relevance to things we have been thinking about for a non-parametric version of The Cannon. It was a beautiful talk, with the theme or message that you do better in your science if you use mathematical tools that are matched well to the structure of your problem. That message is either obvious or deep. Or both! What a privilege to be there.

inferring maps; detecting waves

I had a great conversation this morning with Yashar Hezaveh (Flatiron) about problems in gravitational lensing. The lensing map is in principle a linear thing (once you set the nonlinear lensing parameters) which means that it is possible to marginalize out the source plane analytically, in principle, or to apply interesting sparseness or compactness priors. We discussed computational data-analysis ideas.

Prior to that, I had an interesting conversation with Rodrigo Luger (Flatiron) and Dan Foreman-Mackey (Flatiron) about the priors we use when we do stellar-surface or exoplanet-surface modeling (map-making). Most priors that are easy to use enforce smoothness, but our maps might have sharp features (coastlines!). But we more-or-less agreed that more interesting priors are also less computationally tractable. Duh!

At mid-day, Chiara Mingarelli (Flatiron) argued in a great seminar that pulsar timing will make a detection in the next years. Her argument is conservative, so I am very optimistic about this.

stellar mergers and oscillations; cosmological dictionaries

Today was group-meeting day. In stars group meeting, Matteo Cantiello (CCA) discussed recent results on star–star interactions, including a star–star merger that may have been caught by the OGLE experiment. He gave us some order-of-magnitude thinking about the common-envelope phase and how we might use these events to understand stars. He was pessimistic about being able to do full simulations of the events; there are too many things happening at too many scales. He also showed us another tight binary system which shows period changes that suggest a merger in 2022.

Dan Foreman-Mackey (UW) spoke about linear algebra and asteroseismology. With Eric Agol (UW) he has developed linear algebra techniques such that he can solve matrix equations in linear time (and also take the determinant, which is super-important), provided that the matrix is a kernel matrix of a certain (very flexible) form. This form is capable of modeling a star's light curve as a mixture of stochastically driven oscillators. This raises the hope of automatically getting asteroseismic parameters for all TESS stars! In the discussion, we arrived at the idea of using Kepler to measure the three-point function for stellar variability. David Spergel (Flatiron) predicted that it would lead to constraints on mode coupling and other aspects of stellar physics.

Cosmology group meeting was crashed by Daniel Mortlock (Imperial) and Hiranya Peiris (UCL). Mortlock told us that there are still very high-redshift quasars being discovered, but that he still has the redshift record, and that, given Eddington time-scales, his is still the most extreme high-redshift quasar. This was followed by a wide-ranging discussion (led by Elijah Visbal, Flatiron) of the possibility that we could be using generative models or better estimators than two-point functions in 21-cm surveys designed to discover the physics of reionization of the Universe. Peiris brought up dictionary methods and we spent time discussing these, and the possibility that we could learn sparse dictionaries on simulations and use them on data. It was very vague, but gives me ideas about where we at CCA need to learn more about methodologies.

compressed sensing

It was a health-issue-shortened day today. Andy Casey worked on code, and I worked on words, on our approach to a compressed-sensing version of The Cannon.

Gaia, validation, and K2

First thing in the morning, Andy Casey and I discussed some ideas for immediately following the first Gaia data release, which may be expanding in scope, apparently. We had the idea of gathering multi-band point-source photometry for as much of the sky as possible, and then using the data release to learn the relationship between the photometry and absolute magnitude. Then we can deliver all distances for every single point source we can! With error analysis. There would be a lot of users for such a thing.

By the end of the day, Casey had the validation working to set the regularization hyperparameter at each wavelength of the compressed-sensing version of The Cannon. Everything is looking as we predicted, so we will try to reproduce all of the results from the original Cannon paper, but now with a model with more sensible model freedom. I am extremely optimistic about where this project is going.

Mid-day I had a phone call to talk about photometry in K2 Campaign 9, which is in the bulge (and hence crowded). We discussed forward modeling and more data-driven detrending methods. But most importantly, we decided (more or less; final decision tomorrow) to do a tiny, half-pixel offset (dither) at the mid-campaign break. That might not sound like much, but I think it will significantly improve the calibration we can do and substantially increase the number of things we can learn.

hierarchical photometric redshifts, combining unreliable methods

I am still not well, but well enough for the first time in ages to do my group meeting. It was great! Boris Leistedt (NYU) talked to us about his project to do template-based photometric redshifts but where he learns the templates too. I love this project; it is the only method I like for getting the redshift distribution in LSST past the (effective) spectroscopic magnitude limit. He gave us the philosophy and showed us some preliminary successes on simple fake data.

Andy Casey (Cambridge) talked to us about the different stellar parameter and abundance codes working in the Gaia-ESO spectroscopic survey collaboration. It is complicated! Getting reliable and valuable stellar quantities out of mutually inconsistent codes is not an easy problem. His methodologies are filled with good ideas. Brian McFee (NYU) suggested that he or we look at the crowd-sourcing literature for ideas here, and that turns out to be a very good idea. I proposed to Casey that we do some reading this coming week.

Right after group meeting, Casey diagnosed our bugs from yesterday and got the L1-regularized fitting working! We celebrated with ramen.

extending The Cannon

Foreman-Mackey called with ideas for The Cannon: He thinks we might be able to do some simple things to propagate errors or uncertainties in our labels in the training set into uncertainties in our internal parameters, and from there to uncertainties in the labels we derive for new stars. His idea is based on the “uncertainties in both directions” model in my old fitting-a-line document. He also wants to play around with a fully Bayesian Cannon. We think this will be hard, but conceivably possible. He is thinking forward to some kind of sophisticated TESS input catalog in these projects.

As for Casey and my work on the compressed-sensing version of The Cannon: We turned on optimization with our L1-regularized model and nothing seems to be working right. Diagnosis progresses.

modeling stars and markets

Andy Casey (Cambridge) arrived in NYC for two weeks to work on a compressed-sensing version of The Cannon. We talked about the design matrix, which is built from functions of the stellar labels. We also talked about how to do the cross-validation to both set the hyper-parameters (which are regularization strengths) and also check the prediction accuracy, without double-using the data. We are adopting ideas from Foreman-Mackey, who also encouraged us to think about propagating label uncertainties. We keep deciding this is computationally impossible, but Foreman-Mackey had some cheap-and-dirty suggestions (which come at the cost of severe approximation).

In the afternoon, Tim Geithner (formerly US Secretary of the Treasury) came by the Simons Foundation to talk about his career and involvement in the 2007–2009 financial crisis. This isn't really bloggable (according to The Rules at right), except that he actually used the word “Bayesian” to describe the attitude that he and his colleagues at the Federal Reserve (where he was in 2007 and 2008) took towards global financial markets, where there is no firmly established causal model. They had to act (or not act) in the face of great uncertainty. I am not sure I agree with what they did, but he quasi-espoused a quasi-principle that I hadn't heard before: Instead of optimizing expected utility, they tried to optimize for ability to correct their mistakes after the fact. It is like expected repair-ability! Not sure if that is really a useful guiding principle, but it occurred to me that it might have implications for policies around global climate change. And TESS and LSST?

The Cannon and detailed abundances

[I am on vacation this week; that didn't stop me from doing a tiny bit of research.]

I did a bit of writing for the project of taking The Cannon into compressed-sensing territory, while Andy Casey (Cambridge) structures the code so we are ready to work on the problem when he is here in NYC in a couple of weeks. I tried to work out the most conservative possible train–validate–test framework for training and validation, consistent with some ideas from Foreman-Mackey. I also tried to understand what figures we will make to demonstrate that we are getting better or more informative abundances than other approaches.

Hans-Walter called to discuss the behavior of The Cannon when we try to do large numbers of chemical abundance labels. The code finds that its best model for one element will make use of lines from other elements. Why? He pointed out (correctly) that The Cannon does it's best to predict abundances. It is not strictly just measuring the abundances. It is doing it's best to predict, and the best prediction will both measure the element directly, and also include useful indirect information. So we have to decide what our goals are, and whether to restrict the model.

The Cannon is an interpolator

In a day wrecked by health issues—I was supposed to be spending the day with visiting researchers from Northrop Grumman–I did manage to write a few equations and words into the method section of the document describing our plans for a compressed-sensing upgrade of The Cannon. It is so simple! And I realize we have to correct our argument (in the first paper) that we need exponentially large training sets to go to large numbers of labels. That argument was just plain wrong; it was based on an assumption that The Cannon is effectively a density estimator. It is not; it is essentially an interpolator.

writing and talking

In another day with limited motility, one small victory was drafting an abstract for upcoming work on The Cannon with Andy Casey (Cambridge). I like to draft an abstract, introduction, and method section before I start a project, to check the scope and (effectively) set the milestones. We plan to obtain benefit from both great model freedom and parsimony by using methods from compressed sensing.

I also had a few conversations; I spoke with Dun Wang and Schiminovich about Wang's work on inferring the GALEX flat-field. We made a plan for next steps, which include inferring the stellar flat and the sky flat separately (which is unusual for a spacecraft calibration). I spoke with Magland about colors, layout, and layering in his human interface to neuroscience data. This interface has some sophisticated inference under the hood, but needs also to have excellent look and feel, because he wants customers.