partial differential equations

I am trying to write a proposal to fund the research I do on machine-learning theory. The proposal is to work on ocean dynamics. It's a great application for the things we have done! But it's hard to write a credible proposal in an area that's new to you. Interdisciplinarity and agility is not rewarded in the funding system at present! At least I am learning a ton as I write this.

Dagstuhl, day 1

Today was day 1 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl. The first day was mainly about applications of machine learning, in Earth science, livestock management, astrophysics (dark matter), cells, and mechanical engineering. I had many thoughts and realizations. Here are a few random ones:

The problems that appear in Earth science, and the data types, are very similar to those that appear in astrophysics! But in Earth science, biology is a big driver of global processes, and there is no good mechanistic model for (say) how plants grow and take up carbon. The world is filled with mobile phones, with good cameras, and the methods we could could be employing to be doing science in a distributed way are way, way under-used. Cells are incredibly complicated. The mechanistic model involves literally thousands of individual processes. Like our model for the cell is as complicated as our model for the entire Earth system (which, by the way, depends on cells!), or even more complicated.

In the areas of the cell and the Earth, a theme was that the investigators want to preserve the causal structure we believe, and just use the machine learning to replace one tiny piece, with a data-driven model. Related: You can think of the machine learning as an effective theory for something (a sub-part of the problem) that doesn't work well from first principles. That's a good idea!

how to simulate a spectrum

I had a great conversation today with Matt Daunt (NYU), building on discussion yesterday with also Megan Bedell (Flatiron), about how to simulate data from an extreme-precision radial-velocity spectrograph. We decided to simulate the star, the atmosphere, and the (gasp!) gas cell all at very high resolution, then combine them physically, then reduce resolution to the spectrograph resolution (which is very high nonetheless) and then sample and noisify the resulting data. The idea is: Make the structure of the code like the structure of our physical beliefs, or causal beliefs. We decided to fork this data simulation into its own project.

astrology: Yes, it's true

Today Paula Seraphim (NYU) and I extended our off-kilter research on the possibility that we live in a simulation to off-kilter research on whether astrology has some basis in empirical fact. It does! There are birth-season correlations with many things. The issue with astrology, oddly, is not the data! It is with the theory that it is all related to planets and constellations. And if you think about the causes of birth-season effects on personality and capability, most of them (but not all of them) would have been much stronger 2000 years ago than they are today!

free energy and life

One amusing conversation today was between Ben Pope (NYU) and myself about whether hot stars are more or less likely to host planets with live. We believe (it's not extremely well established yet) that there are more habitable planets around M-type stars than G-type (and there is probably a relatively smooth function of temperature). So why do we live around a G star? Is it because there is more free energy per photon? I have assumed that this is why. But we realized that we can make this argument quantitative. One question that I have is this: Is this argument anthropic? Or is it just the simple observation that Earth hosts life? I think it is anthropic, because it has something to do with whether our place is special.

#tellurics, day 4

Today was the last day and wrap-up from the Telluric Line Hack Week at Flatiron. What an impressive meeting it was; I learned a huge amount. Here are a few highlights from the wrap-up, but I warn you that these highlights are very subjective and non-representative of the whole meeting! If you want to see more, the wrap-up slides are here.

The most surprising thing to me—though maybe I shouldn't be surprised—was the optimism expressed at the wrap-up. The theoretical modelers of atmospheric absorption were optimistic that data-driven techniques could fill in the issues in their models, and the data-driven modelers were optimistic that the theory is good enough to do most of the heavy lifting. That is, there was nearly a consensus that telluric absorption can be understood to the level necessary to achieve 10-cm/s-level radial-velocity measurements.

Okay maybe just as surprising to me was the demos that various people showed of the Planetary Spectrum Generator that can take your location, a time, and an airmass, and make a physical prediction for the tellurics you will see, even broken down by molecular species. It is outright incredible, and remarkably accurate. It is obvious to me that our data-driven techniques would be much better applied to residuals away from this PSG model. That's an example of the kind of hybrid methods many participants at the meeting were interested in exploring.

One of the main things I learned at the meeting (and I am embarrassed to say this, since in retrospect it is so damned obvious) came from Sharon X Wang (DTM): Even if you have a perfect tellurics model, dividing it out even from your extremely high signal-to-noise spectrum is not exactly correct! The reason is duh: The spectrum is generated by a star times tellurics, convolved with the LSF. That's not the same as the LSF-convolved star times the LSF-convolved tellurics. That is a bit subtle, but seriously, Duh! Foreman-Mackey, Bedell, and I spoke a tiny bit about the point that this subtlety could be incorporated into wobble without too much trouble, and we might need to do that for infrared regions of the spectrum, where the tellurics are very strong. We have gotten away with the wobble approximation because HARPS is high resolution, and in the visible.

And finally (but importantly for me), many of the participants tried out the wobble model or understood it or applied it to their data. We have new users and the good ideas in that code (the very simple, but good, ideas) will propagate into the community. That's very good for us; it justifies our work; and it makes me even more excited to be part of the EPRV community.

#tellurics, day 1

Today was the first day of the Telluric Line Hack Week at Flatiron. We got an amazing crowd to New York, to discuss (and, of course, hack on) some pretty technical matters. But of course this is really about extremely high precision radial-velocity spectroscopy, and this is a community that is detail-oriented, technical, and careful!

The first day was a get-to-know-each-other day, in which we introduced ourselves, and then talked through existing projects, data sets, and instruments. I learned a huge amount today; I'm reeling! Here are a few very subjective highlights:

In the introductions, some common themes appeared. For example, many people using physical models for tellurics want to become more data-driven, and people using data-driven techniques want to be more physics-motivated. So there is a great opportunity this week for hybrid methods, that make use of the physical models, but only use data-driven approaches to model residuals away from the physical models.

Information theory came up more than once; we might do a break-out on this. In particular, we discussed the point (that I love) that what is traditionally done in fitting for RVs is an approximation to the Right Thing To Do (tm), possibly with slightly more robustness. Bedell and I really really ought to write a paper on this! But it is interesting and non-trivial to understand what techniques saturate measurement bounds, and under what assumptions. Unfortunately you can't ask these questions without making very strong assumptions.

In the discussion of hardware details, I was even more motivated than usual to say that we ought to be doing our RV fitting in the two-dimensional spectrograph data (rather than extracting to one dimensional spectra first). I was surprised to learn that many of the hardware people in the room agreed with that! So this seems like a productive direction to start looking.

Sharon Wang (DTM) is doing some interesting work trying to figure out what is really the noise floor from unmodeled telluric features in the atmosphere. That is a great question! She is asking it to bolster or criticize or set the context for going to space. Should we be doing RV in space?

Very excitingly for me, there was lots of enthusiasm in the room for learning about and trying wobble, which is Bedell's method and software for simultaneous data-driven fitting of tellurics and star. The discussion at the end of the day was all about this method, and the questions in the room were excellent, awesome, and frightening. But if all goes well we will launch quite a few projects this week.

Math+X Houston, day 1

Today was the first day of the 2019 Math+X Symposium on Inverse Problems and Deep Learning in Space Exploration, which is a meeting to bring together mathematicians and domain scientists to discuss problems of mutual interest. I learned a huge amount today! I can't summarize the whole day, so here are just a few things ringing in my brain afterwards:

Sara Seager (MIT) and I both talked about how machine learning helps us in astrophysics. She focused more on using machine learning to speed computation or interpolate or emulate expensive atmospheric retrieval models for exoplanet atmospheres. I focused more on the use of machine learning to model nuisances or structured noise or foregrounds or backgrounds in complex data (focusing on stars).

Taco Cohen (Amsterdam) showed a theory of how to make fully, locally gauge-invariant (what I would call “coordinate free”) deep-learning models. And he gave some examples. Although he implied that the continuous versions of these models are very expensive and impractical, the discretized versions might have great applications in the physical sciences, which we believe truly are gauge-invariant! In some sense he has built a superset of all physical laws. I'd be interested in applying these to things like CMB and 21-cm foregrounds.

Jitendra Malik (Berkeley) gave a nice talk about generative models moving beyond GANs, where he is concerned (like me) with what's called “mode collapse” or the problem that the generator can beat the discriminator without making data that are fully representative of all kinds of real data. He even name-checked the birthday paradox (my favorite of the statistical paradoxes!) as a method for identifying mode collapse. Afterwards Kyle Cranmer (NYU) and I discussed with Malik and various others the possibility that deep generative models could possibly play a role in implicit or likelihood-free inference.

There were many other amazing results, including finding seismic pre-cursors to landslides (Seydoux) and using deep models to control adaptive optics (Nousianinen) and analyses of why deep learning models (which have unimaginable capacity) aren't insanely over-fitting (Zdeborová). On that last point the short answer is: No-one knows! But it is really solidly true. My intuition is that it has something to do with the differences between being convex in the parameter space and being convex in the data space. Not that I'm saying anything is either of those!

6-volume, myspace, rules, tellurics

Too many things today for one blog post! So just a rapid-fire list. Matt Buckley (Rutgers) and Adrian Price-Whelan (Princeton) and I discussed whether we could, in practice, measure phase-space six-volumes given a point-set in Gaia or a future data set. It isn't clear, so we started by designing some extremely simple simulations to test.

Price-Whelan and I discussed our myspace project to find the nonlinear transformation of the phase-space data near the Sun to make the phase-space structure as compact or informative as possible. We have a plan for implementation of the data-science side of the project, but we have no idea whether anything we find will be interpretable!

We had our first Stars Meeting under the new rules that we established last week. The objectives are, more-or-less: We want the presenters to be less prepared and we want the audience to be more engaged. We created some rules or guidelines to help achieve these objectives. And the meeting went well! Among other things that happened in this meeting, Price-Whelan showed a forming star cluster he found in the Milky Way halo, possibly connected to the Magellanic gas stream, and John Brewer (Yale) showed micro-tellurics (tiny atmospheric absorption lines) found in some of the very first R=150,000 EXPRES spectra.

On that last point: Brewer found these tellurics by observing a B star, which has no narrow lines (and almost no lines at all), so the narrow absorption lines must be intervening. Megan Bedell (Flatiron) has a data-driven method for finding tellurics even in very featured, narrow-lined spectra, by exploiting the causal structure: Star lines move with the star, atmosphere lines move with the atmosphere! She confirms at least qualitatively, at least some of Brewer's lines. I expect that we have some nice points to make in the comparison.

Oh, and: Unmodeled telluric absorption might be the limiting systematic in exoplanet RV surveys, right now or in the near future.

modeling the heck out of the atmosphere

The day started with planning between Bedell (Flatiron), Foreman-Mackey (Flatiron), and I about a possible tri-linear model for stellar spectra. The model is that the star has a spectrum, which is drawn from a subspace in spectral space, and doppler shifted, and the star is subject to telluric absorption, which is drawn from a subspace in spectral space, and doppler shifted. The idea is to learn the telluric subspace using all the data ever taken from a spectrograph (HARPS, in this case). But of course the idea behind that is to account for the tellurics by simultaneously fitting them and thereby getting better radial velocities. This was all planning for the arrival of Ben Montet (Chicago), who arrived later in the day for a two-week visit.

At lunch time, Mike Blanton (NYU) gave the CCPP brown-bag talk about SDSS-V. He did a nice job of explaining how you measure the composition of ionized gas by looking at thermal state. And etc!

micro-tellurics

Today, in an attempt to make our simulated extreme-precision radial-velocity fake data as conservative as possible, Megan Bedell (Chicago) and I built a ridiculously pessimistic model for un-modeled (and unknown) telluric lines that could be hiding in the spectra, at amplitudes too low to be clearly seen in any individual spectrum, but with the full wavelength range bristling with lines. Sure enough, these “micro-tellurics” (as you might call them) do indeed mess up radial-velocity measurements. The nice thing (from our perspective) is that they mess up the measurements in a way that is co-variant with barycentric velocity, and they mess up synthetic-spectrum-based RV measurements less than binary-mask-based RV measurements.

At MPIA Galaxy Coffee, Irina Smirnova-Pinchukova (MPIA) gave a great talk about her trip on a SOFIA flight.

destroyer of worlds

One of my main research accomplishments today was to work up a project proposal for Yuan-Sen Ting (ANU) and others about finding stars whose spectra suggest that they have (recently) swallowed a lot of rocky material. This was inspired by a few things: The first is that Andy Casey (Monash) can find Li-rich stars in LAMOST just by looking at the residuals away from a fit by The Cannon at the location of Li lines. The second is that Semyeong Oh (Princeton) and various collaborators have found Sun-like stars that look like they have swallowed many Earth masses of rock in their recent pasts, by doing (or having John Brewer of Yale do) detailed chemical abundance work on the spectra. The third is that Yuan-Sen Ting has derivatives of spectral expectations with respect to all elements for LAMOST-like spectra.

At the end of the day, Hans-Walter Rix (MPIA) gave a colloquium on the After-Sloan-IV project, which my loyal reader knows a lot about. I learned things in his talk, however: One is that SDSS-III BOSS has found several broad (ish) lined quasars that shut off between SDSS-I and SDSS-III. One relevant paper is here. Another is that he (with Jonathan Bird of Vandy) has made some beautiful visualizations of the point of doing dense sampling of the giant stars in the Milky Way disk.

global ozone

In the morning, researchers from across the Flatiron Institute gathered for a discussion of statistical inference, which is a theme that cuts across the different departments. Justin Alsing (Flatiron) led the discussion, asking for advice on his project to model global ozone over the last few decades. He has data that spans latitude, altitude, and time, and the ozone levels can be affected by many things other than long-term degradation by pollutants. So he wants to build a non-linear, data-driven model of confounders but still come to strong conclusions about the long-term trends. There was discussion of many relevant methods, including large linear models (regularized strongly), independent components analysis, latent variable models, neural networks, and so on. It was a wide-ranging and valuable discussion. The CCB at Flatiron has some valuable mathematics expertise, which could be important to all the Flatiron departments.

#DSIworkshop

I spent most of the day at Columbia's Data Science Institute, participating in a workshop on data science in the natural sciences. I learned a huge amount! There were way too many cool things to mention them all here, but here are some personal highlights:

Andrew Gelman (Columbia) talked about the trade-off between spatial resolution and what he called “statistical resolution”; he compared this trade-off to that in political science between conceptual resolution (the number of questions we are trying to ask) and statistical resolution (the confidence with which you can answer those questions). He also talked about distribution (or expectation) propagation algorithms that permit you to analyze your data in parts and combine the inferences, without too much over-working.

Joaquim Goes (Columbia) talked about ocean observing. He pointed out that although the total biomass in the oceans is far smaller than that on land, it cycles faster, so it is extremely important to the total carbon budget (and the natural sequestration of anthropogenic carbon). He talked about the Argo network of ocean float data (I think it is all public!) and using it to model the ocean.

John Wright (Columbia) pointed out that bilinear problems (like those that come up in blind deconvolution and matrix factorization and dictionary methods) are non-convex in principle, but we usually find good solutions in practice. What gives? He has results that in the limit of large data sets, all solutions become transformations of one another; that is, all solutions are good solutions. I am not sure what the conditions are etc., but it is clearly very relevant theory for some of our projects.

There was a student panel moderated by Carly Strasser (Moore Foundation). The students brought up many important issues in data science, one of which is that there are translation issues when you work across disciplinary boundaries. That's something we have been discussing at NYU recently.

JPL, day 2

My second day at JPL included conversations with Gautam Vasisht (JPL) about adaptive optics systems, Geoff Bryden (JPL) about the abundance of Earth analogs, and Alina Kiessling (JPL) about intrinsic alignments and weak lensing. On the latter we discussed the problem that most of the theory is based on dark-matter-only simulations, but this is precisely a problem where the baryons matter a lot! With Vasisht I learned that JPL has a "clock" up in one of the buildings that shows you the current counts of exoplanet candidates, confirmed exoplanets, and planets in the habitable zone! Plus a huge model of the Mars Lander. Awesome!

At the end of the day, Leonidas Moustakas (JPL), Curt Cutler (JPL), and I argued about the flow from experimental design (think: satellite astrophysics project) to quantitative results on the parameters or scientific questions of greatest interest. They are thinking about making standards and principles for doing this flow, thereby strengthening the quantitative arguments in their proposals for new (and complex) projects. We discussed the challenges of doing something of general value, but they have decided (very sensibly) to start with a few very specific projects to use as "poster children" for the idea. One challenge is doing this with the right "language" such that people from different scientific backgrounds can agree on what's being said at each stage (think of words like "bias" and "noise" and "model" and "systematic" and so on).

dotastronomy, day 2

Today was the Hack Day at dotastronomy. An incredible number of pitches started the day. I pitched using webcam images (behind a fisheye lens) from the Liverpool telescope on the Canary Islands to measure the sidereal day, the aberration of starlight, and maybe even things like precession and nutation of the equinoxes.

I spent much of the day discussing and commenting on other hacks: I helped a tiny bit with Angus and Foreman-Mackey's hack to sonify Kepler data, I listened to Jonathan Fay (Microsoft) as he complained about the (undocumented, confusing) Astrometry.net API, and I discussed testing environments for science with Arfon Smith (github) and Foreman-Mackey and others.

Very late in the evening, I decided to get serious on the webcam stuff. There is an image every minute from the camera and yet I found that I was able to measure sidereal time differences to better than a second, in any pair of images. Therefore, I think I have abundant precision and signal-to-noise to make this hack work. I went to bed having satisfied myself that I can determine the sidereal period, which is equivalent to figuring out from one day's rotation how many days there are in the year. Although I measured the sidereal day to nearly one part in 100,000, my result is equivalent to a within-a-single-day estimate for the length of the year of 366.6 days. If I use more than one image pair, or span more than one day in time, I will do far, far better on this!

black holes and weird pixel effects

In group meeting, Huppenkothen argued out the projects we discussed on Monday related to machine classification of black-hole accretion states of GRS 1915. We talked about all three levels of project: Using supervised methods to transfer classifications for a couple of years of data onto all the other years of data, using unsupervised methods to find out how many classes there plausibly are for the state, and building some kind of generative model either for state transitions or for literally the time-domain photon data. We discussed feature selection for the first and second projects.

Also at group meeting, Foreman-Mackey showed a new Earth-like exoplanet he has discovered in the Kepler data! Time to open our new Twitter (tm) account. He also showed that a lot of his false positives relate to un-discovered discontinuities in the Kepler photometry of stars. After lunch, we spent time investigating these and building (hacky, heuristic) code to find them.

Here are the symptoms of these events (which are sometimes called "sudden pixel sensitivity drops"): They are very fast (within one half-hour data point) changes to the brightness of the star. Although the star brightness drops, in detail if you look at the pixel level, some pixels brighten and some get fainter at the same time. These events appear to have signs and amplitudes that are consistent with a sudden change in telescope pointing. However, they are not shared by all stars on the focal plane, or even on the CCD. Insane! It is like just a few stars jump all at once, and nothing else does. I am confused.

Anyway, we now have code to find these and (in our usual style) split the data at their locations.

Kepler occurrence rate review, day 1

Today I got up at dawn's crack and drove to Mountain View for a review of the NASA Kepler team's planet occurrence rate inferences. It was an incredible day of talks and conversations about the data products and experiments needed to turn Kepler's planet (or object-of-interest) catalog into a rate density for exoplanets, and especially the probabilities that stars host Earth-like planets. We spent time talking about high-level priorities, but also low-level methodologies, including MCMC for uncertainty propagation, adaptive experimental design for completeness (efficiency) estimation, and the relative merits of forward modeling and counting planets in bins. On the latter, the Kepler team is creating (and will release publicly) everything needed for either approach.

One thing that pleased me immensely is that Foreman-Mackey's paper on the abundance of Earth analogs got a lot of play in the meeting as an exemplar of good methodology, and also an exemplar of how uncertain we are about the planet occurrence rate! The Kepler team—and increasingly the whole astronomical community—is coming around to the view that forward modeling methods (as in hierarchical probabilistic modeling or approximate bayesian computation) are preferable to counting dots in bins.

exoplanet photometry, crossing the streams

Nick Cowan (Northwestern, Amherst) was in town today to give a seminar about exoplanet thermodynamics and climate. He showed nice results inferring the temperature distribution on the surfaces of hot jupiters and the same for degraded data on the Earth. He spent a lot of time talking about thermostats and carbon and water cycles on the Earth and Earth-like planets. In the morning I discussed my OWL photometry with him, and we proposed to try it on his massive amounts of Spitzer IRAC data on transiting exoplanets.

After lunch, a group of the willing (Rix, Bovy, Sesar, Price-Whelan, myself) discussed stream fitting in the Milky Way. We decided to fit multiple streams simultaneously, starting with Orphan, GD-1, and Palomar 5. The first step is to gather the data. Price-Whelan and I also have to modify our method so that it can take the heterogeneous kinds of data in the GD-1 data set.

how many Earth analogs are there in the Kepler field?

Foreman-Mackey may have actually finished his paper on exoplanet abundances today! I hope this is true and we submit tomorrow. I did work on the text for him, but only in the form of giving final comments. One of our main points is that the "rate" or "frequency" or "abundance" of Earth analogs should be expressed as an expected number per star per natural logarithm of period, per natural logarithm of radius. However, in the end, he also computed the number of planets that we expect to have in the Kepler field, with period between 200 and 400 days (Petigura's definition) and radius between 1 and 2 Earth radii (Petigura again), orbiting one of the 42,000 Sun-like stars, in such a way (inclination) that it would transit (conceivably) observably. The answer is nine. With large uncertainty. That is, we should be looking very hard for these Earth analogs, because there ought to be a few of them!