Terra Hunting Fall Science Meeting, day 1

Today was the first day of the Terra Hunting annual science meeting. One highlight of the day was a presentation by Yan Liang (Princeton), who is modeling stellar spectral variability (the tiny variability) that affects extremely precise radial-velocity measurements. Her method involves a neural network, which is trained to distinguish RV variations and spectral shape variations through a self-supervised approach (with a data augmentation). Then it separates true stellar RV variations from spectral-variability-induced wrong RV variations by requiring (essentially) that the RV variations be uncorrelated with the (latent) description of the stellar spectral shape. This connects to various themes I am interested in, including wobble by Bedell, a spectral variability project by Zhao, and causal structure in machine learning.

conjectures about pre-training

On Monday of this week, Shirley Ho (Flatiron) gave a talk at NYU in which she mentioned the unreasonable effectiveness of pre-training a neural network: If, before you train your network on your real (expensive, small) training data, you train it on a lot of (cheap, approximate) pre-training data, you get better overall performance. Why? Ho discussed this in the context of PDE emulation: She pre-trains with cheap PDEs and then trains on expensive PDEs and she gets way better performance than she does if she just trains on the expsensive stuff.

Why does this work? One interesting observation is that even pre-training on cat videos helps with the final training! Ho's belief is that the pre-training gets the network understanding time continuity and other smoothness kinds of things. My conjecture is that the pre-training teaches the network about (approximate) diffeomorphism invariance (coordinate freedom). The cool thing is that these conjectures could be tested with interventions!

radical papers I want to write (or will never write)

I have to finish my NSF proposal with Mike Blanton (NYU), so naturally I am in procrastination mode. Here are three papers I wish I would write. Maybe I should post them on my ideas blog:

Occam's Razor is wrong: This paper, co-authored with Jennifer Hill (NYU), would be about the fact that, in the real, observed world, the simplest explanation is always wrong or at least incomplete.

Causation is just causality: This paper, maybe co-authored with David Blei (Columbia) or Bernhard Schölkopf (MPI-IS) or Hill, shows that you don't need to have free will in order to have cogent causal explanations of data. That is, you don't need to phrase causality in terms of predictions for counter-factual experiments that you might have chosen to do.

You don't ever want evidence: This paper shows that any time you are computing the Bayesian evidence—what I call the fully marginalized likelihood (fml)—you are doing the wrong integral and solving the wrong problem. For both practical and theoretical (principled) reasons.

biases from machine learning

Today I gave a talk (with these slides) at a meeting in Denver for the NSF initiative Harnessing the Data Revolution. I spoke about the necessity and also the dangers of using machine-learning methods in scientific projects. I brought up two very serious possible biases. The first is that if emulators are used to replace simulations, and they can't be easily checked (because the simulation requirements are too expensive), the emulators will lead to a confirmation-bias problem: We will only carefully check the emulations if they lead to results that we don't like! The second bias I raised is that if we perform joint analyses on objects (stars, say) that have been labeled (with ages, say) by a machine-learning regression, there will in general be strong biases in those joint analyses. For example, the average value of 1000 age labels for stars labeled by a standard ML regression will not be anything like an unbiased estimate of the true average age of those stars. These biases are very strong and bad! That said, I also gave many example locations where using machine learning methods is not just okay but actually intellectually correct, in areas of instrument calibration, foregrounds, and other confounders.

The question period was great! We had 25 minutes of questions and answers, which ranged across a very wide set of topics, including statistics, experimental design, and epistemology.

machine-learning theory and practice

Today I got invited to be on a panel discussion (hosted by Soledad Villar of JHU) with Alberto Bietti (Flatiron) about the theory and practice of machine learning. It was great! We talked about why ML works for scientific applications, and Bietti said something (obvious maybe) that I loved: Maybe ML only works because of properties of the data. That is, maybe when we are analyzing ML methods we are looking in the wrong place, and we should be analyizing the data to which they are successfully applied? I made fun of interpretation in ML, and that led to interesting comments from both Bietti and the audience. Several audience members suggested taking something more like a causal approach to interpretation: How does the method work under interventions or in conditional situations? That's interesting; it isn't what a physicist would consider interpetation, but it might be sufficient in many cases.

causal structure in ML

Today I had the honor on serving on the PhD advising committee of Yan Liang (Princeton), who is designing her PhD project. She is adding causal structure to an autoencoder such that it can separate stellar variability-induced radial-velocity signals from exoplanet-induced signals in extreme precision radial-velocity data. Her method design is novel, and tests suggest that it might work. The committee recommended adding even more causal structure and physics knowledge (more is probably always better, provided that it isn’t incorrect)! As my loyal reader knows, I think this is the frontier for machine learning in the natural sciences: adding causal structure.

CMB component separation with linear fitting

Today I sat down with Fiona McCarthy (Flatiron) to look at data-driven methods for separating cosmic microwave background data into different components. We implemented a simple polynomial regression to fit foregrounds, using (observed) difference maps as inputs (features) that are designed to contain foregrounds only. We obtained some preliminary results that looked exciting but we’ve only just started. Part of the motivation is that CNNs are hard to train, but linear combinations of image monomials are easy! I realized in all this that there are connections to the group-equivariant stuff I’ve done with Villar’s group, because we use invariants, and also to the causal inference things that Schölkopf’s group does, because we’re trying to impose some causal structure on our functions.

nerve-wracking talk

I spent a ride down to Baltimore preparing a talk for mathematicians. That's outside my comfort zone. I gave the talk with the MINDS institute at Johns Hopkins at lunchtime. It was about passive symmetries, active symmetries, classical physics, and machine learning. There was no math. They only asked me, in the end, a few questions I couldn't answer. I hypothesized that the differences between passive and active symmetries is that the latter are statements about interventions.

SDSS-V Science Festival, day 2

On day two of the Collaboration meeting, I talked to Emily Griffith (Colorado) about data-driven models of nucleosynthetic processes. We were inspired by this paper, on which Griffith is an author. The paper builds an empirical two-process enrichment model based on the observed morphology of the [Fe/Mg] vs [Mg/H] plane. We discussed how to make this model into a full (but constrained) latent-variable model. I am interested in moving it towards causal inference, but we could also look at third processes, anomalous stars, anomalous elements, calibration issues, and so on. We wrote down math and started to write code.

Dagstuhl, day 3

Today was day 3 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl.

We had an open discussion about goals for ML in science today. The idea of explainability came up. I liked the comment that explainability (or what counts as explainability) might depend incredibly strongly on field or context. Like it is different in medicine and in astronomy. And, related, the idea of how models are communicated is very context dependent. And maybe very dependent on history. For example, in the future, models might be communicated through APIs rather than scientific papers maybe?

Causation and causal inference was a big theme of the day with Bernhard Schölkopf, Jonas Peters, Bubacar Bah, and Niki Kilbertus all talking about overlapping ideas in causal inference, mechanism inference, differential equation inference, and symbolic regression. Is causation the new framework for machine learning? Many in the room think so.

Dagstuhl, day 1

Today was day 1 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl. The first day was mainly about applications of machine learning, in Earth science, livestock management, astrophysics (dark matter), cells, and mechanical engineering. I had many thoughts and realizations. Here are a few random ones:

The problems that appear in Earth science, and the data types, are very similar to those that appear in astrophysics! But in Earth science, biology is a big driver of global processes, and there is no good mechanistic model for (say) how plants grow and take up carbon. The world is filled with mobile phones, with good cameras, and the methods we could could be employing to be doing science in a distributed way are way, way under-used. Cells are incredibly complicated. The mechanistic model involves literally thousands of individual processes. Like our model for the cell is as complicated as our model for the entire Earth system (which, by the way, depends on cells!), or even more complicated.

In the areas of the cell and the Earth, a theme was that the investigators want to preserve the causal structure we believe, and just use the machine learning to replace one tiny piece, with a data-driven model. Related: You can think of the machine learning as an effective theory for something (a sub-part of the problem) that doesn't work well from first principles. That's a good idea!

non-separable and generative random catalogs

Standard practice in large-scale structure is to make large-scale structure and cross-correlation measurements using a catalog of tracers (quasars in our case now) with random catalogs taking the role of tracking the selection function. In most cases this random catalog is made by sampling from a model for the angular selection function and, separately, for each object, sampling from a model for the radial selection function (redshift distribution in our case now). But of course the redshift distribution depends, in detail, on the angular selection function (because, for example, some of the angular selection is set by dust extinction). Kate Storey-Fisher (NYU) and I discussed now to capture these issues in the random we are building for the ESA Gaia quasar sample we are using. One idea is to give the randoms quasar-like luminosities and building the random catalog using our causal ideas about how things make it into the catalog

is it ever scientifically conservative to use machine learning?

I gave a talk Is machine learning good or bad for science? in Vienna today (slides here). I spent a lot of time on the ontology and epistemology of it all. One thing that led to some debate afterwards is my claim (at the end of the talk) that using extremely flexible machine learning methods can be extremely conservative in some cases: If you are modeling a nuisance that possibly interferes with your signal of interest, and you used a very flexible model, you have a strong argument that you tried as hard as you could (in some sense) to dilute your signal of interest with that nuisance. My talk was followed by interesting discussion with many, and a lovely dinner with Viennese (not just Austrian, but Viennese) wine.

causality and time ordering

I had a nice chat with David Blei (Columbia) at the end of the day about the question of whether causal inference (a subject in statistics) can be re-phrased in terms of making predictions about the time-ordering of events. He was not extremely positive about that project! But we talked about the causal-inference approaches. I don't like many of them! Because many of them somehow assume that it is possible to intervene on the situation, and how can you intervene on a unitary system (like, say, the Universe)? Does causality not exist in physics? Does the force cause the acceleration or does the acceleration cause the force? There isn't an answer to that in physics.

are data-driven approaches to RV measurement biased?

Matt Daunt (NYU) is re-building our wobble code for measuring stellar radial velocities without any stellar or tellurics model. He is finding that it is slightly biased towards smaller radial velocity amplitudes than what we inject into fake data. This also mirrors things that Bedell (Flatiron) and I have seen in various experiments. I think there is something going on with spectral edges: At the edge of the observed spectral domain, some observations have lines shifted into and out of the observations. The mean spectrum obtained from those measurements isn't necessarily capturing all of this fairly. Or at least it has to be handled carefully. Are we doing this right? Experiments we have suggest that we don't have this quite right yet.

generative models for quasars

I spent part of the day working with Christina Eilers on her Gaussian process latent-variable model for quasar spectra and physical properties. We re-wrote our title and abstract and went through the math in the paper. It's time to finish this up! We find that we can predict quasar masses with good accuracy (based on held-out data) based on single-epoch, limited-coverage optical spectra. It's sweet. And Eilers has beautiful demonstrations that she can predict unobserved spectral regions, because the model is trained on different quasars at different redshifts with different data. The big problem with this model is that it scales poorly; we can't imagine training on thousands of objects without substantial engineering efforts (and maybe not ever).

exoplanet atmospheres projects

I had a nice conversation today with Laura Kreidberg (MPIA) and Jason Dittmann (MPIA) about projects in ultra-precise spectroscopy for exoplanet-atmosphere science. I was pushing for projects in which we get closer to the metal (moving spectroscopy to two dimensions), and in which we use theoretical ideas to improve extraction of very weak spectral signals. Kreidberg was pushing for projects in which we use planet kinematics to separate very weak planet signals from star signals. On the latter, I recommended starting with co-ads of residuals.

talk on instrument calibration

I spent most of the day preparing this talk on instrument calibration, which I gave in the Königstuhl Colloquium this afternoon. I got excellent questions and I actually enjoyed it, despite the zoom/hybrid format, which doesn't always work well.

statistics translation project?

I had a wide-ranging conversation today with former NYU undergraduate Hilary Gao. One thing we discussed is the idea that physicists (and biologists, chemists, and so on) know a lot about statistics and evidence, and yet often find it hard to understand social-science research (like the epidemiology around Coronavirus and the data around race and policing, for two contemporary examples). This is (in my opinion) partly because the social-science literature involves aspects of causal inference that are fundamental, but don't appear in the same form in the natural sciences. We discussed what it would take to usefully write about or intervene into this quasi-translation project.

how to simulate a spectrum

I had a great conversation today with Matt Daunt (NYU), building on discussion yesterday with also Megan Bedell (Flatiron), about how to simulate data from an extreme-precision radial-velocity spectrograph. We decided to simulate the star, the atmosphere, and the (gasp!) gas cell all at very high resolution, then combine them physically, then reduce resolution to the spectrograph resolution (which is very high nonetheless) and then sample and noisify the resulting data. The idea is: Make the structure of the code like the structure of our physical beliefs, or causal beliefs. We decided to fork this data simulation into its own project.