IAIFI Symposium, day two

Today was day two of a meeting on generative AI in physics, hosted by MIT. My favorite talks today were by Song Han (MIT) and Thea Aarestad (ETH), both of whom are working on making ML systems run ultra-fast on extremely limited hardware. Themes were: Work at low precision. Even 4-bit number representations! Radical. And bandwidth is way more expensive than compute: Never move data, latents, or weights to new hardware; work as locally as you can. They both showed amazing performance on terrible, tiny hardware. In addition, Han makes really cute 3d-printed devices! A conversation at the end that didn't quite happen is about how Aarestad's work might benefit from equivariant methods: Her application area is triggers in the CMS device at the LHC; her symmetry group is the Lorentz group (and permutations and etc). The day started with me on a panel in which my co-panelists said absolutely unhhinged things about the future of physics and artificial intelligence. I learned that many people think we are only years away from having independently operating, fully functional aritificial physicists that are more capable than we are.

combining spectral exposures

I wrote words! I got back to actually doing research this week, in part inspired by a conversation with my very good friend Greg McDonald (Rum & Code). I worked on the words in the paper I am finishing with Andy Casey (Monash) about how to combine individual-visit exposures into a mean spectrum. The biggest writing job I did today was the part of the paper called “implementation notes”, which talks about how to actually implement the math on a finite computer.

Florida, day one

I spent today with Sarah Ballard's group, plus others, at the University of Florida. I gave a talk, to a large, lively, and delightful audience. At the end of this talk I was very impressed by the following thing: Ballard had everyone in the room discuss with their neighbors (turn and talk) for about 3 minutes, after the seminar but before the question period began! This is a technique I use in class sometimes; it increases participation. After those 3 minutes, audience members had myriad questions, as one might imagine.

I spoke with many people in the Department about their projects. One highlight was Jason Dittman, who showed me gorgeous evidence that a particular warm exoplanet on an eccentric orbit has an atmosphere that undergoes some kind of phase change at some critical insolation, as it moves away from its host star on its orbit. Crazy!

Late in the day I discussed n-point functions and other cosmological statistics with Zach Slepian and Jiamin Hou. We discussed the plausibility of getting tractable likelihoods for any n-point functions. We also discussed the oddity that n-point functions involve sums over n-star configurations among N stars (N choose n), but there are mathematical results that show that any permutation-invariant function of any point cloud can be expressed with only a sum over stars (N). That sounds like a research problem!

biases from machine learning

Today I gave a talk (with these slides) at a meeting in Denver for the NSF initiative Harnessing the Data Revolution. I spoke about the necessity and also the dangers of using machine-learning methods in scientific projects. I brought up two very serious possible biases. The first is that if emulators are used to replace simulations, and they can't be easily checked (because the simulation requirements are too expensive), the emulators will lead to a confirmation-bias problem: We will only carefully check the emulations if they lead to results that we don't like! The second bias I raised is that if we perform joint analyses on objects (stars, say) that have been labeled (with ages, say) by a machine-learning regression, there will in general be strong biases in those joint analyses. For example, the average value of 1000 age labels for stars labeled by a standard ML regression will not be anything like an unbiased estimate of the true average age of those stars. These biases are very strong and bad! That said, I also gave many example locations where using machine learning methods is not just okay but actually intellectually correct, in areas of instrument calibration, foregrounds, and other confounders.

The question period was great! We had 25 minutes of questions and answers, which ranged across a very wide set of topics, including statistics, experimental design, and epistemology.

planets forming in a disk

At the end of last week I had a great conversation with Valentina Tardugno (NYU) and Phil Armitage (Flatiron) about how planets form. I spent the whole weekend thinking about it: If a few planets are forming in a proto-planetary disk, there are all sorts of interactions between the planets and the disk, and the planets and each other, and the disk with itself. You can think of this (at least) two different ways:

You can think of planets which are interacting not just directly with one another, but also with a disk, and with each other as mediated by that disk. This is the planet-centric view. In this view, the planets are what you are tracking, and the disk is a latent object that makes the planets interact and evolve.

Alternatively, you can think of the disk, with planets in it. In this disk-centric view, the planets are latent objects that modify the disk, creating gaps and spiral waves.

Both views are legitimate, and both have interesting science questions. We will explore and see where to work. I am partial to the planet-centric view: I want to know where planetary systems come from!

teeny tiny cosmological simulations.

Connor Hainje (NYU) is looking at this paper by Chen et al which uses a machine-learning regression to interpolate between cosmological simulation outputs at different cosmological epochs. To build an end-to-end pipeline for testing ideas, he has been running 32-cubed cosmological simulations. These might be the smallest simulations run since the 1980s! But, interestingly, he is finding that the interpolation isn't working great. Is this because it is harder to train a regression on a small simulation than it is on a large simulation? Is a small simulation less predictable or less interpolate-able? It's expensive to find out!

making linear algebra faster in practice

The key thing to make your code run faster is to avoid building large linear-algebra objects. For example, if you need to get the matrix product A.x, where A is a huge matrix and x is a long vector, and you only ever use the matrix A to do this one multiply by x, there is no reason to actually create A. Just create a function that evaluates A.x for any input x. That should be way faster, because of less memory allocation, and because you don't have to make the parts of the matrix that are all zeros (for example). Matt Daunt (NYU) and I discussed all this at length today, as we profiled code. This comment has some overlap with Section 9 of this paper.

high-order integration schemes

I was working on a white paper on ocean dynamics today and I threw in a sentence about how emulators (like machine-learning replacements for simulations) might be working because they might be effectively learning a high-order integration method. I then threw in a sentence about how, in many applications, high-order integrators are known to be better than low-order integrators. I then went to find a reference and... well, I am not sure I can back that up with a reference! I thought this was common knowledge, but it looks like almost all simulations and integrations are done with low-order integrators. Am I living in a simulation? (A simulation integrated with wimpy first-order integrators?)

four kinds of emulators

I wrote in a draft grant proposal related to machine-learning emulators today. I wrote about five different kinds of emulators. Yes I think there are five qualitatively distinct kinds. Here they are:

Full replacement

The most extreme—and most standard—kind of emulator is one that simply replaces the full input–output relationship of the entire simulation. Thus if the simulation starts with initial conditions and boundary conditions, and ends with a final state (after an integration), the full-replacement emulator would be trained to learn the full relationship between the initial and boundary conditions and the final state. A full-replacement emulator is a complete, plug-in replacement for the simulator.

Integrator

Simulation run times generally scale linearly with the number of time steps required to execute the integration. A set of emulators can be trained on a set of snapshots of the simulation internal state at a set of times that is much smaller than the full set of integration time steps. Each emulator is trained to learn the relationship between the internal state of the simulation at one time tA and the internal state of the simulation at a later time tB, such that the emulator can be used to replace the integrator during the time interval from tA to tB. A set of such emulators can be used to replace part or all of the integration performed by the simulator.

Resolution translator

Simulation run times generally scale with the number of grid points or basis functions in the representations of the state. Thus the simulator gets faster as resolution is reduced. An emulator can be trained to learn the relationship between a low-resolution simulation and a matched high-resolution simulation. Then a high-resolution simulation can be emulated by running a fast low-resolution simulation and applying the learned translation.

Physics in-painter

In most physical systems, there are coupled physics domains with different levels of computational complexity. For example, in cosmology, the pure gravitational part of the simulation is relatively low in computational cost, but the baryonic part—the atoms, photons, ram pressures, magnetic fields—is very high in computational cost. The simulator gets faster as physics domains, or equations, or interaction terms, are dropped. An emulator can be trained to learn the relationship between a simulation with some physics dropped and a matched full simulation. Then a full-physics simulation can be emulated by running a partial-physics simulation and applying the learned in-painting of the missing physics.

Statistics generator

In many contexts, the goal of the simulation is not to produce the full state of the physical system, but only certain critical statistics, such as the two-point correlation function (in the case of some cosmology problems). In this case, there is no need to emulate the entire simulation state. Instead, it makes sense to train the emulator to learn only the relationship between the initial and boundary conditions of the simulation and the final statistics of particular interest.

building trust in emulators

I started writing in a possible grant proposal (that would be in collaboration with others) about the trustworthiness of machine-learning emulators. Emulators are systems that learn the input–output relationship of a computationally expensive simulation and produce (or speed the computation of) new simulation outputs, reducing total computational requirements for a given number of simulations. These are so important now that the ESA Euclid and Simons Observatory data-analysis plans crucially involve emulation.

The issue is: How do we trust that the emulators are giving good outputs? There is no obvious way to test them, except by comparing to held-out training data. But in large-scale structure contexts, no amount of held-out data can test the enormous input data space. I don't know how we will ever trust such systems (and damn do we need to!), but I have some ideas about how to improve the situation. One involves enforcing physics symmetries on the emulators. Another involves running adversarial attacks on them.

convection, granulation, stellar spectra

I had a great and long conversation today with Maria Bergemann (MPIA) about building a model of a full stellar spectrum out of the models they build of small patches of stellar surface, with full 3D convection and full radiative transfer. Their models are sophisticated, and give a full spectrum in every direction from the surface patch. Thus we can integrate a set of patches into a surrogate combined spectrum for one rotating star covered in convecting patches. We discussed how we might do that, technically, and what projects we might then do with the output. What I want to do (yes, you guessed it) is build data-driven models of stellar granulation to improve radial-velocity surveys.

Dr Irina Espejo

Today it was my honor to serve on the PhD defense committee of Irina Espejo (NYU), who is one of the first (ever in the world, actually!) PhDs in Data Science. Her PhD research involved making real, practical, scalable, reproducible tools for the (late-in-pipeline) analysis of high-energy physics data from the Large Hadron Collider. She built tools to speed up likelihood-free inferences, and she built a tool to find exclusion regions (upper limits) in complex parameter spaces. She used the latter to put constraints on a (real, not toy) proposed modification to the standard model.

On the first project, the tools that she built (and built on) make the LHC more sensitive to new physics, because they find better test statistics for distinguishing models. They make some searches far better, which makes me wonder whether particle physics is using our money efficiently??

MNIST-minus-minus

Imagine that you want to read all the text in an arbitrary image of the world. That text will lie at different locations, rotations, shears, and even reflections (think signage painted on windows! or mirrors!) in an image. As training for a baby problem in this area I made this training set today.

doing cosmology differently

Today Chirag Modi (Flatiron) gave a really great lunchtime talk about new technologies in cosmology and inference or measurement of cosmological parameters. He beautifully summarized how cosmology is done now (or traditionally): Make summary statistics of the observables, make a theory of the summary statistics, make up a surrogate likelihood function for use in inference, measure covariance matrices to use in the latter, and go. He's trying to obviate all of these things by using the simulations directly to make the measurements. He has nice results in forward modeling of the galaxy field, and in simulation-based inferences. Many interesting things came up in his talk, including the idea that I have discussed over the years with Kate Storey-Fisher (NYU) of enumerating all possible cosmological statistics! So much interesting stuff in the future of large-scale structure.

geometric convolutional networks

Today I had a great meeting with Wilson Gregory (JHU), Drummond Fielding (Flatiron), and Soledad Villar (JHU) about a project to learn partial differential equations from simulation data. This is a toy problem from our perspective, but it is a baby step towards big, real computational problems in physics. Fielding produces training data, Gregory produces geometric convolution networks, Villar proves things, and I cheer from the sidelines.

Our approach is to replace convolutional neural networks with geometric operators that are generalizations of convolutions that know more about geometric forms like vectors, tensors, pseudovectors, and so on. By building methods that use geometric objects responsibly, we automatically enforce coordinate freedom and other deep symmetries.

Jim Peebles

Today was a celebration in Princeton for Jim Peebles (Princeton) and his 2019 Nobel Prize. As my loyal reader knows, I hate the Nobel Prize, and I say so in the slides from my talk. But I love Jim Peebles, who has been incredibly important to my career and life. I spoke about epistemology and large-scale structure.

In the other presentations during the day, Suzanne Staggs (Princeton) gave a deep and hilarious picture of the early days of CMB cosmology. Would she be upset to hear me call her early career the “early days” of CMB? Vicky Kaspi (McGill) showed an amazing result from the study of fast radio bursts: The rotation measures to the bursts increase with redshift in exactly the way you would expect from the cosmological baryon density and the world model in LCDM. That's incredible! Frans Pretorius (Princeton) gave a great talk about numerical relativity, in which he showed almost no numerical relativity computations! He talked about what might happen in the fully relativistic version of the black-hole-black-hole merger problem, in which the incoming black holes have their mass energies overwhelmed by the center-of-mass kinetic energies. He came up with many possible outcomes and explained why the answers aren't known. The answers involve incredibly qualitatively different outcomes!

models for images; talking about stuff in progress

As my loyal reader knows, I have been working on new machine-learning methods for imaging, based on scalars, vectors, and tensors, and group-theoretic invariants. Today I had a long conversation with Kaze Wong (Flatiron) about these things, with ideas about implementation and model structure. With Soledad Villar (JHU) I have been thinking about convolutional layers, and nonlinearities, but Wong wanted to talk about full-network architecture. It's hard! So we discussed, and realized that we need to do some serious work if we are going to find a way to implement something general and useful and computationally tractable.

Before that, in Astronomical Data Group meeting, I got tons of great feedback about my project with Andy Casey (Monash) to combine shifted spectra into a rest-frame average spectrum. I described the project in the words I like to use, and the audience heard something totally different! So I know what I have to be emphasizing in the writing. I can't say enough how important it is to talk to people about your science in progress!

Dagstuhl, day 3

Today was day 3 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl.

We had an open discussion about goals for ML in science today. The idea of explainability came up. I liked the comment that explainability (or what counts as explainability) might depend incredibly strongly on field or context. Like it is different in medicine and in astronomy. And, related, the idea of how models are communicated is very context dependent. And maybe very dependent on history. For example, in the future, models might be communicated through APIs rather than scientific papers maybe?

Causation and causal inference was a big theme of the day with Bernhard Schölkopf, Jonas Peters, Bubacar Bah, and Niki Kilbertus all talking about overlapping ideas in causal inference, mechanism inference, differential equation inference, and symbolic regression. Is causation the new framework for machine learning? Many in the room think so.

Dagstuhl, day 2

Today was day 2 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl. Many great things happened. Here are two highlights:

Bernhard Schölkopf (MPI-IS), in a discussion session, asked what the key questions were for machine learning as a field. I love this question! Astronomy and physics do, I think, have key questions, which guide research and contextualize choices. Machine learning does not really, or if it does, the questions are implicit. I want to work on this.

Philipp Hennig (Tübingen) gave an energizing talk about the relationship between simulations of the world and observations of (or data about) the world. He argued (convincingly!) that we should not think of these as totally different things, and that learning from data and simulating a process could or even should always be integrated and done together. He demonstrated this with a simple model of infectious disease, but the point is extremely general.

is machine learning good or bad for science?

I took the train to Vienna today, for a PhD defense and to give a talk about machine learning. My talk is interdisciplinary so I looked at how to generalize my arguments about astrophysics to all of the natural sciences. It turns out that this isn't as easy as I'd like, since it is hard to be specific outside of astrophysics! I'm going to learn a lot getting this talk ready.