The LLMs, and why do we do astrophysics?

Today my rant on LLMs and the practices of our field hit the arXiv. I was scared to post it, because it is such a weird contribution, and it is so revealing about myself and my own political positions and hangups. But I have to say: I got great and supportive feedback all day.

I got two comments on saying ACAB in the literature. The Astronomer Royal of Scotland quoted (on BlueSky) the last sentence, which I put there because Andy Casey (Monash, Flatiron) insisted. Many people sent me appreciation and thank-yous, and many people sent me comments and objections. Always constructive. The whole experience made me feel very happy about the state of our field and the way we all interact. I think maybe there will be critical mass to write some kind of collection of essays on the subject. That's a plan for 2026.

the transparency of the Universe and the transparency of the university

The highlight of my day was a wide-ranging conversation with Suroor Gandhi (NYU) about cosmology, career, and the world. She made a beautiful connection between a part of our conversation in which we were discussing the transparency of the Universe, and new ways to study that, and a part in which we were discussing the transparency with which the University speaks about disciplinary and rules cases, which (at NYU anyway) is not very good. Hence the title of this post. On transparency of the Universe, we discussed the fact that distant objects (quasars, say) do not appear blurry must put some limit on cosmic transparency. On transparency of the University, we discussed the question of how much do we care about the behavior of our institutions, and changing those behaviors. I'm a big believer in open science, open government, and open institutions.

I've been privileged these years to have some very thoughtful scientists in my world. Gandhi is one of them.

why study astrophysics?

I spent the day with Neige Frankel (CITA), working on various projects. One of the things we discussed was her slides for an upcoming talk. I made the following blanket statement; is it true? There are only two ways to ultimately justify a subject of study in astrophysics. Either it will tell us something important about fundamental physics (think: dark matter, initial conditions of the Universe, or nucleosynthesis, say), or else it will tell us something about our origins (formation of our Galaxy, occurrence of rocky, habitable planets, origin of life, say). I am not entirely sure this is right, but I can't currently think of much in the way of counter-examples. I guess one other justification might be that we are developing technologies that will help people in other areas (CCDs, spacecraft attitude management, or machine learning, say).

informal scientific communication

I have been sending out my draft manuscript on machine learning in the natural sciences to various people I know who have opinions on this. I've been getting great feedback, and it reminds me that there is a lot of important scientific communication that is on informal channels. One thing that interests me: Is there a way to make such conversation more public and viewable and research-able?

philosophy

I've been working on two philosophical projects this month. The first has been an interaction with Jim Peebles (Princeton) around a paper he has been writing, setting down his philosophy of physics. I am pretty aligned with his position, which I expect to hit the arXiv soon. I'm not a co-author of that. But one of the interesting things about science is how much of our work in in anonymous (or quasi-anonymous) support of others.

The second philosophical project is a paper about machine learning and science: I am trying to set down my thoughts about how ML can and can't help the sciences. This is fundamentally a philosophy-of-science question, not a science question.

try bigger writing

I have been buried in job season and other people's projects. That's good! Hiring and advising are the main things we do in this job. But I decided today that I need to actually start a longer writing project that is my own baby. So I started to turn the set of talks I have been giving about machine learning and astrophysics into a paper. Maybe for the new ICML Position Paper call?

radical papers I want to write (or will never write)

I have to finish my NSF proposal with Mike Blanton (NYU), so naturally I am in procrastination mode. Here are three papers I wish I would write. Maybe I should post them on my ideas blog:

Occam's Razor is wrong: This paper, co-authored with Jennifer Hill (NYU), would be about the fact that, in the real, observed world, the simplest explanation is always wrong or at least incomplete.

Causation is just causality: This paper, maybe co-authored with David Blei (Columbia) or Bernhard Schölkopf (MPI-IS) or Hill, shows that you don't need to have free will in order to have cogent causal explanations of data. That is, you don't need to phrase causality in terms of predictions for counter-factual experiments that you might have chosen to do.

You don't ever want evidence: This paper shows that any time you are computing the Bayesian evidence—what I call the fully marginalized likelihood (fml)—you are doing the wrong integral and solving the wrong problem. For both practical and theoretical (principled) reasons.

M dwarfs

I had a great phone call with Madyson Barber (UNC) and Andrew Mann (UNC) today about M dwarf stellar spectroscopy. I love the problem of understanding the spectra of M dwarfs because this is a subject where there is no ground truth: No physical models of M dwarf photospheres work very well! Why not? Probably because they depend on lots of molecular transitions and band heads, the properties of which are not known (and very sensitive to conditions).

I love problems where there is no ground truth! After all, science as a whole has no ground truth! So the M-dwarf spectroscopy problem is a microcosm of all of science. I went off the deep end on this call, and we were all left knowing less than we knew when we started the call. By this post, I apologize to Barber and Mann.

biases from machine learning

Today I gave a talk (with these slides) at a meeting in Denver for the NSF initiative Harnessing the Data Revolution. I spoke about the necessity and also the dangers of using machine-learning methods in scientific projects. I brought up two very serious possible biases. The first is that if emulators are used to replace simulations, and they can't be easily checked (because the simulation requirements are too expensive), the emulators will lead to a confirmation-bias problem: We will only carefully check the emulations if they lead to results that we don't like! The second bias I raised is that if we perform joint analyses on objects (stars, say) that have been labeled (with ages, say) by a machine-learning regression, there will in general be strong biases in those joint analyses. For example, the average value of 1000 age labels for stars labeled by a standard ML regression will not be anything like an unbiased estimate of the true average age of those stars. These biases are very strong and bad! That said, I also gave many example locations where using machine learning methods is not just okay but actually intellectually correct, in areas of instrument calibration, foregrounds, and other confounders.

The question period was great! We had 25 minutes of questions and answers, which ranged across a very wide set of topics, including statistics, experimental design, and epistemology.

Jim Peebles

Today was a celebration in Princeton for Jim Peebles (Princeton) and his 2019 Nobel Prize. As my loyal reader knows, I hate the Nobel Prize, and I say so in the slides from my talk. But I love Jim Peebles, who has been incredibly important to my career and life. I spoke about epistemology and large-scale structure.

In the other presentations during the day, Suzanne Staggs (Princeton) gave a deep and hilarious picture of the early days of CMB cosmology. Would she be upset to hear me call her early career the “early days” of CMB? Vicky Kaspi (McGill) showed an amazing result from the study of fast radio bursts: The rotation measures to the bursts increase with redshift in exactly the way you would expect from the cosmological baryon density and the world model in LCDM. That's incredible! Frans Pretorius (Princeton) gave a great talk about numerical relativity, in which he showed almost no numerical relativity computations! He talked about what might happen in the fully relativistic version of the black-hole-black-hole merger problem, in which the incoming black holes have their mass energies overwhelmed by the center-of-mass kinetic energies. He came up with many possible outcomes and explained why the answers aren't known. The answers involve incredibly qualitatively different outcomes!

Dagstuhl, day 4

Today was day 4 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl.

I spoke today, about passive symmetries and the constraints on machine-learning models they imply. My talk was totally new for me, and based on conversations between Villar, Schölkopf, and me during the meeting. That was fun. So now I have a new way of talking about all this stuff, and the three of us are trying to write a short paper about it.

Among the talks today, one idea I really liked is the idea, from Carl Henrik Ek, that trust and interpretability might be strongly related. Indeed, when I talk about interpretability, it is often in the same context that I am talking about models that make sense to a physicist, which are, in turn, models that I would trust. And that is also very related to what I myself talked about today: If models look more like physical law, then they are much more trustworthy. And maybe also more interptetable.

Dagstuhl, day 3

Today was day 3 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl.

We had an open discussion about goals for ML in science today. The idea of explainability came up. I liked the comment that explainability (or what counts as explainability) might depend incredibly strongly on field or context. Like it is different in medicine and in astronomy. And, related, the idea of how models are communicated is very context dependent. And maybe very dependent on history. For example, in the future, models might be communicated through APIs rather than scientific papers maybe?

Causation and causal inference was a big theme of the day with Bernhard Schölkopf, Jonas Peters, Bubacar Bah, and Niki Kilbertus all talking about overlapping ideas in causal inference, mechanism inference, differential equation inference, and symbolic regression. Is causation the new framework for machine learning? Many in the room think so.

Dagstuhl, day 2

Today was day 2 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl. Many great things happened. Here are two highlights:

Bernhard Schölkopf (MPI-IS), in a discussion session, asked what the key questions were for machine learning as a field. I love this question! Astronomy and physics do, I think, have key questions, which guide research and contextualize choices. Machine learning does not really, or if it does, the questions are implicit. I want to work on this.

Philipp Hennig (Tübingen) gave an energizing talk about the relationship between simulations of the world and observations of (or data about) the world. He argued (convincingly!) that we should not think of these as totally different things, and that learning from data and simulating a process could or even should always be integrated and done together. He demonstrated this with a simple model of infectious disease, but the point is extremely general.

argh new writing projects?

Oh no! I spent the weekend accidentally starting new writing projects. What's wrong with me? One of the things that's wrong with me is that I am about to start teaching a new PhD-level class Statistics and Data Science for Physics and I find that I don't have good reading materials for the students. Here's a lack: A good, sensible discussion of when to take a frequentist approach in your data analysis, and when to take a Bayesian approach, divorced from (or not emphasizing) the philosophical differences.

what to say for Jim Peebles

I have been honored by an invitation to speak at a meeting in honor of Jim Peebles (Princeton) and his 2019 Nobel Prize in Physics. I spent the day working on things I want to say at this event. Obviously I have to be very critical of the Nobel Prize and all prizes! Haha. But I want to talk about large-scale structure and also the problem that we only ever get to observe one Universe. What does that mean for inductive reasoning and epistemology?

so much Gaia; #renameJWST

I spent the day working on ESA Gaia data, parallel to Kate Storey-Fisher (NYU). She was working on the quasar catalog and the correlation with the CMB convergence maps; I was working on estimating stellar luminosities from low-resolution spectral coefficients. We are too much in the thick of it to report how it's going yet. But stellar luminosities are hard to predict from spectra!

Late in the day I sent this letter to NASA:

I served on the US Astronomy and Astrophysics Advisory Committee (AAAC) for several years; I served on the NASA Spitzer Space Telescope Oversight Committee for many years; I served on the NASA Extragalactic Database User Committee for several years; and my research has been funded by NASA for my entire career (since I was a PhD student in the 1990s). I currently do research with HST, Kepler, TESS, 2MASS, WISE, and WMAP data, and now I'm getting ready for JWST and SPHEREx.

I am writing to say that I think it would serve NASA's interests, and the interests of NASA science especially, to rename JWST. There have been plenty of discussions of the name; it is clear that many scientists (and especially those who are part of the LBGT community or who have concerns for the LBGT community) feel disrespected by the name. I also personally think that the evidence is clear that some of the career activities of James Webb did direct harm to patriotic Americans who were gay.

I want to emphasize, however, that I think the important argument about the name goes beyond the question of any individual historical facts: The LBGT communities are of great importance to all of us. These voices must be heard, and the legitimate concerns must be addressed.

Because NASA is a forward-looking agency, and working towards a more equitable, better world, especially for people working in science and engineering in the US, I think it is time that the spacecraft be renamed. I think this could be done easily and without any trouble; many spacecraft have either officially or effectively changed names at or around first light, two examples that come immediately to mind are WMAP and Spitzer.

Doppler shifts and radial velocities

I am a big believer that there is a difference between a Doppler shift and a radial velocity. For one, they have different units! For another, the former is measurable (sometimes) and the latter is not (or rarely). But today I agreed with Megan Bedell (Flatiron) that we should write our paper on the subject in terms of the words “radial velocity” and not “Doppler shift”. After all, we are talking to a community with a common language! I spent some time on the train from Vienna to Heidelberg editing the figures for our paper on this subject.

is it ever scientifically conservative to use machine learning?

I gave a talk Is machine learning good or bad for science? in Vienna today (slides here). I spent a lot of time on the ontology and epistemology of it all. One thing that led to some debate afterwards is my claim (at the end of the talk) that using extremely flexible machine learning methods can be extremely conservative in some cases: If you are modeling a nuisance that possibly interferes with your signal of interest, and you used a very flexible model, you have a strong argument that you tried as hard as you could (in some sense) to dilute your signal of interest with that nuisance. My talk was followed by interesting discussion with many, and a lovely dinner with Viennese (not just Austrian, but Viennese) wine.

is machine learning good or bad for science?

I took the train to Vienna today, for a PhD defense and to give a talk about machine learning. My talk is interdisciplinary so I looked at how to generalize my arguments about astrophysics to all of the natural sciences. It turns out that this isn't as easy as I'd like, since it is hard to be specific outside of astrophysics! I'm going to learn a lot getting this talk ready.

coordinate freedom?

I spent the weekend recovering from the Gaia Fete. During my recovery day, I worked on very long-term projects. For example, I spent some time working on how to express the following issue in my work (with Villar) on exact symmetries:

The mathematics and computer-science communities call these exact symmetries “equivariances” and they are imagining that the data or the laws of physics are precisely equivariant in the sense that if you (say) rotate all the inputs, you get a rotated output. But this is not the main reason that we write the laws of physics in terms of exact symmetries! We write the laws of physics in terms of invariants because we want our laws of physics to be coordinate free. This is required even when the laws aren't equivariant! But I have trouble making this distinction clearly, since the mathematical implementations of the two symmetries are identical. There's some cool philosophy here: Does coordinate freedom enforce symmetries? What would it even look like for the laws of physics to be asymmetric but coordinate free?