The LLMs, and why do we do astrophysics?

Today my rant on LLMs and the practices of our field hit the arXiv. I was scared to post it, because it is such a weird contribution, and it is so revealing about myself and my own political positions and hangups. But I have to say: I got great and supportive feedback all day.

I got two comments on saying ACAB in the literature. The Astronomer Royal of Scotland quoted (on BlueSky) the last sentence, which I put there because Andy Casey (Monash, Flatiron) insisted. Many people sent me appreciation and thank-yous, and many people sent me comments and objections. Always constructive. The whole experience made me feel very happy about the state of our field and the way we all interact. I think maybe there will be critical mass to write some kind of collection of essays on the subject. That's a plan for 2026.

how significant is your anomaly?

So imagine that you have a unique data set Y, and in that data set Y you measure a bunch of parameters θ by a bunch of different methods. Then you find, in your favorite analysis, your estimate of one particular parameter is way out of line: All of physics must be wrong! How do you figure out the significance of your result?

If you only ever have data Y, you can't answer this question very satisfactorily: You searched Y for an anomaly, and now you want to test the significance. That's why so many a posteriori anomaly results end up going away: That search probably tested way more hypotheses than you think it did, so any significances should be reduced accordingly.

The best approach is to use only part of your data (somehow) to search, and then use a found anomaly to propose a hypothesis test, and then test that test in the held-out or new data. But that often isn't possible, or it is already too late. But if you can do this, then there is usually a likelihood ratio that is decisive about the significance of the anomaly!

I discussed all these issues today with Kate Storey-Fisher (Stanford) and Abby Williams (Chicago) today, as we are trying to finish a paper on the anomalous amplitude of the kinematic dipole in quasar samples.

wrote like the wind; frequentist vs Bayes on sparsity

My goal this year in Heidelberg is to move forward all writing projects. I didn't really want to start new projects, but of course I can't help myself, hence the previous post. But today I crushed the writing: I wrote four pages in the book that Rix (MPIA) wants me to write, and I got more than halfway done with a Templeton Foundation pre-proposal that I'm thinking about, and I partially wrote up the method of the robust dimensionality reduction that I was working on over the weekend. So it was a good day.

That said, I don't think that the iteratively reweighted least squares implementation that I am using in my dimensionality reduction has a good probabilistic interpretation. That is, it can't be described in terms of a likelihood function. This is related to the fact that frequentist methods that enforce sparsity (like L1 regularization) don't look anything like Bayesian methods that encourage sparsity (like massed priors). I don't know how to present these issues in any paper I try to write.

should I write a book?

Is it research to have a set of conversations about whether to write a book on data analysis? Hans-Walter Rix (MPIA) thinks I should put together my arXiv-only submissions plus a lot more into a book about data analysis. His point of view is that the most important thing is how to convert an ill-posed question about the Universe into a well-posed operation on data.

combining spectral exposures

I wrote words! I got back to actually doing research this week, in part inspired by a conversation with my very good friend Greg McDonald (Rum & Code). I worked on the words in the paper I am finishing with Andy Casey (Monash) about how to combine individual-visit exposures into a mean spectrum. The biggest writing job I did today was the part of the paper called “implementation notes”, which talks about how to actually implement the math on a finite computer.

what book am I going to write?

One possible new year's resolution this year is for me to decide which book am I going to write? I don't love this, because it is the hallmark of a scientist at the end of the career that they switch to writing books! I guess maybe I'm at the end of my career? But that said, I have (maybe like many scientists at the end of their careers?) a lot to say. Okay anyway, I had a long conversation this morning with Greg McDonald (Rum&Code) about all this, and he strongly encouraged me to make some content for the project code-named ”The Practice of Astrophysics“.

informal scientific communication

I have been sending out my draft manuscript on machine learning in the natural sciences to various people I know who have opinions on this. I've been getting great feedback, and it reminds me that there is a lot of important scientific communication that is on informal channels. One thing that interests me: Is there a way to make such conversation more public and viewable and research-able?

partial differential equations

I am trying to write a proposal to fund the research I do on machine-learning theory. The proposal is to work on ocean dynamics. It's a great application for the things we have done! But it's hard to write a credible proposal in an area that's new to you. Interdisciplinarity and agility is not rewarded in the funding system at present! At least I am learning a ton as I write this.

philosophy

I've been working on two philosophical projects this month. The first has been an interaction with Jim Peebles (Princeton) around a paper he has been writing, setting down his philosophy of physics. I am pretty aligned with his position, which I expect to hit the arXiv soon. I'm not a co-author of that. But one of the interesting things about science is how much of our work in in anonymous (or quasi-anonymous) support of others.

The second philosophical project is a paper about machine learning and science: I am trying to set down my thoughts about how ML can and can't help the sciences. This is fundamentally a philosophy-of-science question, not a science question.

try bigger writing

I have been buried in job season and other people's projects. That's good! Hiring and advising are the main things we do in this job. But I decided today that I need to actually start a longer writing project that is my own baby. So I started to turn the set of talks I have been giving about machine learning and astrophysics into a paper. Maybe for the new ICML Position Paper call?

grant proposals

There is a non-wrong view of academic science that it is all about applying for funding, and evaluating the proposals of others for funding. That's all I did today (evaluated proposals for a foreign funding program; I submitted my own proposal to the NSF yesterday).

radical papers I want to write (or will never write)

I have to finish my NSF proposal with Mike Blanton (NYU), so naturally I am in procrastination mode. Here are three papers I wish I would write. Maybe I should post them on my ideas blog:

Occam's Razor is wrong: This paper, co-authored with Jennifer Hill (NYU), would be about the fact that, in the real, observed world, the simplest explanation is always wrong or at least incomplete.

Causation is just causality: This paper, maybe co-authored with David Blei (Columbia) or Bernhard Schölkopf (MPI-IS) or Hill, shows that you don't need to have free will in order to have cogent causal explanations of data. That is, you don't need to phrase causality in terms of predictions for counter-factual experiments that you might have chosen to do.

You don't ever want evidence: This paper shows that any time you are computing the Bayesian evidence—what I call the fully marginalized likelihood (fml)—you are doing the wrong integral and solving the wrong problem. For both practical and theoretical (principled) reasons.

writing proposal

Mike Blanton (NYU) and I are writing an NSF proposal. That took up most of my research time today!

toning down my language

I spent travel time (at airports and on airplanes) working on the title, abstract, and introduction of the forthcoming paper with Andy Casey (Monash) about combining visit spectra into mean spectra. This was mainly about me changing the tone from “You are all doing it wrong!” to a tone more like “Here's a way to think about it, and the consequences thereof.” After all, no method is the best for all situations and cases. Our method is best for situations where the individual visit spectra are barely sampled or under-sampled.

who owns a research project?

My day ended today with a great conversation about the ownership of research projects with a postdoc. When you make the transition from graduate student to postdoc, whose projects are whose? Are they the projects of your supervisors, or are they the projects of you? And should you keep doing them, or should you move to new things? I don't think there are easy answers, and I think that there are many subtle ways in which people have unresolved differences about these things. Since much of my work these days is postdoctoral mentoring, I've thought about this a lot. My only recommendation, which is hard to implement, is that clear communication about expectations is really, really important. And not just the expectations of the supervisors; the expectations of the (former) student are way more important!

information theory for spectroscopy

I had a meeting this morning with Megan Bedell (Flatiron) about our dormant paper about information theory and extreme-precision radial-velocity measurements. We see the paper a bit differently (is it about methods or is it about concepts?), but we were able to re-state a scope with which we are both happy. We assigned tasks (Bedell writing and me coding, mainly), and promised to make progress before next week. It is very, very, very hard to finish a paper! Especially when all authors are above some seniority, where they spend most of their time with others. I would love to get a lot more personal coding time!

lost

I got really lost with respect to research today. In almost all of my projects I am supposed to be mentoring postdocs and students. Today various blocks came up that interfered with that mentoring. And then I found that I had nothing sensible to work on! Of course that isn't true: I have literally a dozen projects in a mature state waiting on final work from me. But I couldn't figure out how to work on any of them. Research is hard. At the end of the day, Andy Casey (Monash) helped me out by giving me some very specific jobs to do.

regressions for point clouds

I spent my research time today writing in a document that proposes (and demonstrates) some methods for performing machine-learning-style regressions, but where the input objects (features) are variable-size point clouds. Contributions also from Villar (JHU) and Gebhard (MPI-IS). I spent way too long working out the terminology and notation, and I am still wrong.

high-order integration schemes

I was working on a white paper on ocean dynamics today and I threw in a sentence about how emulators (like machine-learning replacements for simulations) might be working because they might be effectively learning a high-order integration method. I then threw in a sentence about how, in many applications, high-order integrators are known to be better than low-order integrators. I then went to find a reference and... well, I am not sure I can back that up with a reference! I thought this was common knowledge, but it looks like almost all simulations and integrations are done with low-order integrators. Am I living in a simulation? (A simulation integrated with wimpy first-order integrators?)

Phi-M radio

I worked today with Abby Shaum (CUNY) on her paper about her phase-demodulator to find exoplanet and substellar companions to stars by the timing of asteroseismic modes. I suggested that we highlight the incredible simplicity of her project by writing the method as an algorithm of just a few lines.