what book am I going to write?

One possible new year's resolution this year is for me to decide which book am I going to write? I don't love this, because it is the hallmark of a scientist at the end of the career that they switch to writing books! I guess maybe I'm at the end of my career? But that said, I have (maybe like many scientists at the end of their careers?) a lot to say. Okay anyway, I had a long conversation this morning with Greg McDonald (Rum&Code) about all this, and he strongly encouraged me to make some content for the project code-named ”The Practice of Astrophysics“.

radical papers I want to write (or will never write)

I have to finish my NSF proposal with Mike Blanton (NYU), so naturally I am in procrastination mode. Here are three papers I wish I would write. Maybe I should post them on my ideas blog:

Occam's Razor is wrong: This paper, co-authored with Jennifer Hill (NYU), would be about the fact that, in the real, observed world, the simplest explanation is always wrong or at least incomplete.

Causation is just causality: This paper, maybe co-authored with David Blei (Columbia) or Bernhard Schölkopf (MPI-IS) or Hill, shows that you don't need to have free will in order to have cogent causal explanations of data. That is, you don't need to phrase causality in terms of predictions for counter-factual experiments that you might have chosen to do.

You don't ever want evidence: This paper shows that any time you are computing the Bayesian evidence—what I call the fully marginalized likelihood (fml)—you are doing the wrong integral and solving the wrong problem. For both practical and theoretical (principled) reasons.

data augmentation

A highlight of my day was a colloquium by Renée Hložek (Toronto) about cosmology and event detection with the LSST/Rubin. Importantly (from my perspective), she has run a set of challenges for classifying transients, based on simulations of the output of the very very loud LSST event-detection systems. The results are a bit depressing, I think (sorry Renée!), because (as she emphasized), all the successful methods (and none were exceedingly successful) made heavy use of data augmentation: They noisified things, artificially redshifted things, dropped data points from things, and so on. That's a good idea, but it shows that machine-learning methods at the present day can't easily (or ever?) be told what to expect as an event redshifts or gets fainter or happens on a different night. I'd love to fix those problems. You can almost think of all of these things as group operations. They are groups acting in a latent space though, not in the data space. Hard problems! But worthwhile.

ARC BOG meeting

I stayed on at Cloudcroft after the SDSS-V Advisory Council meeting for the ARC Board of Governors meeting, which is the meeting of the organization that runs the Apache Point Observatory. I spent a lot of the meeting learning about the 3.5m and the site, which was interesting, and which made me think about how we apportion our resources in astronomy. These are huge facilities, run very lean (money wise), and they produce a lot of science. The SDSS family of projects has had simply immense scientific impact.

One success of the meeting: I have successfully coined and propagated the term SDSS Classic to mean SDSS-I and SDSS-II. Multiple people at the meeting now use this terminology!

who owns a research project?

My day ended today with a great conversation about the ownership of research projects with a postdoc. When you make the transition from graduate student to postdoc, whose projects are whose? Are they the projects of your supervisors, or are they the projects of you? And should you keep doing them, or should you move to new things? I don't think there are easy answers, and I think that there are many subtle ways in which people have unresolved differences about these things. Since much of my work these days is postdoctoral mentoring, I've thought about this a lot. My only recommendation, which is hard to implement, is that clear communication about expectations is really, really important. And not just the expectations of the supervisors; the expectations of the (former) student are way more important!

biases from machine learning

Today I gave a talk (with these slides) at a meeting in Denver for the NSF initiative Harnessing the Data Revolution. I spoke about the necessity and also the dangers of using machine-learning methods in scientific projects. I brought up two very serious possible biases. The first is that if emulators are used to replace simulations, and they can't be easily checked (because the simulation requirements are too expensive), the emulators will lead to a confirmation-bias problem: We will only carefully check the emulations if they lead to results that we don't like! The second bias I raised is that if we perform joint analyses on objects (stars, say) that have been labeled (with ages, say) by a machine-learning regression, there will in general be strong biases in those joint analyses. For example, the average value of 1000 age labels for stars labeled by a standard ML regression will not be anything like an unbiased estimate of the true average age of those stars. These biases are very strong and bad! That said, I also gave many example locations where using machine learning methods is not just okay but actually intellectually correct, in areas of instrument calibration, foregrounds, and other confounders.

The question period was great! We had 25 minutes of questions and answers, which ranged across a very wide set of topics, including statistics, experimental design, and epistemology.

Bayesian evidence?

Kate Storey-Fisher, Abby Williams, and I spent some time discussing unpublished work that relies heavily on calculations of the Bayesian evidence. Bayesian evidence—what I call the “fully marginalized likelihood”—relates to the volume of the posterior in parameter space. It is generally extremely sensitive to the width of the prior pdf, since if you are comparing two models with different parameterizations, the numbers you get depend on how you normalize or scale out the units of those parameter-space volumes. Indeed, you can get any evidence ratios you want by tuning prior pdf widths. That's bad if you are trying to conclude something, scientifically! Bayesian inference is only principled, imho, when you can quantitatively state the prior pdf that correctly describes your beliefs, prior to seeing the new data. And even then, your evidence is special to you; any other scientist has to recompute from scratch.

planning your science

I had two interactions today that made me think seriously about big-picture and design things. I like design language: How do you design your whole research program, and how do you design individual projects so they fit into it. One interaction was in the Astronomical Data Meeting at Flatiron, where Vivi Acquaviva (CUNY) talked about the intersection between what you are good at, what is important, and what brings you joy. That's a hard intersection to find. Or way too easy; I am not sure. The other interaction was a conversation with Jiayin Dong (Flatiron), who is thinking about faculty job applications and the like. How to talk about your research in terms of the next decade instead of the next year?

One comment that is frequently made by Hans-Walter Rix (MPIA) is that he feels like most early-career (and even mid-career) people spend too much time doing their science and not enough time planning and justifying their science. It is important to be able to answer “why” questions about your research, and in the medium term it helps all your projects.

classification to save labor

I spent part of the day discussing with Valentina Tardugno (NYU) and Nora Eisner (Flatiron) the goals of a machine-learning classification that Tardugno is creating to help the PlanetFinders project. The deal is: Citizen scientists find candidate planets and (currently) a human (Eisner) has to vet them, to remove contamination by various sources of false positives. This turns out to be a hard problem! When problems are hard, it becomes critical to very precisely specify what you are trying to achieve. So we spent time discussing what, exactly, it is that Eisner needs from a classifier. Is it to find good planets? Is it to remove obvious contaminants? Are some contaminants more problematic than others? Is it to save her hours of wall-clock time? Etc.

other kinds of machine learning

Astronomy is very focused on machine learning in the sense of regression and classification, but machine learning can do many other things. In addition, machine learning is a sub-field of machine intelligence, which is broader. I started today working on a proposal for the NSF (to be written with Mike Blanton, NYU) in which we propose using other kinds of machine learning and machine intelligence, and apply them earlier in the scientific process (like at operations and calibration) instead of at the end (like at source classification and labeling).

can a (say) 5-sigma result be interpreted as a p-value?

In preparing for class today (I am teaching NYU #data4physics), I worked through the relationship between a p-value (like what's used in medical research) and a physicist's n-sigma measurement. They are related in some very special cases, like in particular when the value being measured is a linear parameter (like an amplitude) and the noise is Gaussian. But those cases are special. And also: Converting n-sigma to p-value depends very critically on the noise model. So I don't like thinking of it as a p-value. That said, maybe there is no difference?

Dagstuhl, day 3

Today was day 3 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl.

We had an open discussion about goals for ML in science today. The idea of explainability came up. I liked the comment that explainability (or what counts as explainability) might depend incredibly strongly on field or context. Like it is different in medicine and in astronomy. And, related, the idea of how models are communicated is very context dependent. And maybe very dependent on history. For example, in the future, models might be communicated through APIs rather than scientific papers maybe?

Causation and causal inference was a big theme of the day with Bernhard Schölkopf, Jonas Peters, Bubacar Bah, and Niki Kilbertus all talking about overlapping ideas in causal inference, mechanism inference, differential equation inference, and symbolic regression. Is causation the new framework for machine learning? Many in the room think so.

latent-variable model

On the weekend Adrian Price-Whelan and I decided that we have to implement the linear latent-variable model (our name!), which is a generative model for both features and labels, if we are going to do well labeling stars with ESA Gaia XP spectra at low signal-to-noise (faint magnitudes). The reason is: The latent-variable model generates the data, so it deals naturally with the different noise in stars of different brightnesses (or different signal-to-noise). We think this is important! We'll see this coming week.

GPRV, day 4

Today was day 4 and the last day of GPRV in Oxford. The day ended with a discussion led by Heather Cegla (Warwick) and Jennifer Burt (JPL) about EPRV and national priorities. Exoplanet science is obviously extremely important to the 2021 Decadal Survey, but in detail, the first seven chapters of that Survey (the chapters to which NASA and NSF must respond) do not actually mention radial velocity! The conversation in the room today was extremely wide-ranging; it covered hardware, software, science goals, and community-building goals. But it also covered months, years, and decade-long time-scales.

The highest level recommendation of the Decadal Survey was that we need to do preparatory work to design and assess feasibility of a large IR–visible–UV telescope that will discover habitable worlds. There is no doubt (I think it's uncontroversial) that this preparatory work will require lots and lots of EPRV science and observations. Of course the fact that this is obvious is separated somewhat from the question of whether there will be abundant funding!

It will come as no surprise to my loyal reader that I was a proponent, in this discussion, of building open-science communities around open data, open-source software, and open science collaborations. I think we have so much evidence now that open-science communities science way better. What I loved is that there were absolutely no objections in the room to this idea. The only controversies were about exactly how open data should be managed and released in that utopian future. I'm optimistic about this business!

And I thank Suzanne Aigrain (Oxford) and her OC for a great meeting!

GPRV, day 2

Today was day two of GPRV. It was a delight! Here's another highly unfair summary of the day:

Hara (Geneva) kicked it off with a discussion of a Bayesian-decision-theory-like method for deciding on the reality and correctness of exoplanet discoveries. He made clever choices to deliver really strong probabilistic results. I was about to object to all this but then he disarmed me at the end of the talk by noting that everything is extremely sensitive to noise models and that is the biggest issue. He gave some chilling examples.

Shahaf (Tel Aviv) showed some very nice results in old-school statistics that generalize the periodogram to a correlation between phase differences and distances between pairs of any quantities you like, as a function of period. This can be used to perform causal inferences for periods seen in naive periodograms. He uses a very interesting phase variable for these phase differences; this is extremely relevant to things I have been discussing with Zhao and Bedell.

Mortier (Cambridge) mesmerized the room with her work with six (yes 6) years of solar data from HARPS-N (maybe). She can show amazing relationships between pipeline RVs, activity indicators, and spectral shape measures. But she showed that often the correlations are not at zero-time-lag. Often the correlations are strongest with delays of 1/9 to 1/8 of a rotation period. When she sub-samples the data to typical kinds of long-term monitoring campaigns we are going to do on distant stars, it is a bit scary. That led to a lot of discussion over lunch and dinner.

Zhao (Flatiron), Bucchave (DTU), and Dumusque (Geneva) led discussions on community-building, hardware, instrument calibration, and other things. The meeting is set up for lots of discussion and is itself an extremely good example of a community-building activity, around the hard challenges of EPRV. I opined in one of these sessions that EPRV now looks like cosmology around 2000, when everything was just about to go open and the world community started working together. This meeting is part of this change that we want to see.

Finally, a theme of the day was representations for spectral signals. Dumusque (inadvertently, I think) made a strong case that we should be working in the 2-d spectrograph images directly! That's music to my heart. He also emphasized that the stellar surface is a complex physical place. I agree! And Cretignier (Geneva) showed a beautiful representation of the spectral residuals to disentangle Doppler and spectral-variability variations. I think his work and Shahaf's could be combined in interesting ways; I am excited to get back to the lab.

six-quark state?

Today was a great blackboard talk at CCPP by Glennys Farrar (NYU) about a possible six-quark state in QCD. She has been thinking about this for a decade or so, because it might have implications for dark matter and issues in QCD. Today she focused on the latter: There are terms in the g−2 calculation for the muon that can be estimated either with lattice QCD or by integrating some observed branching ratios from experiment. These two methods disagree, and the observational method disagrees (more strongly) with the g−2 measurement. But Farrar shows that if there is a long-lived 6-quark state, it can potentially affect the QCD calculation (implicitly) but would be evaded by the branching-ratio measurements (because it would evade all event triggers). Her model requires some good luck with QCD parameters and bound states, but if that luck holds, she can pull dark matter into the standard model and solve some precision-measurement issues! After her talk we discussed a bit about just how hard lattice QCD is. It's absurd!

what are the assumptions underlying EPRV?

I reopened an old paper started (many years ago now) by Megan Bedell (Flatiron) and me, about the precision possible in extreme precision radial-velocity spectroscopy. Most of the results in the literature on how precisely you can measure a radial velocity (the information-theoretic question) depend on a very large number of assumptions, which we try to enumerate. The thing we'd like to do with this paper (and frankly, it will take many more after it) is to weaken or break those assumptions and see what gives. I have an intuition that if we understand all of that information theory, it will help us with observation planning and data analysis.

scalings for different methods of inference

Excellent long check-in meeting with Micah Oeur (Merced) and Juan Guerra (Yale) about their summer-school projects with me and Adrian Price-Whelan (Flatiron). The projects are all performing inferences on toy data sets, where we have fake observations of a very simple dynamical system and we try to infer the parameters of that dynamical system. We are using virial theorem, Jeans modeling, Schwarzschild modeling, full forward modeling of the kinematics, and orbital torus imaging. We have results for many of these methods already (go team!) and more to come. Today we discussed the problem of measuring scalings of the inferences (the uncertainties, say) as a function of the number of observations and the quality of the data. Do they all scale the same? We also want to check sensitivity to selection effects, and wrongnesses of the assumptions.

how much calibration data does a spectrograph need?

There was an SDSS-V operational telecon today, in which we discussed the plans for the first year of data, and how those plans should depend on, or be conditional on, what we learn in the commissioning phase. One of the most important things about SDSS-V is that it is robotic and fiber-fed, so we can move fast and do time domain things. But how fast? This depends, in turn, on how much calibration data we need as a function of time. I proposed that we could ask how well we can synthesize calibration data in one telescope configuration at one time, given calibration data from other telescope configurations at other times. This is not unlike the approach we took to calibrating EXPRES. So of course it was put back to me: Design the experiments that will answer this question! The main issue is that the BOSS spectrogaphs hang off the back of the pointing, tracking telescope.

SDSS-V MWM target selection

Today Jennifer Johnson (OSU) crashed the weekly Manhattan-area SDSS-V discussion meeting to give us the current state of Milky Way Mapper (a component of SDSS-V) target selection. It was a great discussion, because there are many, many target categories, and many of them are interesting to Manhattan-area locals. For me the most impressive thing about the meeting was that Johnson could answer almost any question from anyone on any of the literally dozens of target categories! It was a tour de force as they say. And we learned a lot. One of my goals with this meeting (which was started and is operated by Katie Breivik, Flatiron) is to increase excitement in Manhattan for SDSS-V and Johnson did that admirably.