number of exposures per visit?

I wrote code this weekend to look at the question of how we should visit a star in the upcoming Terra Hunting Experiment. The current (straw-person) plan is that we will observe each visible star once per night for ten years, with one exposure of a sensibly-chosen exposure time at each visit. Is this a good idea? I was interested in this problem for two reasons. The first is that binning is sinning, with the corollary that bigger bins are worse than finer bins, and a single, long exposure is a very big bin. The second reason is that when there are non-trivial noise sources (like the quasi-periodic variations from p-mode oscillations of the surfaces of Sun-like stars), a few negatively- (or interestingly-) correlated noise draws can be combined in ways that are substantially more informative than by taking the average.

Of course, if you split an exposure (with a standard CCD, say) into sub-exposures, you take on real costs: There is a read time, which is time you aren't integrating, and there is a read noise, that affects each new exposure. So the best strategies are a complicated function of read time, read noise, and the signal-to-noise at which the stellar p-mode oscillations are visible in any realistic data. Related: There are amazingly different and interesting strategies with up-the-ramp detectors that are used in the infrared.

One final comment is that the objective, in my strongly held view, is to optimize the amount of information (about, say, the center-of-mass radial-velocity changes of the target star) per unit wall-clock time. We are paying for wall-clock time; let's get as much as we can out of it.

radical papers I want to write (or will never write)

I have to finish my NSF proposal with Mike Blanton (NYU), so naturally I am in procrastination mode. Here are three papers I wish I would write. Maybe I should post them on my ideas blog:

Occam's Razor is wrong: This paper, co-authored with Jennifer Hill (NYU), would be about the fact that, in the real, observed world, the simplest explanation is always wrong or at least incomplete.

Causation is just causality: This paper, maybe co-authored with David Blei (Columbia) or Bernhard Schölkopf (MPI-IS) or Hill, shows that you don't need to have free will in order to have cogent causal explanations of data. That is, you don't need to phrase causality in terms of predictions for counter-factual experiments that you might have chosen to do.

You don't ever want evidence: This paper shows that any time you are computing the Bayesian evidence—what I call the fully marginalized likelihood (fml)—you are doing the wrong integral and solving the wrong problem. For both practical and theoretical (principled) reasons.

ARC BOG meeting

I stayed on at Cloudcroft after the SDSS-V Advisory Council meeting for the ARC Board of Governors meeting, which is the meeting of the organization that runs the Apache Point Observatory. I spent a lot of the meeting learning about the 3.5m and the site, which was interesting, and which made me think about how we apportion our resources in astronomy. These are huge facilities, run very lean (money wise), and they produce a lot of science. The SDSS family of projects has had simply immense scientific impact.

One success of the meeting: I have successfully coined and propagated the term SDSS Classic to mean SDSS-I and SDSS-II. Multiple people at the meeting now use this terminology!

Bayesian evidence?

Kate Storey-Fisher, Abby Williams, and I spent some time discussing unpublished work that relies heavily on calculations of the Bayesian evidence. Bayesian evidence—what I call the “fully marginalized likelihood”—relates to the volume of the posterior in parameter space. It is generally extremely sensitive to the width of the prior pdf, since if you are comparing two models with different parameterizations, the numbers you get depend on how you normalize or scale out the units of those parameter-space volumes. Indeed, you can get any evidence ratios you want by tuning prior pdf widths. That's bad if you are trying to conclude something, scientifically! Bayesian inference is only principled, imho, when you can quantitatively state the prior pdf that correctly describes your beliefs, prior to seeing the new data. And even then, your evidence is special to you; any other scientist has to recompute from scratch.

planning your science

I had two interactions today that made me think seriously about big-picture and design things. I like design language: How do you design your whole research program, and how do you design individual projects so they fit into it. One interaction was in the Astronomical Data Meeting at Flatiron, where Vivi Acquaviva (CUNY) talked about the intersection between what you are good at, what is important, and what brings you joy. That's a hard intersection to find. Or way too easy; I am not sure. The other interaction was a conversation with Jiayin Dong (Flatiron), who is thinking about faculty job applications and the like. How to talk about your research in terms of the next decade instead of the next year?

One comment that is frequently made by Hans-Walter Rix (MPIA) is that he feels like most early-career (and even mid-career) people spend too much time doing their science and not enough time planning and justifying their science. It is important to be able to answer “why” questions about your research, and in the medium term it helps all your projects.

classification to save labor

I spent part of the day discussing with Valentina Tardugno (NYU) and Nora Eisner (Flatiron) the goals of a machine-learning classification that Tardugno is creating to help the PlanetFinders project. The deal is: Citizen scientists find candidate planets and (currently) a human (Eisner) has to vet them, to remove contamination by various sources of false positives. This turns out to be a hard problem! When problems are hard, it becomes critical to very precisely specify what you are trying to achieve. So we spent time discussing what, exactly, it is that Eisner needs from a classifier. Is it to find good planets? Is it to remove obvious contaminants? Are some contaminants more problematic than others? Is it to save her hours of wall-clock time? Etc.

other kinds of machine learning

Astronomy is very focused on machine learning in the sense of regression and classification, but machine learning can do many other things. In addition, machine learning is a sub-field of machine intelligence, which is broader. I started today working on a proposal for the NSF (to be written with Mike Blanton, NYU) in which we propose using other kinds of machine learning and machine intelligence, and apply them earlier in the scientific process (like at operations and calibration) instead of at the end (like at source classification and labeling).

Dagstuhl, day 4

Today was day 4 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl.

I spoke today, about passive symmetries and the constraints on machine-learning models they imply. My talk was totally new for me, and based on conversations between Villar, Schölkopf, and me during the meeting. That was fun. So now I have a new way of talking about all this stuff, and the three of us are trying to write a short paper about it.

Among the talks today, one idea I really liked is the idea, from Carl Henrik Ek, that trust and interpretability might be strongly related. Indeed, when I talk about interpretability, it is often in the same context that I am talking about models that make sense to a physicist, which are, in turn, models that I would trust. And that is also very related to what I myself talked about today: If models look more like physical law, then they are much more trustworthy. And maybe also more interptetable.

Dagstuhl, day 3

Today was day 3 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl.

We had an open discussion about goals for ML in science today. The idea of explainability came up. I liked the comment that explainability (or what counts as explainability) might depend incredibly strongly on field or context. Like it is different in medicine and in astronomy. And, related, the idea of how models are communicated is very context dependent. And maybe very dependent on history. For example, in the future, models might be communicated through APIs rather than scientific papers maybe?

Causation and causal inference was a big theme of the day with Bernhard Schölkopf, Jonas Peters, Bubacar Bah, and Niki Kilbertus all talking about overlapping ideas in causal inference, mechanism inference, differential equation inference, and symbolic regression. Is causation the new framework for machine learning? Many in the room think so.

GPRV, day 4

Today was day 4 and the last day of GPRV in Oxford. The day ended with a discussion led by Heather Cegla (Warwick) and Jennifer Burt (JPL) about EPRV and national priorities. Exoplanet science is obviously extremely important to the 2021 Decadal Survey, but in detail, the first seven chapters of that Survey (the chapters to which NASA and NSF must respond) do not actually mention radial velocity! The conversation in the room today was extremely wide-ranging; it covered hardware, software, science goals, and community-building goals. But it also covered months, years, and decade-long time-scales.

The highest level recommendation of the Decadal Survey was that we need to do preparatory work to design and assess feasibility of a large IR–visible–UV telescope that will discover habitable worlds. There is no doubt (I think it's uncontroversial) that this preparatory work will require lots and lots of EPRV science and observations. Of course the fact that this is obvious is separated somewhat from the question of whether there will be abundant funding!

It will come as no surprise to my loyal reader that I was a proponent, in this discussion, of building open-science communities around open data, open-source software, and open science collaborations. I think we have so much evidence now that open-science communities science way better. What I loved is that there were absolutely no objections in the room to this idea. The only controversies were about exactly how open data should be managed and released in that utopian future. I'm optimistic about this business!

And I thank Suzanne Aigrain (Oxford) and her OC for a great meeting!

sailing upwind; lift ratios; sail-to-keel ratio

In writing up a description of some of my physics-of-sailing results, I realized some things about sailboat design. For deep reasons you want the ratio of the area of the sail to the area of the keel to be roughly the ratio of the density of water to the density of air, or 700. That flows from the point that the sail and the keel have symmetric roles in sailing. But if you set this ratio to 700, you can only sail upwind if the sail lift ratio (the ratio of useful sail force to drag force) is very high. Since it is hard to make this lift ratio high on a commercial boat, the alternative is to make the sail smaller (relative to the keel). Looking at the data I can find on real sailboats, most have okay lift ratios but small sail-to-keel ratios (smaller than 700 anyway), so that they can sail quickly upwind. The cost of these design choices is that you can't go downwind faster than the wind. If you want to be able to sail both downwind faster than the wind—and also upwind—you have to have amazing sail and keel lift ratios.

what are the assumptions underlying EPRV?

I reopened an old paper started (many years ago now) by Megan Bedell (Flatiron) and me, about the precision possible in extreme precision radial-velocity spectroscopy. Most of the results in the literature on how precisely you can measure a radial velocity (the information-theoretic question) depend on a very large number of assumptions, which we try to enumerate. The thing we'd like to do with this paper (and frankly, it will take many more after it) is to weaken or break those assumptions and see what gives. I have an intuition that if we understand all of that information theory, it will help us with observation planning and data analysis.

SDSS-V MWM target selection

Today Jennifer Johnson (OSU) crashed the weekly Manhattan-area SDSS-V discussion meeting to give us the current state of Milky Way Mapper (a component of SDSS-V) target selection. It was a great discussion, because there are many, many target categories, and many of them are interesting to Manhattan-area locals. For me the most impressive thing about the meeting was that Johnson could answer almost any question from anyone on any of the literally dozens of target categories! It was a tour de force as they say. And we learned a lot. One of my goals with this meeting (which was started and is operated by Katie Breivik, Flatiron) is to increase excitement in Manhattan for SDSS-V and Johnson did that admirably.

a model for sailing (yes, sailing)

I've had a lifetime of conversations with Hans-Walter Rix (MPIA) about the point that you could in principle sail with a sailboat with flat sails: Nothing about the curvature of the sails is integral or required by sailing. The curvature helps, but isn't necessary. I have had another lifetime of conversations with Matt Kleban (NYU) about the point that sailing depends on the relative velocity between the air and the water, and this leads to some hilarious physics problems involving sailing on rivers in zero wind (it's possible because a flowing river is moving relative to the dead air).

These worlds collided this weekend because—inspired by a twitter conversation—I finally built a proper ram-pressure model of a flat-sail, flat-keel sailboat and got it all working. It's sweet! It sails beautifully. Much more to say, but question is: Is there a paper to write?

machine learning at #AAS238

Today I spoke at the “meeting-in-meeting” on machine learning at the summer AAS meeting. My slides are here. I started out a bit negative but I ended up saying very positive things about what machine learning can do for astrophysics. I got as much feedback on the twitters afterwards (maybe more) than I did in real time. Several of the other speakers in my session mentioned or discussed contrastive learning, which looks like it might be an interesting unsupervised technique.

Dr Lily Zhao

I had the great honor to be on the PhD committee of Lily Zhao (Yale), who defended her dissertation today. It was great and remarkable. She has worked on hardware, calibration, software, stellar astrophysics, and planets. Her seminar was wide-ranging, and the number and scope of the questions she fielded was legion. She has already had a big impact on extreme precision radial-velocity projects, and she is poised to have even more impact in the future. One of the underlying ideas of her work is that EPRV projects are integrated hardware–software systems. This idea should inform everything we do, going forward. I asked a million technical questions, but I also asked questions about the search for life, and the astronomical community's management and interoperation of its large supply of diverse spectrographs. In typical Zhao fashion, she had interesting things to say about all these things.

adversarial attacks and robustness

Today Teresa Huang (JHU) re-started our conversations about adversarial attacks against popular machine-learning methods in astrophysics. We started this project (ages ago, now) thinking about test-time attacks: You have a trained model, how does it fail you at test time? But since then, we have learned a huge amount about training-time attacks: If you add a tiny change to your training data, can you make a huge change to your model? I think some machine-learning methods popular in astronomy are going to be very susceptible to both kinds of attacks!

When we discussed these ideas in the before times, one of the objections was that adversarial attacks are artificial and meaningless. I don't agree: If a model can be easily attacked, it is not robust. If you get a strange and interesting result in a scientific investigation when you are using such a model, how do you know you didn't just get accidentally pwned by your noise draw? Since—in the natural sciences—we are trying to learn how the world works, we can't be putting in model components or pipeline components that are capable of leading us very seriously astray.

how accurately can we do closed-loop robot control?

Conor Sayres (UW) and I spoke again today about the fiber positioning system (fiber robots) that lives in the two focal planes of the two telescopes that take data for SDSS-V. One of the many things we talked about is how precisely do we need to position the fibers, and how accurately will we be able to observe their positions in real time. It's interestingly marginal; the accuracy with which the focal-plane viewing system (literally a camera in the telescope that looks the wrong way) will be able to locate the fiber positions depends on details that we don't yet know about the camera, the optics, the fiducial-fiber illumination system, and so on. There are different kinds of sensible procedures for observatory operations that depend very strongly on the accuracy of the focal-viewing system.

best setting of hyper-parameters

Adrian Price-Whelan (Flatiron) and I encountered an interesting conceptual point today in our distance estimation project: When you are doing cross-validation to set your hyper-parameters (a regularization strength in this case), what do you use as your validation scalar? That is, what are you optimizing? We started by naively optimizing the cost function, which is something like a weighted L2 of the residual and an L2 of the parameters. But then we switched from the cost function to just the data part (not the regularization part) of the cost function, and everything changed! The point is duh, actually, when you think about it from a Bayesian perspective: You want to improve the likelihood not the posterior pdf. That's another nice point for my non-existent paper on the difference between a likelihood and a posterior pdf. It also shows that, in general, the data and the regularization will be at odds.

why make a dust map? and Bayesian model elaboration

Lauren Anderson (Carnegie) and I had a wide-ranging conversation today. But part of it was about the dust map: We have a project with statisticians to deliver a three-dimensional dust map, using a very large Gaussian-process model. Right now the interesting parts of the project are around model checking and model elaboration: How do you take a model and decide what's wrong with it, in detail. Meaning: Not compare it to other models (that's a solved problem, in principle), but rather, compare it to the data and see where it would benefit from improvement.

One key idea for model elaboration is to check the parts of the model you care about and see if those aspects are working well. David Blei (Columbia) told us to climb a mountain and think on this matter, so we did, today. We decided that our most important goals are (1) to deliver accurate extinction values to stellar targets, for our users, and (2) to find interesting dust structures (like spiral arms) if they are there in the data.

Now the challenge is to convert these considerations into posterior predictive checks that are informative about model assumptions. The challenge is that, in a real-data Bayesian inference, you don't know the truth! You just have your data and your model.