APOGEE spectra as a training set

I spent a lot of the day building a training set for a machine-learning problem set. I am building the training set out of the SDSS-V APOGEE spectra, which are like one-dimensional images for training CNNs and other kinds of deep learning tasks. I wanted relatively raw data, so I spent a lot of time going deep in the SDSS-V data model and data directories, which are beautiful. I learned a lot, and I created a public data set. I chose stars in a temperature and log-gravity range in which I think the APOGEE pipelines work well and the learning problem should work. I didn't clean the data, because I am hoping that contemporary deep learning methods should be able to find and deal with outliers and data issues. If you want to look at my training set (or do my problem set), start here.

Galactic cartography

Neige Frankel (CITA) and I discussed measurements of the age and metallicity gradients in the Milky Way today. In my machine-learning world, I am working on biases that come in when you use the outputs of regressions (label transfer) to perform population inferences (like mean age as a function of actions or radius). We are gearing up to do a fake but end-to-end simulation of how the Milky Way gets observed, to see if the observed Galaxy looks anything like (what we know in this fake world to be) the truth.

predicting spectra from spectra

Saakshi More (NYUAD) came into my office during office hours today to ask about possible data science projects in physics. I pitched to her predicting ESA Gaia RVS spectra from Gaia XP spectra, and vice versa. Has anyone done that? In one direction, you have to predict high resolution detail from low-resolution input; in the other direction, you have to predict a wide wavelength range from narrow input. It seems like perfect for something like a linear auto-encoder (at least for a small patch of the color–magnitude diagram; non-linear for a large patch). Later in the day I talked to Gaby Contardo and she said: If you want to go simple, how about nearest neighbor? Good idea!

Florida, day two

Today was day two of my visit to University of Florida. I had many interesting discussions. One highlight was with Dhruv Zimmerman, who wants to infer big labels (non-parametric functions of time) from small features (a few bands of photometry). That's my kind of problem! We discussed different approaches, and we discussed possible featurizations (or dimensionality reductions) of the labels. I also pitched an information-theoretic analysis. If there's one thing I've learned in the last few years, it is that you shouldn't be afraid to solve problems where there are fewer data than parameters! You just have to structure the problem with eyes wide open.

After many more (equally interesting) discussions, the day ended with Sarah Ballard's group out at a lovely beer garden. We discussed the question: Should students be involved in, and privy to, all the bad things with which we faculty interact as academics, or should we protect students from the bad things? You can imagine my position, since I am all about transparency. But the positions were interesting. Ballard pointed out that in an advisor–student relationship, the student might not feel that they can refuse when the advisor wants to unload their feelings! That power asymmetry is very real. But Ballard's students (Chance, Guerrero, Lam, Seagear) said that they want to understand the bad things too; they aren't in graduate school just to write papers (that comment is for you, Quadry!).

machine-learning theory and practice

Today I got invited to be on a panel discussion (hosted by Soledad Villar of JHU) with Alberto Bietti (Flatiron) about the theory and practice of machine learning. It was great! We talked about why ML works for scientific applications, and Bietti said something (obvious maybe) that I loved: Maybe ML only works because of properties of the data. That is, maybe when we are analyzing ML methods we are looking in the wrong place, and we should be analyizing the data to which they are successfully applied? I made fun of interpretation in ML, and that led to interesting comments from both Bietti and the audience. Several audience members suggested taking something more like a causal approach to interpretation: How does the method work under interventions or in conditional situations? That's interesting; it isn't what a physicist would consider interpetation, but it might be sufficient in many cases.

uncertainty estimation for regression outputs

Most methods for performing regressions don't provide natural uncertainties. Some do, of course! But few deliver uncertainties you will believe. I discussed these issues with Contardo (SISSA) today, in the context of our project to (confidently) find infrared excesses around boring old main-sequence stars. One option is to look at the performance on held-out data. But then you have to decide how to aggregate this information in a way that is relevant for each object in your sample: They probably don't all have the same uncertainty! Another option is to look at the variation of prediction across training sets. That's good! But it requires that you have lots of training data. In this case, we do, so that's where we are at right now.

teeny tiny cosmological simulations.

Connor Hainje (NYU) is looking at this paper by Chen et al which uses a machine-learning regression to interpolate between cosmological simulation outputs at different cosmological epochs. To build an end-to-end pipeline for testing ideas, he has been running 32-cubed cosmological simulations. These might be the smallest simulations run since the 1980s! But, interestingly, he is finding that the interpolation isn't working great. Is this because it is harder to train a regression on a small simulation than it is on a large simulation? Is a small simulation less predictable or less interpolate-able? It's expensive to find out!

CMB component separation with linear fitting

Today I sat down with Fiona McCarthy (Flatiron) to look at data-driven methods for separating cosmic microwave background data into different components. We implemented a simple polynomial regression to fit foregrounds, using (observed) difference maps as inputs (features) that are designed to contain foregrounds only. We obtained some preliminary results that looked exciting but we’ve only just started. Part of the motivation is that CNNs are hard to train, but linear combinations of image monomials are easy! I realized in all this that there are connections to the group-equivariant stuff I’ve done with Villar’s group, because we use invariants, and also to the causal inference things that Schölkopf’s group does, because we’re trying to impose some causal structure on our functions.

infrared-excess stars

My day started with a call with Gaby Contardo (SISSA) and Trevor David (Flatiron) about Contardo's project to find stars in ESA Gaia and NASA WISE that have infrared excesses. These stars should be young or dust-enveloped or host aliens! We are trying to phrase this problem as a prediction problem: How well can we predict infrared brightness from Gaia (visible) information, and are there stars with significant excess infrared? The answer seems to be yes: The histogram of differences between predicted and observed skews nicely to infrared-excess. Now: Are any of these known objects? And can we rediscover (say) star-forming regions?

quasars and stellar density

Kate Storey-Fisher (NYU) has made really nice random catalogs that look very very similar, in sky coordinates, to the quasars we have. However, there is obviously more exclusion of quasars from the Galactic plane region than we can explain with any reasonable model of how dust is affecting things. It's the stellar density of course: ESA Gaia selection is very sensitive to stellar density, especially (as now) when you are using the XP spectra. Today she included the stellar density in the random-catalog regression and boom: Excellent random! Our model is not mechanistic, it is effective and data-driven.

local linear regressions

For some reason, even though I dislike deep learning, I love local linear regressions. My friends tell me that RELU networks are locally linear, so I am really just a hypocrite. Anyway, today Adrian Price-Whelan and I built a regression in which we find nearest neighbors (among the training-set objects) in the space of ESA Gaia DR3 Bp/Rp spectral coefficients and, among those neighbors, we fit a locally linear model to predict the parallax of the test object. Technically we use a clever trick called the ”schmag“ but which should probably be called the reduced parallax, in which we correct the parallax into the inverse of the square root of the luminosity. Why? It's so we can use the parallax errors fairly, and include training-set objects with negative parallaxes.

Hill I will die on: If you cut your sample to high SNR parallaxes or positive parallaxes, you will bias any regressions you do to predict parallaxes or distances or distance moduli!

discovering quantum physics, automatically?

I have been working on making machine-learning methods dimensionless (in the sense of units). In this context, a question arises: Is it possible to learn that there is a missing dimensional input to a physics problem, using machine learning? Soledad Villar (JHU) and I ignored some of our required work today and wrote some code to explore this problem, using as a toy example the Planck Law example we explained in this paper. We found that maybe you can discover a missing dimensional constant? We have lots more to do to decide what we really have.

making Fourier fitting super fast

At the request of Conor Sayres (UW), I have been looking at distortion patterns in the SDSS-V Focal Viewing Camera (FVC), which is the part of the system that looks at whether the focal-plane fiber robots are where they need to be. The distortions are extremely bad; they are large in amplitude and vary on extremely small scales on the focal plane. So I have to fit an extremely flexible model. Here are my comments:

First, you should use mixtures of sines and cosines for problems like this. Not polynomials! Why? Because sines and cosines do not blow up at the edges.

Second, you should punk fast Fourier transform (FFT) codes to speed up your regressions. I wrote code to do this, which wraps the finufft best-in-class non-uniform FFT code in scipy.sparse linear-algebra code. This wrapping makes the FFT operators into linear-algebra operators and permits me to do solve() operations. That move (wrapping FFT in linear algebra) sped up my code by factors of many!

wacky shape scalars

Kate Storey-Fisher (NYU) showed me today the results of her work predicting stellar contents of dark-matter halos in hydrodynamic n-body simulations. She is building her shape scalars from geometric properties (scalars, vectors, and tensors) of the position-space and velocity-space distributions of the n-body particles. She did a very principled feature-importance study, including one feature at a time, combinatorically, and seeing how each feature helps, differentially. The most important features are... strange! Why? Because most of the regression work is done by very simple features (halo mass, halo size, halo velocity) so the (dimensionless) shape scalars we have made are fixing up non-trivial problems. Time to write the paper!

GPRV, day 1

The GPRV meeting started in Oxford today. The meeting brings together people working on data analysis in extreme precision radial-velocity projects, but united by interests in and uses of Gaussian processes. The first day ended with a very nice tutorial by Foreman-Mackey (Flatiron) on applied-math and computational tools for scalable Gaussian processes. He even live-coded and blew everyone's mind with Python jax.

Many talks (including Barragán (Oxford), Delisle (Geneva), and Tran (UT Austin) to name a few) are using Gaussian processes and their derivatives or two Gaussian processes to model the star's variability, with photometry, radial-velocity measurements, and activity indicators modeled as linear combinations of these latent processes. That's a really interesting theme, and connects somehow to my evil plan (with Bedell, Luger, Zhao, et al) of modeling the whole stellar surface. It is definitely an exciting time.

One issue that came up is how to judge or assess over-fitting. There was no consensus or answer, and most of the GP practitioners are very Bayesian. But Bayesian approaches aren't always sensitive to true statistical violations of the model; I want to see some cross-validation in this house.

In other news, Halverson (JPL) told us about publicly available solar data (and lots of it) from NASA NEID. I might want to play with that when I get home!

the black-body law

Today Miles Cranmer (Princeton) chatted with Weichi Yao (NYU), Soledad Villar (JHU), and me about things related to symbolic regression, dimensional analysis, and so on. He brought up a very interesting problem in the history of physics: The black-body radiation law, which is attributed to Planck. Planck knew about temperature, wavelength, the speed of light, and Boltzmann's constant k. Dimensionally, these can be combined into only one thing that has units of intensity, and that one thing is the long-wavelength black-body law. At short wavelengths, the behavior can't be explained without the introduction of a new constant, and that constant has to have non-trivial dimensions (units). He figured it out, and that constant ended up governing the hydrogen atom spectrum, quantum mechanics, and everything else. Indeed that constant, h, bears his hame. Could we have learned this ourselves just from the data directly with a machine? After all, that's what Planck did, right?

are data-driven approaches to RV measurement biased?

Matt Daunt (NYU) is re-building our wobble code for measuring stellar radial velocities without any stellar or tellurics model. He is finding that it is slightly biased towards smaller radial velocity amplitudes than what we inject into fake data. This also mirrors things that Bedell (Flatiron) and I have seen in various experiments. I think there is something going on with spectral edges: At the edge of the observed spectral domain, some observations have lines shifted into and out of the observations. The mean spectrum obtained from those measurements isn't necessarily capturing all of this fairly. Or at least it has to be handled carefully. Are we doing this right? Experiments we have suggest that we don't have this quite right yet.

how can linear regression be hard?

Maybe I'm known in astronomy for being both a machine-learning developer and a machine-learning skeptic. I hope so! Anyway, I love linear regression, because it has a lot of the power of bigger ML models, but it's easy to implement and to understand. And yet!

Today Kate Storey-Fisher (NYU) and I looked at her code to predict galaxy properties given dark-matter-halo properties in a set of n-body simulations. We are doing very simple regressions but the condition numbers of the matrices are blowing up and some of our answers don't look great. And this is generic: Many linear-regression models are messed up by condition numbers and numerical linear algebra, and it is hard to diagnose, and it is hard to treat. And if linear regression is hard—and hard for us—why do I believe anything that inovolves 42 layers of fully-connected RELU network?

feature engineering for dark-matter halos

Kate Storey-Fisher and I spoke about adding eigenvalues (scalars) and eigenvectors (vectors) to our geometric features of dark-matter halos. For regression! But the problem with this is that the eigenvectors have ambiguous sign; is that an issue? Yes! They are descriptions of an order-2 tensor, not order-1 vectors. Hmmm. We also spoke about whether the feature engineering should be deterministic, or chosen by hand.

dimensional scalings improve predictions

As my loyal reader knows, I have been working on the possibility that machine-learning or regression methods could be sensitive to dimensions or units and thus get better generalization and so forth. Today we had a success! Weichi Yao (NYU) has been converting the model in this paper on geometric methods to a model that respects dimensional scalings (or dimensional symmetry, or units equivariance). It hasn't been doing better than non-dimemsional-symmetric versions, in part (we think) because the dimensionless invariants we produce have worse conditioning properties than the raw labels. But Soledad Villar (JHU) had a good idea: Let's test on data that are outside the distribution used to generate the training set! That worked beautifully today, and we have far better predictions for the out-of-sample test points than the competitive methods. Why? Because the dimensional scalings guarantee certain aspects of generalization.