Showing posts with label noise. Show all posts
Showing posts with label noise. Show all posts

2024-03-14

IAIFI Symposium, day one

Today was the first day of a two-day symposium on the impact of Generative AI in physics. It is hosted by IAIFI and A3D3, two interdisciplinary and inter-institutional entities working on things related to machine learning. I really enjoyed the content today. One example was Anna Scaife (Manchester) telling us that all the different methods they have used for uncertainty quantification in astronomy-meets-ML contexts give different and inconsistent answers. It is very hard to know your uncertainty when you are doing ML. Another example was Simon Batzner (DeepMind) explaining that equivariant methods were absolutely required for the materials-design projects at DeepMind, and that introducing the equivariance absolutely did not bork optimization (as many believe it will). Those materials-design projects have been ridiculously successful. He said the amusing thing “Machine learning is IID, science is OOD”. I couldn't agree more. In a panel at the end of the day I learned that learned ML controllers now beat hand-built controllers in some robotics applications. That's interesting and surprising.

2024-03-10

APOGEE spectra as a training set

I spent a lot of the day building a training set for a machine-learning problem set. I am building the training set out of the SDSS-V APOGEE spectra, which are like one-dimensional images for training CNNs and other kinds of deep learning tasks. I wanted relatively raw data, so I spent a lot of time going deep in the SDSS-V data model and data directories, which are beautiful. I learned a lot, and I created a public data set. I chose stars in a temperature and log-gravity range in which I think the APOGEE pipelines work well and the learning problem should work. I didn't clean the data, because I am hoping that contemporary deep learning methods should be able to find and deal with outliers and data issues. If you want to look at my training set (or do my problem set), start here.

2023-09-20

uncertainty estimation for regression outputs

Most methods for performing regressions don't provide natural uncertainties. Some do, of course! But few deliver uncertainties you will believe. I discussed these issues with Contardo (SISSA) today, in the context of our project to (confidently) find infrared excesses around boring old main-sequence stars. One option is to look at the performance on held-out data. But then you have to decide how to aggregate this information in a way that is relevant for each object in your sample: They probably don't all have the same uncertainty! Another option is to look at the variation of prediction across training sets. That's good! But it requires that you have lots of training data. In this case, we do, so that's where we are at right now.

2023-09-06

is a periodic signal in a time series statistically significant?

I had conversations with Nora Eisner (Flatiron) and Abby Shaum (CUNY) today about how we report the significance of a signal we find in a time series. In particular a periodic signal. It's an old, unsolved problem, with a lot of literature. And various hacks that are popular in the exoplanet community (and binary-star community!). My position is very simple: Since all methods for determining significance are flawed, and since when you fit a signal you have to estimate also an uncertainty on that signal's parameters, the simplest and most basic test of significance is the significance with which you measure the amplitude of the proposed signal. That is, if the amplitude is well measured, the signal is real. Of course there are adversarial data sets I can make where this isn't true! But that's just a restatement of the point that this is an unsolved problem. For deep reasons!

2023-07-26

elusive quasar dipole

There should be an imprint of the kinematic dipole observed in the cosmic microwave background in any cosmological tracer: The dipole is set by the Solar System barycentric velocity relative to the local Hubble flow, and that same velocity should imprint a dipole on anything cosmological. I have been working on this in part because it is a good measurement to make with Quaia, and in part because the dipole in the quasars is controversial. I have many thoughts, but I will save them for later.

Anyways, Abby Williams (NYU) has been working on making this measurement, and her dipole amplitude and direction depend on what we hold fixed and what we vary (in particular selection-function components), and they also depend on what sky region we use. None of this is surprising; the selection function has a strong dipole in it, and it is not known precisely. But then I don't understand how the studies published previously have such good error bars. Maybe they didn't consider the various different fitting regimes?

2023-02-08

guarantees on diffusion

Today at JHU I went to the joint group meeting of Jeremias Sulam (JHU) and Soledad Villar (JHU) in which Jacopo Teneggi (JHU) showed guarantees on correctness for some diffusion-based de-noising schemes for images. The quarantees are a bit weak, because it is hard to put a hard boundary on coverage in image space! But essential for medical applications (the illustrated domain). I have to say, it was nice to see a rigorous approach to errors from machine-learning methods. I think this is necessary for our uses in cosmology, where we are expecting machine-learning emulators to be as accurate as simulations!?!

2022-12-06

new LIGO events

Today Mathias Zaldarriaga (IAS) gave the NYU Astro Seminar. He told us about work he has been doing to increase the sensitivity of the LIGO data to inspiral events, and how that is impacting beliefs about populations of black holes.

2022-09-28

can a (say) 5-sigma result be interpreted as a p-value?

In preparing for class today (I am teaching NYU #data4physics), I worked through the relationship between a p-value (like what's used in medical research) and a physicist's n-sigma measurement. They are related in some very special cases, like in particular when the value being measured is a linear parameter (like an amplitude) and the noise is Gaussian. But those cases are special. And also: Converting n-sigma to p-value depends very critically on the noise model. So I don't like thinking of it as a p-value. That said, maybe there is no difference?

2022-08-18

continuum normalization of spectra

How do you continuum-normalize a spectrum in a low-resolution spectrograph? You can't really, unless you have exceedingly good models for stars (which you could use to fit the normalization); but of course if you had extremely good models for stars you probably wouldn't have to normalize!

I had an interesting conversation about this with Alex Dimoff (MPIA). He is continuum-normalizing spectra in a high-resolution echelle spectrograph. He has a good measurement of the “blaze function” so I suggested that he just add some polynomial or sine-and-cosine adjustments to that in each order. My main advice about continuum normalization is to avoid things that are very sensitive to signal-to-noise: As you degrade the signal-to-noise, your continuum estimate should not systematically deviate low. Most methods in play right now have this problem, and bad. Think: Fit to pixels that are “consisten” with being continuum pixels. That's going to depend very strongly on signal-to-noise.

2022-08-11

jackknife practice and intuitions

Last week I gave a colloquium at MPIA in which I advocated the use of jackknife and bootstrap resampling to obtain empirical uncertainty estimates in a complex data analysis. Today I actually implemented jackknife in my project on cosmic homogeneity (and isotropy). I jackknifed by sky position: I split the sky into 12 nearly-equal regions for 12-fold leave-one-out. I have intuitions about when it is a good idea to jackknife on a quantity (like sky position) and when it is a good idea to jackknife on a quantity that is completely random, but I don't know exactly where my intuition comes from. In general it must be the case that jackknifing on different things answers different questions about your noise.

2022-07-31

jackknife the sky?

On the weekend, Kate Storey-Fisher (NYU) and I implemented jackknife uncertainty estimation for KSF's cosmology-with-Gaia-quasars project. We jackknifed by cutting the sky into RA slices. This is standard practice but I don't love it! It assumes that you know that your main source of error is calibration or sample consistency over the sky. It might be something way more insidious. In principle I guess you should jackknife over many things, and also randomly.

2022-07-29

uncertainty propagation for a neural network

Today Matthias Samland (Stockholm) gave a nice Königstuhl Colloquium at MPIA about direct imaging of exoplanets with high-contrast imaging. He showed some beautiful results from ESO Gravity and from NASA JWST. One of his main take-away points is that the situation is changing fast, and we might achieve very much higher contrast ratios in the near future than we've ever had, and thus get many more planets.

I spent some time late in the day looking at uncertainty propagation for neural networks: Given that you can optimize a NN, and given that it makes good predictions for held-out data, and given that you can take all derivatives of everything with respect to everything, does that mean you can propagate errors or noise from the data to the results? I think the answer is yes in a limited sense: You can see how the output depends on the input at the training step. But what you can't do—and probably will never be able to do—is propagate the uncertainties that come from your training set (the uncertainties in your weights, as it were). And these uncertainties can be very large, especially since the models tend to be enormously over-parameterized, and also contain combinatorially large exact and near-exact degeneracies. (I think maybe the near-exact degeneracies are worse than the exact ones.) I vaguely recall Tom Charnock making strong statements about all these things at Ringberg.

2022-07-17

latent-variable model

On the weekend Adrian Price-Whelan and I decided that we have to implement the linear latent-variable model (our name!), which is a generative model for both features and labels, if we are going to do well labeling stars with ESA Gaia XP spectra at low signal-to-noise (faint magnitudes). The reason is: The latent-variable model generates the data, so it deals naturally with the different noise in stars of different brightnesses (or different signal-to-noise). We think this is important! We'll see this coming week.

2022-02-19

how to compare radial-velocity spectrographs?

Yesterday, in Data Group Meeting at Flatiron, Lily Zhao (Flatiron) showed a formula for the effectiveness of a spectrograph for measuring a radial velocity that she was confused about. It appeared in a paper by someone who builds spectrographs. We discussed it and ended up a bit confused. So today I sat down and wrote a short document, which represents my Official Position (tm) on spectrograph radial-velocity precision. My loyal reader will not be surprised to learn that I think the right way to compare is by looking at radial-velocity information per unit time where “information” is inverse variance (Fisher information). In that context, the answer is that you get more information linearly with wavelength coverage and you get more information linearly with spectrograph resolution. It's that simple!

[Note added later: This post is WRONG. See here.]

2022-01-21

more data is always better, right?

In Astronomical Data Group Meeting at Flatiron today, Lily Zhao (Flatiron) asked a great question about how to combine radial-velocity data: Imagine you have a very high-resolution echelle-type spectrograph with many spectral orders. You get a stellar radial-velocity out of every order (or, in some cases, out of many small spectral patches in every order. Now: How to combine those radial-velocity measurements into one, true radial velocity measurement? Obviously you just do an inverse-variance weighted average, no? Well, no! It doesn't work right. There are bad orders, and the only way to know is to see that they make the measurements worse in some end-to-end variance sense. So what to do? How do you empirically determine how to do the combination of data? This problem is simultaneously trivial and impossible. It's a great subject of discussion, and one I've mentioned here previously.

2021-08-09

what are the assumptions underlying EPRV?

I reopened an old paper started (many years ago now) by Megan Bedell (Flatiron) and me, about the precision possible in extreme precision radial-velocity spectroscopy. Most of the results in the literature on how precisely you can measure a radial velocity (the information-theoretic question) depend on a very large number of assumptions, which we try to enumerate. The thing we'd like to do with this paper (and frankly, it will take many more after it) is to weaken or break those assumptions and see what gives. I have an intuition that if we understand all of that information theory, it will help us with observation planning and data analysis.

2021-08-02

scalings for different methods of inference

Excellent long check-in meeting with Micah Oeur (Merced) and Juan Guerra (Yale) about their summer-school projects with me and Adrian Price-Whelan (Flatiron). The projects are all performing inferences on toy data sets, where we have fake observations of a very simple dynamical system and we try to infer the parameters of that dynamical system. We are using virial theorem, Jeans modeling, Schwarzschild modeling, full forward modeling of the kinematics, and orbital torus imaging. We have results for many of these methods already (go team!) and more to come. Today we discussed the problem of measuring scalings of the inferences (the uncertainties, say) as a function of the number of observations and the quality of the data. Do they all scale the same? We also want to check sensitivity to selection effects, and wrongnesses of the assumptions.

2021-06-22

how to simulate a spectrum

I had a great conversation today with Matt Daunt (NYU), building on discussion yesterday with also Megan Bedell (Flatiron), about how to simulate data from an extreme-precision radial-velocity spectrograph. We decided to simulate the star, the atmosphere, and the (gasp!) gas cell all at very high resolution, then combine them physically, then reduce resolution to the spectrograph resolution (which is very high nonetheless) and then sample and noisify the resulting data. The idea is: Make the structure of the code like the structure of our physical beliefs, or causal beliefs. We decided to fork this data simulation into its own project.

2021-06-10

information theory at Cambridge

Today I gave a colloquium at the University of Cambridge. My slides are here. I spoke about how to make precise measurements, how to design surveys, and how to exploit structure in noise. It's a rich set of things, and most of the writing about information theory in astronomy is only in the cosmology domain. Time to change that, maybe? It is also the case that the best book about information and inference ever written was written in Cambridge! So I was bringing coals to Newcastle, ish!

2021-06-04

objective functions and Nyquist sampling

Adrian Price-Whelan and I discussed today some oddities that Matt Daunt (NYU) is finding while trying to measure radial velocities in extremely noisy, fast APOGEE sub-exposures. He finds that the objective function we are using is not obviously smooth on 10-ish km/s velocity scales. Why not? We don't know. But what we do know is that a spectrograph with resolution 22,500 cannot put sharp structures into a likelihood function on scales smaller than about 13 km/s.

There's a nice paradox here, in fact: The spectrograph can't see features on scales smaller than 13 km/s, and yet we can reliably measure radial velocities much better than this! How? The informal answer is that the radial-velocity precision is 13 km/s divided by a certain, particular signal-to-noise. The formal answer involves information theory—the Fisher information, to be precise.