partial differential equations

I am trying to write a proposal to fund the research I do on machine-learning theory. The proposal is to work on ocean dynamics. It's a great application for the things we have done! But it's hard to write a credible proposal in an area that's new to you. Interdisciplinarity and agility is not rewarded in the funding system at present! At least I am learning a ton as I write this.

conjectures about pre-training

On Monday of this week, Shirley Ho (Flatiron) gave a talk at NYU in which she mentioned the unreasonable effectiveness of pre-training a neural network: If, before you train your network on your real (expensive, small) training data, you train it on a lot of (cheap, approximate) pre-training data, you get better overall performance. Why? Ho discussed this in the context of PDE emulation: She pre-trains with cheap PDEs and then trains on expensive PDEs and she gets way better performance than she does if she just trains on the expsensive stuff.

Why does this work? One interesting observation is that even pre-training on cat videos helps with the final training! Ho's belief is that the pre-training gets the network understanding time continuity and other smoothness kinds of things. My conjecture is that the pre-training teaches the network about (approximate) diffeomorphism invariance (coordinate freedom). The cool thing is that these conjectures could be tested with interventions!

data augmentation

A highlight of my day was a colloquium by Renée Hložek (Toronto) about cosmology and event detection with the LSST/Rubin. Importantly (from my perspective), she has run a set of challenges for classifying transients, based on simulations of the output of the very very loud LSST event-detection systems. The results are a bit depressing, I think (sorry Renée!), because (as she emphasized), all the successful methods (and none were exceedingly successful) made heavy use of data augmentation: They noisified things, artificially redshifted things, dropped data points from things, and so on. That's a good idea, but it shows that machine-learning methods at the present day can't easily (or ever?) be told what to expect as an event redshifts or gets fainter or happens on a different night. I'd love to fix those problems. You can almost think of all of these things as group operations. They are groups acting in a latent space though, not in the data space. Hard problems! But worthwhile.

linear regression

Valentina Tardugno (NYU) and I are looking at the NASA TESS housekeeping data: What parts of it are relevant to understanding the light curves? The weird thing is: We are asking this by asking: What housekeeping data can be reliably predicted using the light curves? Why this way? Because the light curves are higher in signal-to-noise (in general) than most channels of the housekeeping data. Today we went through all the relevant linear algebra for big linear models (which is where we are starting, of course!).

unitary evolution of the Universe

I spent the day with Juna Kollmeier (CITA) talking about epistemology, physical cosmology, and project management (especially academic management). I found myself saying to her the following argument (which I have not seen written down anywhere): Imagine that our Universe is hamiltonian (or lagrangian; it doesn't matter for these purposes). And imagine that our Universe is a simulation being run inside some bigger universe, which is also hamiltonian.

If our Universe is being observed in any sense by any system in that bigger universe, then there ought to be a loss of unitarity in our Universe. That is, there should be a violation of Liouville's theorem, or a violation of key conservation laws, or an information sink. And there is! At black hole horizons, there is an information paradox: Information that goes in never comes back (an evaporating black hole evaporates thermally, or so we think). Thoughts?

area of a triangle?

On Friday and the weekend, I came up with (what I think is) a novel formula for the area A of a triangle! That's weird. I was looking for a formula in the Deep Sets (or map-reduce) format. Here it is. It's ridiculous and useless, but it involves only sums over functions of the individual corners of the triangle. It was hard to find! But it's exact (I believe).

symmetry day: crossing, permutation

Today's brown-bag talk, by Grant Remmen (NYU), was about (in part) crossing symmetry. This is the symmetry that any Feynman diagram can be rotated through 90 degrees (converting time into space and vice versa) and the interaction will have the same scattering amplitude. This symmetry relates electron–positron annihilation to electron–electron scattering. The symmetry has an important role in string theory, because it is a constraint on any possible fundamental theory. This symmetry has always seemed incredible to me, but it is rarely discussed outside very theoretical circles.

After the talk, and in the Blanton–Hogg group meeting, I brought up things about invariant functions that I learned from Soledad Villar (JHU) that are really confusing me: It is possible (in principle, maybe not in practice) to write any permutation-invariant function of N objects as a function of a sum of universal functions of the N objects (that's proven). How does that relate to k-point functions? Most physicists believe that any k-point function estimate will require a sum over all N-choose-k k-tuples. That's a huge sum, way bigger than a sum over N. What gives? I puzzled some of the mathematical physicists with this and I remain confused.

Florida, day one

I spent today with Sarah Ballard's group, plus others, at the University of Florida. I gave a talk, to a large, lively, and delightful audience. At the end of this talk I was very impressed by the following thing: Ballard had everyone in the room discuss with their neighbors (turn and talk) for about 3 minutes, after the seminar but before the question period began! This is a technique I use in class sometimes; it increases participation. After those 3 minutes, audience members had myriad questions, as one might imagine.

I spoke with many people in the Department about their projects. One highlight was Jason Dittman, who showed me gorgeous evidence that a particular warm exoplanet on an eccentric orbit has an atmosphere that undergoes some kind of phase change at some critical insolation, as it moves away from its host star on its orbit. Crazy!

Late in the day I discussed n-point functions and other cosmological statistics with Zach Slepian and Jiamin Hou. We discussed the plausibility of getting tractable likelihoods for any n-point functions. We also discussed the oddity that n-point functions involve sums over n-star configurations among N stars (N choose n), but there are mathematical results that show that any permutation-invariant function of any point cloud can be expressed with only a sum over stars (N). That sounds like a research problem!

representation of flexible functions

Emily Griffith (Colorado) and I met today to look at replacing a spline interpolation function deep inside some of our code with a Fourier series. The idea is that we need a flexible function of one variable, and we were using a spline of a set of control points, but (for many reasons) we wanted to change to a sum of sines and cosines. The code work was a mess! The small change hits a lot of places inside our model, which is our K-process data-driven nucleosynthetic model. This same problem appears in the new version of wobble by Matt Daunt (NYU). I love flexible functions, but it's hard to implement them in a properly abstracted way. That is, it is hard to write a model so that you can just swap in a Fourier series or a Gaussian process where you used to have an interpolation of control points.

is the world lagrangian?

My day started with a long and very fun conversation with Monica Pate (NYU) about conservation laws in classical physics. As we all know, conservation laws are related to symmetries; each symmetry of the laws of physics creates a conservation law. Or does it? Well, it's a theorem! But it's a theorem when the laws of physics are lagrangian (or hamiltonian). That is, every symmetry of a hamiltonian system is associated with a conservation law in that system. So I asked: How do we know if or whether the world is lagrangian or hamiltonian? How could we know that? My best guess is that we know it because of these very conservation laws! The situation is complex.

regressions for point clouds

I spent my research time today writing in a document that proposes (and demonstrates) some methods for performing machine-learning-style regressions, but where the input objects (features) are variable-size point clouds. Contributions also from Villar (JHU) and Gebhard (MPI-IS). I spent way too long working out the terminology and notation, and I am still wrong.

gradients of unit vectors

When you work in a curvilinear coordinate system, and you need to take gradients or tensor derivatives of scalar, vector, and tensor functions, the gradients of the unit vectors appear in your expressions. The unit vectors have gradients because, in a curvilinear coordinate system, they have orientations that depend on position. I gestured and imagined and guessed these derivatives for a spherical coordinate system by thinking geometrically. I got strange expressions I didn't believe. Then, today, I checked them by painstakingly taking derivatives, and my intuitive derivatives turned out to be exactly correct?

making linear algebra faster in practice

The key thing to make your code run faster is to avoid building large linear-algebra objects. For example, if you need to get the matrix product A.x, where A is a huge matrix and x is a long vector, and you only ever use the matrix A to do this one multiply by x, there is no reason to actually create A. Just create a function that evaluates A.x for any input x. That should be way faster, because of less memory allocation, and because you don't have to make the parts of the matrix that are all zeros (for example). Matt Daunt (NYU) and I discussed all this at length today, as we profiled code. This comment has some overlap with Section 9 of this paper.

high-order integration schemes

I was working on a white paper on ocean dynamics today and I threw in a sentence about how emulators (like machine-learning replacements for simulations) might be working because they might be effectively learning a high-order integration method. I then threw in a sentence about how, in many applications, high-order integrators are known to be better than low-order integrators. I then went to find a reference and... well, I am not sure I can back that up with a reference! I thought this was common knowledge, but it looks like almost all simulations and integrations are done with low-order integrators. Am I living in a simulation? (A simulation integrated with wimpy first-order integrators?)

CZS Summer School, day 5: diffusion

Diffusion models are all the rage in machine learning these days. Today Laurence Levasseur (Montréal) gave a beautiful talk at the CZS Summer School about how diffusion works. She started with a long physics introduction, which was great, and also insightful, about how diffusion works in small physical systems. Then she showed how it can be turned into a method for sampling very difficult probability distributions.

I have a history of working on MCMC methods. These permit you to sample a posterior pdf when you only know a function f that is related to your posterior pdf by some unknown normalization constant. Similarly, diffusion lets you sample from a pdf when you only know the gradient of f. Again, you don't need the normalization. That makes me wonder: Should we be using diffusion in places where we currently use MCMC? I bet the answer is yes, for at least some problems.

CZS Summer School, day 4: GNNs

Today Andreea Deac (Montréal) gave a talk at the CZS Summer School about graph neural networks, and enforcing exact symmetries. It was a great talk, because it was useful to the students and filled with insights even for the experienced machine-learners. She did a great job of connecting GNNs to other methods in use in ML, including convolutional neural networks, and Deep Sets.

CZS Summer School, day 3: Deep Sets

Today was day 3 of the CZS Summer School, in which I am helping mentor a group of students working on equivariant methods on point clouds. In our working session today, Soledad Villar (JHU) (who is the main mentor for this group of students) gave a short, spontaneous explanation of the main Deep Sets result in machine learning: (Almost) any permutation-invariant function of a set of objects x_i can be written in an amazingly simple form: h(Σ_ig(x_i)), where h and g are potentially nonlinear functions. That result is super-strong, and super-useful!

spherical harmonics for tensor fields

I have been kicking around the generalization of spherical harmonics to vector spherical harmonics, and how that might generate the tensor spherical harmonics to all orders of tensor and all parities. I think I got it today! For every spherical harmonic (ell and em), there are three vector spherical harmonics obtained by multiplying by the radial vector, taking the transverse gradient, and taking the transverse gradient and crossing it into the radial direction. I think these can be generalized (using, say, the Ricci calculus) to make the 2-tensors and so on. If I am right, this is a new way to represent tensor fields on the sphere. Use cases: Cosmic backgrounds, and ocean dynamics.

Dr Kate Storey-Fisher

Kate Storey-Fisher (NYU) defended her PhD here at NYU today. She killed it! She talked about emulating cosmological simulations (at the level of statistics, not maps), making invariant scalars that encode the shapes and dynamics of dark-matter halos, and her awesome 1.2 million all-sky quasar catalog from ESA Gaia and NASA WISE. It was all things my loyal reader knows lots about but I loved it. It has been an honor and a privilege to work with KSF these years, and I will miss her very very much.

distances between point clouds

I spent the last two days working at Apple Paris, which was fun! I worked with the open-source ott-jax package, which can do some amazing things. I worked with Soledad Villar (Apple & JHU) to generalize the k-means algorithm to point clouds! It can cluster point clouds morphologically, even if the different point clouds have different numbers of points, and even if the different point clouds live in spaces of different dimensions! Everything obeys permutation and rotation symmetries.