Dagstuhl, day 1

Today was day 1 of Machine Learning for Science: Bridging Data-driven and Mechanistic Modeling at Schloss Dagstuhl. The first day was mainly about applications of machine learning, in Earth science, livestock management, astrophysics (dark matter), cells, and mechanical engineering. I had many thoughts and realizations. Here are a few random ones:

The problems that appear in Earth science, and the data types, are very similar to those that appear in astrophysics! But in Earth science, biology is a big driver of global processes, and there is no good mechanistic model for (say) how plants grow and take up carbon. The world is filled with mobile phones, with good cameras, and the methods we could could be employing to be doing science in a distributed way are way, way under-used. Cells are incredibly complicated. The mechanistic model involves literally thousands of individual processes. Like our model for the cell is as complicated as our model for the entire Earth system (which, by the way, depends on cells!), or even more complicated.

In the areas of the cell and the Earth, a theme was that the investigators want to preserve the causal structure we believe, and just use the machine learning to replace one tiny piece, with a data-driven model. Related: You can think of the machine learning as an effective theory for something (a sub-part of the problem) that doesn't work well from first principles. That's a good idea!

coordinate freedom vs equivariance, again

With Soledad Villar (JHU) and others I have been discussing making generalizations (or restrictions?) of image convolution operators to make machine learning respect more symmetries. One kind of generalization is going to 3-d images, and another is making the weights in the convolution filters geometric objects, like vectors, pseudovectors, and tensors. Then we developed a group-averaging technique to make these geometric filters equivariant. And now we are considering products and contractions of these geometric objects to make universally approximating function spaces. I don't love the word “equivariant” here: In my view the symmetries are coordinate freedoms, not relations between inputs and outputs. But the machine-learning world has spoken.

why does deep learning work?

There is a paradox about deep learning. Which everyone either finds incredibly unconvincing or totally paradoxical. I'm not sure which! But it is this: It is simultaneously the case that deep learning is so flexible it can fit any data, including randomly generated data, and the case that when it is trained on real data, it generalizes well to new examples. I spent some time today discussing this with Soledad Villar (NYU) because I would like us to understand this a bit better in the context of possible astronomical applications of deep learning.

In many applications, people don't need to know why a method works; they just need to know that it does. But in our scientific applications, where we want to use the deep-learning model to de-noise or average over data, we actually need to understand in what contexts it is capturing the structure in the data and not just over-fitting the noise. Villar and I discussed how we might test these things, and what kinds of experiments might be illuminating. As my loyal reader might expect, I am interested in taking an information-theoretic attitude to the problem.

One relevant thing that Villar mentioned is that there is research that suggests that when the data has simpler structure, the models train faster. That's interesting, because it might be that somehow the deep models still have some internal sense of parsimony that is saving them; that could resolve the paradox. Or not!

translation, geometry, deep learning, fast GPs

Friday-morning parallel-working session was brief today. I talked to Shiloh Pitt (NYU) about verifying matrix identities using numerical methods. And then we went downstairs for a mini-workshop at NYU CCPP organized by Kyle Cranmer (NYU) and Glennys Farrar (NYU) about physics and data science.

Cranmer led it off with an informal discussion of the different language used by statistics and computer science and applied math and physics. There are lots of words used differently, or that trigger different things. He mentioned “bias” and “correlation” and the uses or meanings of graphs and flowcharts. During the talks more words came up. One subtle one is that data scientists think of a data record as a point in data space (so, say, an image is a point in image space). That isn't always natural for physicists.

Joan Bruna (NYU) gave a nice talk about the geometric properties of deep learning, keying off of the success of convolutional neural networks. He said many interesting and insightful things, but here are a few that stuck with me: The convolutional symmetry at small scales in image space aids the NN in finding a distance metric (or something like that) between images that respects symmetries or structure that is really there. And it does that tractably, or in reasonable time. He claims that any compact symmetry group can be incorporated: That is, he claims that deep learning models can be made to exactly respect any symmetry that has certain properties. That's very exciting for physical applications. Distances between nodes on a graph also represent a geometry; it can be extremely different from geometry on simple manifolds! But the same ideas apply: If there are symmetries, they can be respected by the deep learning algorithms.

Life intervened! But by the end of the day, I made it to Flatiron to see a talk by Dan Foreman-Mackey (Flatiron) about data science, interdisciplinarity, open science, and finding planets around other stars. He gave a lot of credit to his interdisciplinary collaborations. He also mentioned the kinds of translation issues that Cranmer opened with at NYU. On the technical side, he showed his Gaussian-process methods and code and the near-linear scaling that they deliver. As I like to say: If you are doing linear algebra faster than N-squared (and he is, by far) then you can't even represent your matrices. That is, building the matrix itself is already N-squared. After his talk the Flatiron applied mathematicians were in heated arguments about exactly why (in a math sense) his methods are so fast. Foreman-Mackey's code is making possible things in astrophysics that have never been possible before.

talks all day; Dr Gorbenko

Roberto Sanchis-Ojeda (Berkeley) arrived for a few days of hacking. He is responsible for finding some very short-period transiting exoplanets. We didn't get a chance to set our goals for the day, because the day was packed with talks:

Victor Gorbenko (NYU) gave an absolutely excellent PhD defense. Congratulations Dr Gorbenko. Among other things, he was considering the difference between the theoretical (analytic) action of massless modes on the 1+1 worldsheet of a string (not a 0+1 wordline but s 1+1 worldsheet) and a lattice calculation. In this comparison, he found massive modes. He very nicely connected this work to larger questions in string theory, but also to the origins of string theory in QCD.

At lunch, Guido D'Amico (NYU) spoke about axions, axion–photon interactions, and cosmic transparency. This is related to work I did with Bovy and with More in the optical. He advertised some clever ideas he has about about using the Planck data to constrain this sector using millimeter wavelengths. He has been implementing some of these ideas at #astrohackny.

At the end of the day, Neil Lawrence (Sheffield) spoke about deep machine-learning models built from nested Gaussian Processes. The methods show a lot of promise and connect well to things we are thinking about in CampHogg.

cosmology, now with less approximation; RNADE

In the morning, Mike O'Neil (NYU), Iain Murray (Edinburgh), Malz, Vakili, and I met for coffee to talk cosmology and cosmological inference. We discussed the linear algebra of making full (non-approximate) likelihood calls for cosmic background inference, which includes determinant and solve calls for enormous (dense) matrices. A few days ago, Wandelt made the (perhaps obscure) comment to me that he did his Gibbs sampling in the CMB to avoid making determinant calls. I finally understood this: The determinant is big and expensive in part because it is like an integral or marginalization over the nuisance parameters (which are the initial conditions or phases). If you can compute the determinant, you get something you can think of as being a marginalized likelihood function. The Gibbs sampling is part of a system that does posterior sampling, so it does the marginalization but doesn't return the amplitude of the likelihood function. If you need a marginalized likelihood function, you can't do it with the Gibbs sampling. Murray once again made the simple point that the posterior (for the cosmological parameters) will in general be much easier to compute and represent than the likelihood function (the probability of the data given the cosmological parameters) because the former is a much smaller-dimensional thing. That's a deep point!

That deep point played also into his talk in the afternoon about his RNADE method (and code) to use deep learning to represent very complex densities, or estimate a density given a sampling. One of his application areas is a project to obtain posterior constraints on the mass of the Milky Way given (as data) the properties of the Magellanic Clouds. The theory in this case is represented by a large number of numerical simulations, but Murray wants a continuous density to do inference. RNADE is really impressive technology.

stellar modes, reading minds

At group meeting today Angus talked about her attempts to reproduce the asteroseismological measurements in the literature from Kepler short-cadence data. I think there is something missing, because we don't observe all the modes as clearly as they do. Our real goal is not just to reproduce the results of course; we discussed our advantages over existing methods: We have a more realistic generative model of the data; we can do multiple frequencies simultaneously; we can handle not just non-uniform time sampling but also non-uniform exposure times (which matters for high frequencies), and we can take in extremely non-trivial noise models (including ones that do detrending). I am sure we have a project and a paper, but we don't understand our best scope yet.

Just before lunch, Kyunghyun Cho (Montreal) gave a Computer Science colloquium about deep learning and translation. His system is structured as an encoder, a "meaning representation", and then a decoder. All three components are interesting, but the representation in the middle is a model system for holding semantic or linguistic structure. Very interesting! He has good performance. But the most interesting things in his talk were about other kinds of problems that can be cast as machine translation: Translating images into captions, for example, or translating brain images into sentences that the subject is currently reading! Cho's implication was that mind reading is just around the corner...

products of deep networks, SKA

I spent the day (which is a Monday, but I am blogging it on a Sunday, because of international date-line insanity) with Marcus Frean (University of Victoria, Wellington) and Brewer (who came down to Wellington from Auckland). We spent the day arguing about what we should try to accomplish this week. When Frean and I last spoke, it was about probabilistic models for astronomical imaging, where he was trying to find subtle sources (like low surface-brightness galaxies) in radio imaging from the SKA pathfinders. Today we spent much of the day arguing about things we might try to accomplish this week, which ranged from tiny tweaks of things we have already done, or running existing software on new data, all the way to building a probabilistic generative model of all astronomical data ever (my usual plan!).

In the afternoon, we talked about deep networks and why they are powerful and yet tractable to train. I still don't fully understand, but I am closer. But then we discussed Frean's observation that most data sets are generated by multiple causes, so you really want not just a deep network, but a graphical model that has two (or more) independent deep networks pointing at the data. That is, we want products of deep networks, if we really want to represent the structure of the data in the world. We stared at that for a while; it appears to maybe require the probabilistic generalization of deep networks (about which Frean has no fear). He has some example code that operates the product of deep belief nets on toy problems!

At lunch, we were joined by Melanie Johnston-Hollitt (Victoria), who is deeply involved in New Zealand's (proposed, I think) buy-in to the SKA project. We discussed radio astronomy data analysis and the management and operation of global astronomical projects, including Sloan and SKA. We discussed New Zealand's role in the SKA; I am very pleased to see New Zealand in the SKA; this could be huge for New Zealand and for the whole astronomical community.

The Cannon hack week, day 5

A highlight of group meeting today was Sander Dieleman (Ghent) explaining to us how he won the kaggle Galaxy Zoo challenge. His blog post about it is great too. We hope to use something like his technology (which he also applies to music analysis) to Kepler light-curves. The convolutional trick is so sensible and clever I can't stand it. His presentation to my group reminds me that no use of deep learning or neural networks in astronomy that I know about so far has ever really been sophisticated. Of course it isn't easy!

Ness and I spent the full afternoon working on The Cannon, although a lot of what we talked about was context and next projects, which informs what we write in our introduction and discussion, and what we focus on for paper 2. One thing we batted around was: What kind of projects can we do with these labels that no-one has done previously? I was asking this because we have various assumptions about our "customer" and if our customer isn't us, we might be making some mistakes. Reminds me of things we talked about at DDD.