Simons Symposium on Evidence

I spent the day at the Simons Foundation, where there were talks on evidence all day. Highlights for me were the following. Thomas Hales (Pitt) spoke about his proof of the Kepler Conjecture (about packings of spheres) and the problem of verifying it. A dozen (yes, 12) referees were assigned, a conference was held and the refereeing work proceeded over a three-year seminar process. And yet they could neither verify nor refute the proof! The proof relied heavily on computers. Hales's response: To eschew the refereeing process and work with formal computational verification, which is a system for replacing mathematics referees with computer referees. The project may complete soon, 25 years after his original proof. The proof runs to 300 pages and contains thousands of inequalities, each of which is numbered with a seven-digit random integer hash. And so on!

David Donoho (Stanford) and Bill Press (Texas) spoke about the problem of reproducibility in medical studies and trials: There is now solid evidence that most biomedical results do not hold up under replication, that most effect sizes decrease as samples get larger, and that independent sets of studies come to different conclusions. There are many possible and probable reasons for this, including perverse incentives, investigator freedoms, and editorial decisions at journals. Interestingly, irreproducibility increases with study citation rate and impact. Donoho argued for moving to method pre-registration for confirmatory trials. Press argued for changing incentive structures. They both also argued for changes to educational practices, which relates to things we are thinking about in the Moore–Sloan Data Science Environments.

Tim Maudlin (NYU) talked about the foundations of quantum mechanics, building heavily on old work by J. S. Bell, who he argued is one of the deepest thinkers ever on the foundations of physics. He asked the key question of whether an effective theory might also be a true theory, and what that would mean. He argued that the foundational issues that plague quantum mechanics undermine its claim to be predictive in a principled way: Sure you can predict g−2 of the electron to 11 decimals, but if you don't know what the fundamental objects of the theory are or mean (you don't have a proper ontology), you are making these predictions with heuristic decisions or distinctions. For example, the idea of "measurement" that has to be invoked in most descriptions of quantum mechanics is not well defined nor non-arbitrary.

calibration problems as correlated noise; shear mapping

In addition to writing a small amount in my Atlas today, I read a few papers and parts of papers. One was an appendix by Michael Cushing (Toledo) about fitting sets of data that include both photometry and spectroscopy with calibration issues. Cushing treats the calibration issues as a form of exceedingly correlated noise. This is very related to things we worked on at #UnDisLo last month, and he has some very good ideas about how to structure the problem. Some of the ideas are related to things Cushing and I talked about when I was visiting Toledo last year.

Another read was a paper proposal by Michael Schneider (LLNL) about doing weak lensing by fitting a Gaussian Process model of the three-dimensional shear map to the data, and then doing cosmological analysis on the samples or posterior for that shear map. This latter idea grew out of discussions in the MBI team that entered the weak-lensing GREAT3 Challenge. The MBI team is trying to think as probabilistically as our computational resources allow.

time-domain astrophysics; changing variables

I spent an hour or so talking about summer projects with Mykytyn, who will be paid to work on the Galex Photon Catalog and related matters in time-domain astrophysics. I also pitched some possible side projects involving machine learning, old-school statistics, and astrophysics. I spent a short time talking with Foreman-Mackey about the difference between a probability density in R and the same density but in lnR. Trivial, of course, but visually (and sometimes pretty conceptually) very different somehow. We decided to put both in the final results of our paper on Earth Analogs (coming soon!).

writing

I wrote text in my Atlas technical description. That is all.

why are catalogs cut at 5 sigma or harder?

Many years ago I wrote a paper with Turner that is totally wrong in its language but not totally wrong in its gist; it argues that maximum-likelihood flux estimates for faint stars tend to be over-estimates, for the same reason that maximum-likelihood parallax-based distances tend to be under-estimates (look up Lutz-Kelker if you want to know more). My old paper uses terrible language to ask and answer the question. I won't even link to it, it is so embarrassing!

This weekend, Rix sent me a note on a substantially similar point. He asks (paraphrasing): When you detect a source at N sigma (in a likelihood sense), what are the posterior betting odds that you are qualitatively wrong about the existence, position, or flux of that source? The answer is a bit surprising: Even at four sigma, most sources have a large posterior probability of non-existence. The reason is partly that there are far more faint sources than bright, and also (in many cases) most of the sky is empty of (conceivably detectable) sources. So your priors are not uninformative on these points. We went back and forth today on language and explication. I am trying to argue that we should write an arXiv-only note about it all this summer.

chaos in the Milky Way

Now that Price-Whelan's paper is done (will appear on arXiv next week), we met with Johnston and Sarah Pearson (Columbia) to discuss next projects. Price-Whelan has been looking at regular and chaotic orbits in the Milky Way. Even in the standard models people use to model the Sagittarius stream, many of the possible orbits are chaotic. We discussed how to establish chaos given finite integrations with finite computation time. The nice thing that Price-Whelan is doing is that he is integrating not just the orbits but also a (literally) infinitesimal tensor bundle around the orbit to see the evolution of the bundle with time. This permits measurement of Lyapunov times. From my perspective, the chaos question is not "is the Lyapunov time infinity?" but rather "is it longer than a Hubble time?" I think that's the key question when we are thinking about modeling a real dynamical system.

Earth analogs

I spent some of the morning pair-writing (the equivalent of pair-coding) the discussion of Foreman-Mackey's paper on the radius and period distribution of small planets. We have re-analyzed with fewer assumptions than previous studies and got different answers about the rate of Earth analogs, which is a (relatively) unusual example of better statistical methods leading to changed results. Most of the differences between what we find and what has come previously can be attributed to the strength of the assumptions about the period distribution as a function of planet radius; separability as it were.

Patel and Mykytyn

It's been a bad week, research-wise. But congratulations to the NYU class of 2014, and especially my great researchers David Mykytyn and Ekta Patel!

DIBs, and Gamma-Earth

I spent the day at the Institute for Advanced Study in Princeton, talking to Bovy (IAS), Menard (JHU), Zasowski (JHU), and Lan (JHU) about the possible use of diffuse interstellar bands observed in stellar spectra from SDSS-III APOGEE to measure the kinematics of the Milky Way disk. We came up with some first steps. The coolest thing we realized is that Bovy's MW disk models (built from stellar velocities) make zero-free-parameter predictions for the line-of-sight DIB velocities, under some "smooth model" assumptions. That is worth doing immediately. The team also gave me lots of good feedback on my work with Jeffrey Mei (NYUAD) on the mean (or really regression-created) spectrum of interstellar absorption from SDSS calibration stars.

Early in the day, Tremaine (IAS) gave me very valuable feedback and reactions to the exoplanet populations paper that Foreman-Mackey and I are finishing up. In particular, he and we agree completely that the thing we should be talking about (which Foreman-Mackey and I call "Gamma-Earth") should be the number of planets per star per unit natural-logarithmic radius per unit natural-logarithmic period, evaluated at the properties of Earth. This obviates atmospheric modeling and does not specify any "bin size". Great minds think alike!

nothing

Not much, other than a very brief conversation with Hill about causality.

PICSciE Symposium on Data Science

I spent the day at Princeton at the PICSciE Symposium, run by the Research Computing group at Princeton. There were impressive talks about data-driven science all day. The highlight for me was a neural science talk by Uri Hasson (Princeton) about simultaneous monitoring of multiple brains in fMRI machines while they listen to coordinated audio, or tell and hear stories. He is able to show, first, that there are reliably correlations in the fMRI-indicated activity in the brain, and those correlations are a function of time in the story. Second, there are reliably correlations between brains that are listening to the same story. Third, there are reliably correlations between the brain of the speaker and the listener! It looks like speech might actually be a way to synchronize, in certain ways, the brain activities of pairs of people. That would be awesome if it holds up, and relates closely to things I read in Wittgenstein many many years ago.

methods across disciplines

In the part of the day that was not spent writing a NASA ADAP proposal, Foreman-Mackey, Fadely, and I met with Jennifer Hill, Vince Dorie, and Marc Scott (all NYU PRIISM) to talk about translating statistical methods from domain to domain and from stats to a domain. Hill and I had called the meeting after thinking that there might be some kind of natural-language project here.

In the end we decided to start along a different path, which is to consider the citation graph in the literature, to find papers that use the same methods but are in different fields or subfields. This would give us a corpus to look at of documents that might require translations. We decided to start in or near astrophysics, because the NASA ADS has the citation graph (and will probably give it to us!).

the Bronx

This does not count as research (see rules at right), but I spent a good chunk of the day up at middle school CIS 303, talking on a couple of panels for "College and Career Day". I learned that middle schoolers (age 12-ish) love astronomy. Also, that they think a bachelors degree and a PhD is way too much time spent in school. The best question I was asked: "Do you use the scientific method?". My answer: "No!" That shocked a few of the teachers!

radio-telescope calibration to string theory

At #NYCastroML, Josh Peek and I had a public conversation about how to self-calibrate single-dish radio observations of the Milky Way HI gas. We were able to cast it into a linear-ish form and develop the hope that he could use a sparse-coding method advocated (effectively) in the Ivezic et al book.

In the afternoon, Kilian Walsh presented the work he has done on cosmology to his PhD committee. He can use the void probability function to constrain the halo model beyond the abundance and clustering constraints usually in use. He is now a PhD candidate; congratulations!

At lunch time, Roberto Gobbetti (NYU) defended his PhD. He presented a mechanical model for slow-roll inflation based on flux discharge around compact dimensions in the string landscape. It is a beautiful model in which a great many things are computable. Let's hear it for the observational consequences of string-like theories! And let's congratulate Dr Gobbetti.

causality, web scraping for science, visualization

A great day! Avoiding proposal-writing, I guess: I met with Jennifer Hill (NYU Steinhardt) to discuss ways we might bring causal inference to astronomy. The idea is not to improve what is being done or concluded in astronomical studies, but rather to clarify the implicit assumptions. We tentatively decided to look at galaxy evolution, because it is a sub-field in which the models are currently very data-driven and aren't backed by strong theoretical ideas. It all might fail, of course, because it is hard to think of things like galaxy environments as "treatments" and things like star-formation rates as "outcomes". But the problems do map onto each other, I think.

At lunch, Raphael Flauger (NYU) gave a beautiful talk on foreground uncertainties related to the BICEP2 results. He built his foreground models as did the BICEP2 team by scraping data out of Keynote (tm) presentations posted on the web! I have to say that again: The Planck team showed some maps of foregrounds in some Keynote presentations and posted them on the web. Flauger (and also the BICEP2 team before him) grabbed those presentations, scraped them for the all-sky maps, calibrated them using the scale bars, and worked from there. The coolest thing is that Flauger also simulated this whole process to account in his analysis for the digitization (scraping?) noise. Awesome! He concludes that the significance of the BICEP2 results is much lower than stated in the paper, which makes him (and many others) sad: He has been working on inflation models that produce large signals.

Late in the day, Enrico Bertini (NYU Poly) rocked the Data Science Showcase series with a nice talk on visualization. Among many other things, he showed that there are known rankings of the value of different kinds of visual cues (bar lengths, dot positions, color, and so on) for displaying different kinds of quantities (real value, rank, binary). Afterwards, there were some conversations about what we should do about scientists who are blind. That's an important question we need to address when we speak about the importance of visualization!

getting funded

A few years ago, I modified The Rules (at right) to permit discussion of proposal writing and related fundraising. I am glad I did that, because otherwise quite a few days might be labeled "not research". Today Foreman-Mackey and I put in a full weekend day on proposal-writing. The big one is for the (excellent, valuable, and influential) NASA Astrophysical Data Analysis Program. This program capitalizes on NASA's public archival data sets by providing funding for individual-investigator re-analyses.

Foreman-Mackey and I are proposing to re-analyze the Kepler data, of course, in order to make it more sensitive (to exoplanets and stellar variability) and more precise. We have four approaches: The first is to build a data-driven pixel-level calibration of the device, in which we use covariances across pixels to model all spacecraft-induced variability. The second is my optimized weighted linear (OWL) photometry, that produces (under strong assumptions) optimal signal-to-noise photometry from the data. The third is our Gaussian-Process code to model stellar (and residual spacecraft) variability or (equivalently) to generalize our exoplanet-transit likelihood function. The fourth is our nascent project to fully model the Kepler focal plane, including point-spread function, sub-pixel flat-field, and the spacecraft Euler Angles. We wrote like the wind.

text, causality

I spent a short meeting in the morning with Alexander Rush (MIT, Columbia), talking about text and ideas for text as data. He works on methods for understanding and translating text, making use of inferences of the structures of the sentences. We discussed translations between different scientific domains, and parsings or understandings or summaries of papers on the arXiv. In particular, he suggested that we might be able to trace idea provenances, or find articles that confirm or contradict one another at a higher level than just pure citation-following.

In the afternoon, I presented to Jennifer Hill (NYU) some ideas about bringing causal inference to astronomy. We are both suspicious that it will work, but it is certainly the case that a lot of questions being asked in astronomy map onto the kinds of questions that causal inference is designed to address in the social sciences. Even if we fail or it doesn't help, there is still a paper to be written.

Dr Hou

In an important day, my student—co-advised by Goodman in Applied Mathematics—Fengji Hou successfully defended his PhD dissertation. In his defense talk he showed different methods for doing inference within a model and over qualitatively different models, both in the context of exoplanet astronomy. Hou's work at NYU has had a huge impact on me; it has got me collaborating with Applied Mathematics and Statistics, and brought valuable techniques to astronomy from those fields. His influence goes way beyond his publication list, since he taught us how to use the affine-invariant sampler that is the guts of emcee and he helped us understand the relationships between adaptive importance sampling, nested sampling, annealing, and other methods. Congratulations Dr Hou!

making GPs faster; Dr Kreiss

Ah, a day where research dominated; the first in a long while. I spent the morning up at Columbia for NYCastroML, where we discussed regression and cross validation. I am a huge supporter of cross validation. We spent some time arguing about the difference between regression, fitting, and "learning". Most importantly, we came up with some good rules of thumb for doing stats in astronomy, and I hope very much someone wrote them down!

After that, Foreman-Mackey and I came downtown for a meeting with Greengard, O'Neil, Ambikasaran, Fadely (all NYU), and Epstein (Penn) about the possibility that we should be propagating the Good News of Ambikasaran et al out to various scientific domains, like machine learning, astrophysics, and neural science. We agreed to search out places where people want to use Gaussian Processes but are stuck on the computational complexity.

Late in the day, Sven Kreiss (NYU) gave a great PhD defense talk on his work on the Higgs at ATLAS. He showed how it was discovered and how its mass was determined. In both of these important results ("Breakthrough of the Year" at Science), he had a leading role. A great PhD and very well deserved, Dr Kreiss.

photon potential-density pairs?

Gabe Perez-Giz (NYU) gave a blackboard talk at lunch about constructing a stable, bound swarm of photons. Yes, photons! It is the relativistic generalization of the search for potential-density pairs for gravitationally bound objects: Can you find a radial mass distribution that is made up of orbits that are themselves orbits in the potential generated by that mass distribution? The audience was skeptical that any stable solutions could exist, but divided on whether any solutions at all exist. I am with Perez-Giz that it is likely that some solutions do exist. Stability, that's another matter.

source detection, causality, translation

It's been a low-research week! Today I got in a brief conversation with Lang, about priorities (for writing up our results) and also source detection and measurement. As my loyal reader knows, Lang has understood in a deep way how we ought to detect stars in images, which, even in the limit of isolated sources, sky-limited noise, and well-known point-spread function, is not absolutely trivial. I committed to writing some text for the paper, which is nearly done.

In the afternoon I had a valuable conversation with Jennifer Hill (NYU), who works on (among other things) inferring causality in social science contexts. I asked her about whether it would be interesting or useful to re-cast any of the work going on in galaxy evolution in causal-inference language. The reason I think "galaxy evolution" is that this is an area where a lot of the important ideas about mechanistic relationships come (directly or indirectly) from playing around in the data. She also encounters many social-science contexts in which (like astronomy) controlled experiments and investigator interventions are not possible, so there is definitely a connection. That said, because there are no "potential outcomes" or even really "outcomes" at all for galaxies, it is not totally clear that it will be useful to perform this translation.

All that talk of "translation" caused Hill and me to formulate a possible natural-language project: Take as a training set a collection of studies in two fields (psychology and astronomy, say) that use the same underlying technical machinery; use this to develop a translator that can translate between the fields. The idea is that even though methods are general and apply across disciplines, often we can't understand one another's papers because of historical differences in language, notation, and context. This seems like a valuable natural-language project that might not require a solution that is also AI complete.

Dr Schillo

Marjorie Schillo (NYU) defended her PhD today. She showed that eternal inflation implies a non-zero spatial curvature for the Universe. More specifically, it links the CMB low-ell amplitudes to the spatial curvature. Since all string-based inflation models are eternally inflating, she can construct hard tests of string theory by measurement of the spatial curvature. Right now there are just upper limits, but there is some chance her tests will be executed. She gave a great talk and handled some heavy interrogation from the floor. Congratulations, Dr Schillo.

Very late in this low-research day, I did some work on the 2014 Kentucky Derby. As of writing, I like Dance With Fate, Wicked Strong, and Medal Count. But that's not investment advice!