non-convolutional neural networks

Here's a quotation from an email I sent to Schölkopf (MPI-IS) and Villar (JHU) today:

First, I believe (am I wrong?) that a CNN works by repeating the precisely identical weights for every pixel. So if, in a CNN layer, there are k channels of 3x3 filters, there are only 9k weights that set all the k responses of every pixel in the layer to the 3x3 pixels centered on that pixel in the layer below. The sparsity comes not just from the fact that each pixel in one layer connects to only the 9 pixels below it, but also from the fact that the 9k weights are the same for every pixel (except maybe at edges). That enforces a kind of translation symmetry.

Okay, now, we could make a non-convolutional neural net (NCNN) layer as follows: Each pixel is connected, like in the CNN, to just the 3x3 pixels in the layer below, centered on that pixel. And again, there will be k channels and only 9k weights for the whole layer. The only difference is that at each pixel, a rotation (of 0, 90, 180, or 270 degrees) gets applied and a flip (by the identity or across the x direction) gets applied to the weight maps. That is, every pixel has the same k filters applied but at each pixel, there has been one of the 8 rotation-reflection transformations assigned to the 9k-element 3x3 weight map. This NCNN layer would, like the CNN layer, have 9k weights in the layer, and it would be just as local and sparse as the matching CNN layer.

My conjecture is that the NCNN will perform far worse on image-recognition tasks than the CNN. It is also (fairly) easy (I believe) to build a NCNN from a light modification of a CNN code. Comparison is clean and straightforward. I am ready to bet substantial cash on this one.

hypothesis testing and marginalization

I had a valuable chat in the morning with Adrian Price-Whelan (Princeton) about some hypothesis testing, for stellar pairs. The hypotheses are: unbound and unrelated field stars, co-moving but unbound, and comoving because bound. We discussed this problem as a hypothesis test, and also as a parameter estimation (estimating binding energy and velocity difference). My position (that my loyal reader knows well) is that you should never do a hypothesis test when you can do a parameter estimation.

A Bayesian hypothesis test involves computing fully marginalized likelihoods (FMLs). A parameter estimation involves computing partially marginalized posteriors. When I present this difference to Dustin Lang (Toronto), he tends to say “how can marginalizing out all but one of your parameters be so much easier than marginalizing out all your parameters?”. Good question! I think the answer has to do with the difference between estimating densities (probability densities that integrate to unity) and estimating absolute probabilities (numbers that sum to unity). But I can't quite get the argument right.

In my mind, this is connected to an observation I have seen over at Andrew Gelman's blog more than once: When predicting the outcome of a sporting event, it is much better to predict a pdf over final scores than to predict the win/loss probability. This is absolutely my experience (context: horse racing).