model complexity

On the plane home, I continued reading Ke Xiao's dissertation and wrote a bit about model complexity. I worked on an example from Yann LeCun (NYU) that illustrates very clearly that model complexity is not directly related to the number of free parameters. It is related to the range of (reasonable) observables that could be fit by the model, which—for non-linear models—is related to the sensitivity of the observables to the parameters, and not the number of parameters per se.