Heiko H. Schütt¹; ¹University of Luxembourg

A fundamental methodological question for vision science is how we test whether the representations in different models and brain areas are similar to each other or not. Due to the high dimensionality of visual representations comparing them is not easy and it remains unclear which methods are most useful with variations of encoding models and kernel or distance based methods being the main contenders. Here, I propose novel methods to compare representations based on Bayesian statistics about read out models from the representations. When we apply a prior to the read out weights, we can compute the predictive distribution for read outs from our representation. The predictive distribution is a full description of the inductive bias and generalization of a model in Bayesian statistics, making it a great basis for comparisons. To compare representations, one can assume that one representation is read out from the other one and use the likelihood of getting that representation. Alternatively, one can use distances for probability distributions like the total variation distance or Jensen-Shannon distance to compare the predictive distributions. For a linear readout with Gaussian priors, we can analytically solve all computations without dimensionality reductions of the representations and our metrics just depend on the linear kernel matrix of the representations. Thus, the new methods connect linear read-out based comparisons to kernel based metrics like centered kernel alignment and representational similarity analysis. I apply these new methods to compare deep neural networks trained on ImageNet to each other and to fMRI data from the natural scenes dataset. The new methods broadly agree with existing metrics, but consider smaller sets of representations to be equivalent. They vary less across different random image samples, and have some theoretical advantages. Thus, these new metrics nicely extend our toolkit for comparing representations.