D. Bernhardt‐Walther
OSU, Psychology
Decoding natural scene categories from fMRI activity patterns
Abstract:
Humans are remarkably efficient at categorizing natural scenes. However, there is little evidence of how, or even where, this happens in the brain. Using fMRI, we ask where in the ventral visual cortex there is information relevant for natural scene categories. To do this, we apply statistical pattern recognition algorithms to fMRI data from different regions of interest to ask whether we can differentiate which natural scene category (e.g. forests, mountains, beaches) a participant was viewing. Completely different exemplars of our natural scene categories were used for training and testing the algorithm in order to ensure that it was learning patterns associated with the category in general and not specific exemplars. We show that scene categories can be decoded from fMRI data throughout ventral visual cortex, including primary visual cortex (V1) and the parahippocampal place area (PPA). Moreover, decoding rates are higher for good exemplars of a category than bad exemplars, confirming that our algorithm is learning category‐related representations. Even more remarkably, despite the marked difference in scene statistics, we were able to decode scene category from fMRI data of subjects viewing line drawings of our natural scene categories. Not only was decoding from line drawings comparable to that from color photographs, in later visual areas, including PPA, decoding was just as good when the decoder was trained on color photographs and tested on line drawings (or vice versa) as it was when it was trained and tested on the same image type; that is, we were able to decode across image types (photographs versus line drawings) just as well as within image types. These data suggest that, in these regions, category information in a photograph relies heavily on the edge and line information present. We conclude that scene structure, which is preserved in line drawings and is more similar among good exemplars of a category than bad, plays an integral part in representing scene categories.Humans are remarkably efficient at categorizing natural scenes. However, there is little evidence of how, or even where, this happens in the brain. Using fMRI, we ask where in the ventral visual cortex there is information relevant for natural scene categories. To do this, we apply statistical pattern recognition algorithms to fMRI data from different regions of interest to ask whether we can differentiate which natural scene category (e.g. forests, mountains, beaches) a participant was viewing. Completely different exemplars of our natural scene categories were used for training and testing the algorithm in order to ensure that it was learning patterns associated with the category in general and not specific exemplars. We show that scene categories can be decoded from fMRI data throughout ventral visual cortex, including primary visual cortex (V1) and the parahippocampal place area (PPA). Moreover, decoding rates are higher for good exemplars of a category than bad exemplars, confirming that our algorithm is learning category‐related representations. Even more remarkably, despite the marked difference in scene statistics, we were able to decode scene category from fMRI data of subjects viewing line drawings of our natural scene categories. Not only was decoding from line drawings comparable to that from color photographs, in later visual areas, including PPA, decoding was just as good when the decoder was trained on color photographs and tested on line drawings (or vice versa) as it was when it was trained and tested on the same image type; that is, we were able to decode across image types (photographs versus line drawings) just as well as within image types. These data suggest that, in these regions, category information in a photograph relies heavily on the edge and line information present. We conclude that scene structure, which is preserved in line drawings and is more similar among good exemplars of a category than bad, plays an integral part in representing scene categories.
