Computational Modeling and fMRI Demonstrate how the Brain Represents Meaning

Scientists have taken an important step toward understanding how the human brain codes the meanings of words by constructing the first computational model that can predict the unique brain activation patterns associated with names for objects that an individual can see, hear, feel, taste, or smell.

Researchers from Carnegie Mellon University (Pittsburgh, PA, USA) previously have shown that they can use functional magnetic resonance imaging (fMRI) to detect which areas of the brain are activated when an individual thinks about a specific word. A Carnegie Mellon team of investigators has taken the next step by predicting these activation patterns for concrete nouns--things that are experienced through the senses--for which fMRI data does not yet exist.

The study could ultimately lead to the use of brain scans to identify thoughts and could have applications in the study of autism, disorders of thought such as paranoid schizophrenia, and semantic dementias such as Pick's disease.

The team, led by computer scientist Dr. Tom M. Mitchell and cognitive neuroscientist Dr. Marcel Just, constructed the computational model by using fMRI activation patterns for 60 concrete nouns and by statistically analyzing a set of texts totaling more than a trillion words, called a text corpus. The computer model combines this information about how words are used in text to predict the activation patterns for thousands of concrete nouns contained in the text corpus with accuracies significantly greater than chance.

The findings are being published in the May 30, 2008, issue of the journal Science. In the study, nine subjects underwent fMRI scans while concentrating on 60 stimulus nouns--five words in each of 12 semantic categories including animals, body parts, buildings, clothing, insects, vehicles and vegetables.

To design their computational model, the researchers used machine-learning techniques to evaluate the nouns in a trillion-word text corpus that reflects typical English word usage. For each noun, they calculated how frequently it co-occurs in the text with each of 25 verbs associated with sensory-motor functions, including see, hear, listen, taste, smell, eat, push, drive, and lift. Computational linguists typically perform this statistical analysis as a way of characterizing the use of words.

These 25 verbs appear to be basic building blocks the brain uses for representing meaning, according to Dr. Mitchell. By using this statistical information to analyze the fMRI activation patterns gathered for each of the 60 stimulus nouns, they were able to determine how each co-occurrence with one of the 25 verbs affected the activation of each voxel, or three-dimensional (3D) volume element, within the fMRI brain scans.

To predict the fMRI activation pattern for any concrete noun within the text corpus, the computational model determines the noun's co-occurrences within the text with the 25 verbs and builds an activation map based on how those co-occurrences affect each voxel. In tests, a separate computational model was trained for each of the nine research subjects using 58 of the 60 stimulus nouns and their associated activation patterns. The model was then used to predict the activation patterns for the remaining two nouns. For the nine participants, the model had a mean accuracy of 77% in matching the predicted activation patterns to the ones observed in the subjects' brains.

The model was shown to be capable of predicting activation patterns even in semantic areas for which it was untrained. In tests, the model was retrained with words from all but two of the 12 semantic categories from which the 60 words were drawn, and then tested with stimulus nouns from the omitted categories. If the categories of vehicles and vegetables were omitted, for instance, the model would be tested with words such as celery and airplane. In these cases, the mean accuracy of the model's prediction decreased to 70%, but was still well above chance (50%).

Plans for future work include studying the activation patterns for adjective-noun combinations, prepositional phrases, and simple sentences. The team also plans to examine how the brain represents abstract nouns and concepts.


Related Links:
Carnegie Mellon University