This article is taken from the monthly Sciences et Avenir – La Recherche n°910, dated December 2022.
It is hard to believe, but it is now possible, almost literally, to read minds. To be precise, we would rather say “decode cognitive content in the brain “, but the results are there: images, words read or heard, movements, even emotions… In the brain, everything is beginning to be seen, even intimate feelings, thanks to progress in neuroimaging coupled with computing power increased tenfold in recent years.
“The idea is that any cognitive activity corresponds to a certain pattern of brain activation, asks Bertrand Thirion, researcher from the Mind team at the National Institute for Research in Digital Sciences and Technologies (Inria). However, if we can observe the activity of your brain, we can go the opposite way and say what is the cognitive activity corresponding to such neuronal activation; and thus, reconstruct an image that you see, or that you imagine, or even sentences with certain semantics.“
Reconstruct the words a person hears “in her head”
Analyzing a person’s brain electromagnetic waves to reconstruct the words they hear “in their head”, this is precisely what a French team from the European laboratory Meta AI Research (ex-Facebook), from the Inria and the universities of Paris-Saclay and Paris Sciences & Lettres.
The study published in Scientific Reports this fall has something to impress: the brain wave data of 169 subjects, captured while they listened to recordings of people talking, was cut into three-second sequences. These brainwave sequences were fed to an AI along with the corresponding sound files to learn to spot patterns of brain activity corresponding to words.
If the demonstration is striking, the accuracy of the algorithm is still too low to consider short-term applications: for each brain recording, artificial intelligence (AI) predicted a list of 10 words, and 73% of the time, this list included the word actually spoken. Can do better then… or not, the resolution of neuroimaging has its limits.
As far as images are concerned, Bertrand Thirion had, in 2006, with Stanislas Dehaene, an expert in cognitive neuroscience, mapped a correspondence between the points of the cortex of the visual areas and the retinal coordinates in the eye. So that by looking at the activated points of the cortex, it was possible to reconstitute simple images – letters in particular – corresponding to the neuronal imprint of the retina in the visual areas. Since then, artificial intelligence has gotten involved, making it possible to use not only the retina, but all the areas activated in the brain.
In 2019, a team from Kyoto University (Japan) caused a sensation with reconstructions of the perceptual content of patients’ brains based on data collected through so-called functional magnetic resonance imaging (fMRI). This application of classic MRI makes it possible to follow brain activity almost live. Almost, because what fMRI actually tracks are variations in blood flow in the vessels of the brain.
In this case, the principle is simple: volunteers placed in the MRI device concentrate on images that are shown to them for a few seconds (a swan, a leopard, a stained glass window, an airplane, etc.), the scientists analyze the imagery data and an artificial intelligence algorithm tries to render the shapes and colors of the image in question. “What’s new in recent years is that we have more powerful generative models to create increasingly realistic text and images. “, specifies Bertrand Thirion.
These programs have learned, on vast databases, to automatically generate new images or texts from criteria defined by the text, or, in this case, neuroimaging data; most often through fMRI or magnetoencephalography (MEG), or even through brain implants. It is still necessary to know how to finely analyze the activity of the various networks of neurons which take part in “mental” perception.
Another striking experience from this point of view: that published in Scientific Reports by neuroscientists from Radboud University (Netherlands) at the very beginning of this year. They managed to reconstruct faces with stunning realism from, again, patterns of neural activation imaged by fMRI while volunteers looked at photos of faces.
“These are the most accurate reconstructions of face perception to date, rejoices Thirza Dado, first author of the study. But given the rapid advances in generative modeling, we expect even more impressive reconstructions of perception, and perhaps even mental imagery in anear future. In the future, we will be able to decode and recreate subjective experiences, perhaps even our dreams. These techniques will also have clinical applications, for communication with patients who are paralyzed or in a state of minimal consciousness in particular. Here we are developing brain implant cameras that will stimulatethe brain in an attempt to restore sight to the blind. “
Using MRI of the neural activity of two volunteers looking at photos of faces (line 1), an AI reconstructed these faces with stunning realism (lines 2 and 3). Credits: THIRZA DADO ET AL. NATURE 2022
Knowledge is progressing, and algorithms are not everything. To reconstruct these faces, it is still necessary to know that their neural encoding with their fine characteristics is done in a specialized area of the visual system assigned to their recognition. The precise parameters are encoded in this zone: shape of the face, color of the skin and hair, orientation of the head, illumination, whether or not to wear glasses, beardless or hairy face, etc. This is what the AI compiles and analyzes to generate these faces.
“This is what I call a reconstruction in a hallucinatory mode, comments Bertrand Thirion. It’s like constructing a perfect hallucination, because the reconstructed image is not exactly what you saw at the time, but will resemble the characteristics assigned in the brain to the face seen: female, with hair long, more or less young, etc. Several dimensions will guide the generative model. “But, however accurate and graphic the face rendered by the AI is, it will always be slightly off. “We kind of hallucinate the psychological experience of the person from his brain imagery “, explains the researcher.
“The neural code is very mixed in the brain”, Stanislas Dehaene, professor at the Collège de France*
“Reading minds? A number of very impressive results have been published in recent years. But we must not overestimate what we are capable of doing. The experiments are carried out on voluntary, concentrated people who agree to remain focused on an image or a thought. I don’t really believe in the fact that non-invasive imaging methods such as MRI or MEG can achieve a level of sophistication capable of perfectly deciphering the neural code.
In general, this one is very mixed, and the more one looks for detailed information on cognition and psychology, the more it is mixed. That is to say, the groups of neurons are increasingly small and distributed in the brain. But assuming these techniques are perfected to the point that they can decode a person’s cognitive content on demand, this will raise serious, and frankly very complicated, ethical issues.”
*Holder of the Experimental Cognitive Psychology Chair. He is also responsible for the cognitive neuroimaging research unit at NeuroSpin in Paris-Saclay.
50 mapped and decodable cognitive tasks
Impressive as these experiences are, they do not serve ultimately only to validate the knowledge acquired, confirming by an objective measure what scientists have managed to decipher in brain activity. The objective is to understand how the distributed cognitive functions work in the brain. However, these experiments focus on very specific tasks.
This is why the Inria-CEA Mind team and the NeuroSpin center in France are now focusing on a challenge that is both more basic and more complex: “Learning to guess what a subject is doing at a given time, without presupposing anything, from imaging data alone “, summarizes Bertrand Thirion. With his team, the researcher published in April in Scientific Reports a census of 50 mapped and decodable cognitive tasks: movements of the toes and fingers, left or right, recognition of faces but also facial emotions, mental calculation, decision-making, recognition of sentences, understanding of a story…
“We have demonstrated for the first time that a totally open decoding, which does not anticipate any precise activity, makes it possible to guess roughly what the person is doing, mentally or actually, among these 50 cognitive activities. “As disturbing as it is, this increasingly detailed understanding of brain activity opens the way to applications that are still difficult to conceive. As long as the purposes remain strictly scientific or medical, everything will be fine. But the The prospect of technologies capable of uncovering the conscious content of a brain and, thanks to implants, modifying it is fascinating as well as worrying.
In search of the perfect neuroimaging
To get as close as possible to active neurons, scientists have two non-invasive techniques: functional MRI, which has a spatial resolution of the order of a millimeter, allowing a fine division of the brain into about a million points or voxels (3D pixels). On the other hand, the activity of the neurons being captured indirectly, the information is recorded with a slight delay, to within two or three seconds. This is fine when a person is exposed to a still image, but not for the rapid transitions involved, for example, in understanding language.
This is where magnetoencephalography (MEG) becomes relevant, because by focusing on the electromagnetic waves emitted by neurons, it can record what is happening down to the millisecond. On the other hand, the electrical signal picked up by the electrodes on the scalp being deformed by the cranial box, the spatial resolution is of the order of a centimeter.
To obtain both millimeter resolution and millisecond signal capture, there are still electrodes implanted in the brain, particularly in epileptics equipped against seizures. But then only a small region is recorded and the overall view of the brain is lost. In theory, to perfectly decipher this neural code, it would be necessary to have a technique capturing the whole of the brain while having a resolution to the nearest neuron and an image acquisition time of the order of a millisecond; almost a scalable 4D digital twin.
Short of slicing the brain to use microscopy, it is still impossible to achieve such resolution. And even so, with around 100 billion neurons in a brain, the amount of information would probably be unmanageable.
We would like to thank the author of this write-up for this outstanding material
