AI Tool Reconstructs Viewed Images From Brain Scans, Researchers Warn of Privacy Risks

Michal Irani, who developed the tool with colleagues at the institute, hopes the 'mindreading' tool will ultimately reveal more about how the brain works. Judy Illes, a neuroethicist and professor of neurology at the University of British Columbia in Canada, who was not involved in the research, described the work as 'magnificent.' She said the idea of using the approach to help people with neurologic conditions therapeutically is 'tremendously exciting.'

Other scientists warned that a similar approach could reveal people’s inner thoughts and mental imagery, potentially without their consent. 'The results seem very impressive,' said Tommy Sprague, a neuroscientist at the University of California Santa Barbara. 'But if there is a way to surreptitiously extract information about what you are thinking about, then…150 years of sci-fi can come true anytime, and that is worrisome in a lot of ways.'

Neuroscientists have worked for years on ways to reconstruct what people see and what is going on in their minds. Early attempts produced blurry, hard-to-interpret images. Advances in fMRI scans and in the tools used to analyze results have improved the reconstructions over time, according to MIT Technology Review.

fMRI uses a giant magnet to track the flow of oxygenated blood through the brain. Brain areas that 'light up' on scans are thought to be particularly active at a given moment. In typical fMRI scanners, each highlighted voxel of activity covers around three cubic millimeters and contains around 16,000 neurons. Irani and her colleagues used newer datasets collected with higher-resolution scanners, with each voxel covering around one cubic millimeter of neurons, she said.

Other teams have also used brain scan data to recreate images, but Irani said those tools are not good enough. If a person saw a banana, for example, existing models can generate an image of a banana, but it would look different, lacking the same structure and position.

To more closely recreate viewed images, the team trained an AI model on already available data from eight people who had each been shown around 9,000 images while in a high-resolution fMRI scanner. Their 'brain decoder' has two branches: one predicts the structure of an image, such as where colors are, and another predicts its content, such as a bunch of bananas on a plate. The predictions allow a diffusion model, a type of AI known for generating images by gradually cleaning up noisy pixels, to produce a more accurate representation of what the person saw.

The team needed more data than was available. To address that, they trained another model in the opposite direction—an encoder that can predict brain activity from an image. The encoder and decoder were then used together to improve both tools. Starting with a new image, such as a leopard, the encoder predicts what a person’s fMRI scan would look like when viewing it, and the decoder reconstructs the image again. At first the reconstruction may not look much like a leopard, Irani said, but repeated training this way eventually leads to dramatic improvements.

Irani said around 70% of the training data came from images that were not originally paired with fMRI scans. By combining data from multiple studies, the team also identified brain regions that appear to share functions across individuals. One region seemed to respond to images of food, for example, while another responded to images of sports. Irani is a computer scientist.