Scientists extract images directly from brain

Pink Tentacle reports that researchers at Japan’s ATR Computational Neuroscience Laboratories have developed a system that can "reconstruct the images inside a person’s mind and display them on a computer monitor."

The scientists were able to reconstruct various images viewed by a person by analyzing changes in their cerebral blood flow. Using a functional magnetic resonance imaging (fMRI) machine, the researchers first mapped the blood flow changes that occurred in the cerebral visual cortex as subjects viewed various images held in front of their eyes. Subjects were shown 400 random 10 x 10 pixel black-and-white images for a period of 12 seconds each. While the fMRI machine monitored the changes in brain activity, a computer crunched the data and learned to associate the various changes in brain activity with the different image designs.
Then, when the test subjects were shown a completely new set of images, such as the letters N-E-U-R-O-N, the system was able to reconstruct and display what the test subjects were viewing based solely on their brain activity.

Mark Frauenfelder is the founder of Boing Boing and the editor-in-chief of MAKE and Cool Tools. Twitter: @frauenfelder. Come and hear Mark speak at the ALA conference in Chicago on July 1.

MORE: Science

More at Boing Boing

Eurovision 2013: An American in London

The technology that links taxonomy and Star Trek

djmaxdare

Apologies if anyone mentioned this already, but I have strong doubts that this technology could ever be used to create watchable video of a person’s imaginings and dreams, even if such dreams light up the visual cortex of the brain.
It’s a common misconception that when you think of a familiar object, say your wife’s face, that you can conjure up a photographic image of your wife in your mind. Your mind’s eye is a lot more fuzzy and indistinct than that. I’m a visual artistâ€“if I’m drawing my wife while either she or a photograph of her is present, I can reproduce a realistic image of her in pencil. If she’s not around, though, and I have no photograph, I can’t. At best I can create sort of a vague caricature of her. And that takes a lot of focus and concentration.
I think even high-res visual imaging of a test subjects dreams would just look like a mess to the outside observer.
Guysmiley

Finally! now I can stop (re)drawing my dreams from scratch and just hit the record button!

You, ah, have a fMRI machine in your bedroom?
PeerB

What about sounds? As sounds are happening in time, would it be that they’d have to be recorded sequentially, i. e. in a film?
Would we be able to recognize the sound by seeing it? Or would we have to find a way to translate the visual film into at soundtrack? By which parameters, then?
One way or another it seems we are a little stuck here, as we need to hear what we see to be able to know what we see. To be able to associate this with a sound.
But I guess these obstacles could be overcome by extensionally mapping a great amount of basic sounds to their vascular counterparts.
zuzu

Why this film’s Director’s Cut version hasn’t been released in the U.S. on DVD is one of the most frustrating mysteries ever.

The 280min 3-disc DVD version could be easily imported from Germany (or Italy); Amazon.com had it listed. MacTheRipper 3.0 r14 easily removes the region code (assuming you’ve already flashed your optical drive to RPC-1 firmware), and then you can reburn backups to DVD-R DL discs set to region 0 for use in any DVD player.
Anonymous

If this proves to be good science, it’s flippin’ incredible.

Many people have already commented on other outlets about the possibilities for recording dreams, and I will admit the possibility of that thrills me to no end. Lucid dreaming is a (frequently difficult) hobby of mine, but my dream recall sucks. I’m excited.
dstntmbrk

wow!
scothampton

Finally! now I can stop (re)drawing my dreams from scratch and just hit the record button!
ill lich

STOP STEALING MY THOUGHTS!
Antinous

Yeah, it’s all fun and games until your doctor sees the image of himself with the handcuffs and the goat.
ridl

AAAAAAAAAAAHHHHHHHHHHHHH!!!!!!!

(fetal and shaking in corner)
Cowicide

I bet a lot of people’s dreams will be for some interesting watching… on weed!

http://www.sfgate.com/blogs/images/sfgate/bronstein/2008/08/26/halfbaked4.jpg
ridl

(still shaking in corner, but looking around cautiously)

actually, the dream recording idea sounds pretty cool…
valerierose

We really ARE going to need tin foil hats!
wynneth

comments were TL;DR

However, the main article may point this out (tl;dr) but the boinged section does not – this does not map out the images in a person’s mind, it only maps out what the person SEES. They may simply be reading the part of the brain that is receiving ocular input and nothing else.
arkizzle

I’m fine..
pork musket

Most of the commenters are missing a key thing here… in order to understand the output (in this case, the brain activity), the input has to be a known value. Dreams as input are not known, reproducible values, and thus it is incredibly unlikely you’d ever get something meaningful from the output.
Takuan

baby steps. Imagine what that tweeting baby will see and use.
seyo

The best part is that all they needed to accomplish this feat was an old graphite G4 from about seven years ago.
cinemajay

Link has been Boinged.

“This Account Has Exceeded Its CPU Quota”
Cowicide

@#13 POSTED BY SEYO

Yeah, well… I used to do this stuff many years ago with an SE-30, Photoshop and a bunch of weed.
theWalrus

While this is a pretty cool achievement in neuroscience, I’m more than a little skeptical of predicted abilities to image dreams, or other thoughts.

What will happen when we can read each others thoughts? We’ll probably realize how dull and neurotic each of us really is, and we’ll want to turn it off right away.
Cowicide

@#11 POSTED BY PORK MUSKET

Actually, if I remember correctly the visual cortex fires up nicely during dreams. Pork, we will eventually capture dreams and I predict the home version will be call DreamCatcher 1.0

I’m never wrong about these things.
Samurai Gratz

Using this technology, we can finally build a visor-like device that will allow Geordi LaForge to see!
mdh

I bet a lot of people’s dreams will be for some interesting watching… on weed!

“Until The End of the World” – one (two, really) of the longest movies I’ve ever seen, is about just that.
zuzu

Finally! now I can stop (re)drawing my dreams from scratch and just hit the record button!

Until the End of the World

The prototype is a device for recording and translating brain impulsesâ€”a camera for the blindâ€”and her hitchhiker is travelling the world, filming his widely scattered family to show footage of them to his blind mother (Jeanne Moreau). The chase continues around the world, the nuclear satellite is shot down, causing an EMP effect that wipes out all unshielded electronics worldwide, and the characters wind up in a hidden cave in the Australian Outback, where the recordings are played back. After the death of the hitchhiker’s mother, his scientist father discovers a way to use the device to record human dreams. Several of the central characters become addicted to viewing the playback of their own dreams, while Claire’s estranged lover, a novelist, remains unaffected while he works on a novel about the adventure. It is this novel, ultimately, that rescues Claire from the throes of her addiction via the power of words.

(or Strange Days)
ridl

maybe ten years ago, cowicide. It’ll be “Dreamcatcher Beta”. Tt won’t reach 1.0 for at least five years after initial release.
Cowicide

@#8 POSTED BY RIDL

RidL, they will know if you ate cheese or not.
http://www.npr.org/templates/story/story.php?storyId=4851485
Cowicide

Until the End of the World
http://www.mininova.org/com/830603
Justin Razmus

Wow. How accurate is this information? What an amazing advancement. Science just moves ahead so fast, those of us that don’t keep up with it feel so behind.
Teej

I just threw up in my mouth
Pyros

I’ll be impressed as soon as the Japanese develop a love making robot…that has real feelings.
Petra

‘No one really knows the ones they love
If you knew everything they thought
I bet that you’d wish that they’d just shut up’
Anonymous

I’m not sure what to make of this, did they show each letter of the word one-at-a-time ? The reason i ask is that I got an email the other day (obviously a good reliable source of factual information!) which stated that the brain does not see individual letters when reading words anyway, and that is the reason we can read the following passage:

—snip—
I cdnuolt blveiee taht I cluod aulaclty uesdnatnrd waht I was rdanieg. The phaonmneal pweor of the hmuan mnid Aoccdrnig to rscheearch at Cmabrigde Uinervtisy, it deosn’t mttaer in waht oredr the ltteers in a wrod are, the olny iprmoatnt tihng is taht the frist and lsat ltteer be in the rghit pclae. The rset can be a taotl mses and you can sitll raed it wouthit a porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe. Amzanig huh?

—snip—

And certainly, i read that without any trouble at all. This seems to contradict them pulling each letter out at a time. As we apparently don’t focus on the letters anyway.
JustDisGuy

This is an amazing first step. How long until this technology enables the blind to see?
zuzu

Zuzu, do you know if the US theatrical version of *Until the End of the World* is also on the German DVD?

I’m almost certain it’s not. Maybe if the Criterion Collection ever picked it up, they could do what they did for Brazil.
Cowicide

@#21 POSTED BY JUSTDISGUY

http://news.bbc.co.uk/2/hi/technology/6507189.stm
pollyannacowgirl

Why is everyone so excited? You KNOW this technology, when fully developed will NOT be used for good.
Cowicide

shit, wrong link… here’s the correct one:
http://www.newscientist.com/article/dn7216–bionic-eye-may-help-reverse-blindness.html
lockdowncbp

I don’t want to be one of the cold water throwers but here are a few things that at least raised my eyebrows at this piece. 1)The technique is fMRI, the signal is spatially decent but temporally poor. Even so I find the claim in the article hard to credit as written. I agree with one poster who said that what we likely have is an interesting case of machine learning, in and of itself very worthwhile.
2) As written, this would be big news in my own field, yet I didn’t see or hear anything about it recently at the annual Society for Neuroscience meeting. Not by itself a killer. 3) This will never be able to read thoughts (see 1-temporal resolution). What I have not claimed is that we will never be able to read thoughts, only that fMRI, for all the things it can bring us, will never get us there. Same goes for reading dreams. I’m much sorrier for the later.
Cowicide

@#23 POSTED BY POLLYANNACOWGIRL

I see… the secrets that you keep…

duh.. duh.. duh.. duh… duhnt…

When we’re scanning in your sleep….
Teller

Hello, Texas Hold ‘Em.
Dillenger69

Too bad it works on blood flow.
Imagine recording what happens when someone dies.

This whole thing reminds me of the movie Brainstorm.
Cowicide

@#27 POSTED BY DILLENGER69

Imagine recording what happens when someone dies…

…on weed!

http://www.sfgate.com/blogs/images/sfgate/bronstein/2008/08/26/halfbaked4.jpg
ridl

@This thread:

HAHAHAHAHA!

wotta rollercoaster
Frank W

@ #23 PollyannaCowgirl:

Why is everyone so excited? You KNOW this technology, when fully developed will NOT be used for good.

Well duh. It is fully developed. It’s used to hook you up on a fully immersive MMORPG in which you’re living in the early twenty-first century, that is real reality for all you know. You’re floating in a vat being farmed by AIs. Believe me, you really don’t want to know what the world really looks like.
Takuan

yes, yes she does:
http://ca.youtube.com/watch?v=9vpZnT_unZs
Nylund

If you had a small, portable (or remote) recording device our eyes would become cameras. See something interesting? Just hit record and look at it.

Although this would have potential uses for spies, crime victims, criminals, cops and artists. First and foremost, most assuredly, this would lead to an EXPLOSION in home made sex tapes and peeping toms.
sojourner strange

An Dearthair Mor is coming for us all, srsly.
Flyne

Another report on the same subject.
ill lich

STOP STEALING MY THOUGHTS. . . ON WEED!
Cowicide

#30 POSTED BY NYLUND:
> most assuredly, this would lead to an EXPLOSION
> in home made sex tapes and peeping toms.

About time I earned some cash for my peeping tom skills.

Would I put my recordings up on craigslist?
Anonymous

Dibs on the Guantonamo franchise!
Schorsch

Spoil-sport alert: There is a theoretical limit of resolution for fMRI, because it is based on blood flow. Even if you can resolve a single vessicle, the blood flow still tells you that there was activity surrounding that vessicle that required oxygen, nothing more. By averaging over long periods of time, these researchers could tell the difference between activity related to different letters. It’s not even generalizable between brains.

Stuff like this is just showing off expensive toys, it tells us nothing new or interesting about the brain. I bet none of you would be surprised if I told you the same part of your brain is “active” when you look at the same letter twice. Yet millions are funneled into it every year.
/bitter
Anonymous

i agree with pork. this seems like it is only useful for recalling data that’s already been catalogued. i.e. we now know that if A=B and C=D that B=A and D=C. it doesn’t seem like something that will be able to interpret novel data with much specificity at all.
bookninja

I feel like I’m going to vomit. Damn you, William Gibson.
abe lugo

After the blood flow code is broken, all you need is a nano DVR mounted through a chip in you neck.
Big Brother will then be watch MeTV 24/7.
bpratt

@11: It depends on whether or not the visual cortex acts the same way during dreams as it does when processing actual vision.

What they’re doing is actually a pretty common idea called (and a nifty application of) machine learning. The idea is to feed known inputs (images in this case) while monitoring the not-well-understood outputs (brain activity in this case) then letting the computer’s machine learning algorithm figure out how the two relate (buzzwords here include “neural networks”, “pattern recognition”, “data mining” etc). Once the algorithm has built a mathematical model that relates the known inputs and recorded outputs, you can toss in any old input and figure out what it must have been based on watching the output through the figurative lens of the mathematical model.

So, if the visual cortex really does fire up in dreams the same way it does with actual vision, if it *acts as if it was processing actual vision*, then yeah, we could record dreams. Dead cool/creepy.

But my guess is that it does not…
jimbuck

it’s probably easier to look at what the person is looking at.

but seriously, this is pretty amazing.
KWillets

Machine Learning =~ Curve Fitting.

This type of signal enhancement via averaging has been around for a long time. It’s certainly ambitious to try to extend it to imaging though.

There is the question of how much signal comes from the retina and optic nerve, which would be somewhat x-y mapped already.
JIMWICh

Yes, this is closest to the technology at the heart of Wim Wender’s “Until The End Of The World” as mentioned by several others here.

It’s one of my all-time favorite films, and I recently bought a VHS-to-DVD deck so that I could transfer a video of it to digital for watching on my computers and projectors. It’s a remarkable film in so many ways.

Henry Farber (Max von Sydow), who’s absconded with his research technology from “the Sunset Laboratories in Palo Alto” is, literally, holed up in a cave in central Australia with Aboriginal assistants developing a system that can both read images captured by a person wearing special headgear (which records the biochemical event and visual recording of something being witnessed), secondarily process this with the help of the original recorder watching it being played back, and then finally transmit it to a blind person. In the film, the blind person is Henry Farber’s wife Edith (Jeanne Moreau).

But eventually Henry begins to experiment with recording and playing back dreams. This is where the Aboriginals say, “screw this!” and leave (one says, “Do you think we want you walking through our peoples’ sacred dreams with your fancy cameras?!”), and the remaining people become addicted to recording and playing back their own dreams (the “disease of images”).

Why this film’s Director’s Cut version hasn’t been released in the U.S. on DVD is one of the most frustrating mysteries ever.
Oskar

This is undoubtedly very cool, but I have some small reservations…

It appears from the article that the researchers knew what they were looking for. They knew that the letters from “neuron” would appear. It might certainly be that the data from the neurons were essentially random noise and they calibrated it until they saw an N, an E, a U, and so on.

Maybe this really is working as advertised. But the only way to test it is to test it blind on a random sample. Show ten random letters and don’t tell the data-crunching people what they are. If they can figure it out, then I’ll be much more excited about the news.
RedShirt77

Imagine how folks like steven hawking could use this to do things like write, draw, or any number of things. maybe some similar technologies could help people control cyborg body parts. or give us enough information to create cybernetic eyes that can actually feed an image into your brain.
Cowicide

#27 POSTED BY DILLENGER69:

“Research for a better tomorrow”… only with a shitty today…
http://www.youtube.com/watch?v=OXt0_rD_D2Q

► Also, like that udder movie ‘Strange Days’:
http://en.wikipedia.org/wiki/Strange_Days_(film)
Modusoperandi

pork musket “Dreams as input are not known, reproducible values, and thus it is incredibly unlikely you’d ever get something meaningful from the output.”
So…the output would be a computer’s dream of a human’s dream. How cool is that?
noen

Cowicide
“they will know if you ate cheese or not.”

There are easier, more reliable ways of determining that. heh….
barryfandango

Hmm, must call shenanigans on the images displayed. The pixelation on the first and last “n” of “neuron” is identical.
Cowicide

@#40 POSTED BY BPRATT

Yes, yes… but, we’ll just use a SQUID instead.
http://en.wikipedia.org/wiki/SQUID
Cowicide

@#48 POSTED BY BARRYFANDANGO

You can tell by the pixels and seeing a lot of images extracted from brains in your time?
akirabergman

This will be a direct threat to privacy and security of an individual. Maybe this will be why we will have to migrate to non-organic or hybrid forms. Eventually no hardware would be safe. Then we may migrate to hyper dimensions and become spirits. And maybe this is also why god migrated to hyper dimensions and left us here ;-)
Anonymous

I don’t see why any image could be interpreted. You don’t need a known pattern to decipher what is there…in fact, the best method I can think of is using a neural network. Once enough “known” inputs are given to the computer program it will be able to determine what is being seen in the brain.
Cowicide

@#51 POSTED BY AKIRABERGMAN

Gimme some of ur weed, Akira.
Anonymous

Zuzu, do you know if the US theatrical version of *Until the End of the World* is also on the German DVD?
wolfiesma

I’d like to see them try. There is no way anyone could possibly understand or iterpret the results.
akirabergman

I must do a follow up to my half-joking comment #51;

Consciousness works like the eyes and ears. It can open, close, focus, anti-focus, and also go fuzzy. Zen and Buddhist meditation is all about practicing the dynamics of this quintet. The center is the fuzzy domain and it has the most power.

The measurements referred to in this post are all about the focus part. If you don’t focus they can not measure anything. Don’t you worry about it. We are safe. Start practicing meditation though. Just in case.

Don’t forget the uncertainty principle of quantum mechanics and information theory. The universe and it’s subsets are no dummy.
Cowicide

Someone had to do it…
http://flickr.com/photos/49403380@N00/3101672444/
Yorgle

There was a paper published, and on the net about 8 or so years ago that did some image reconstruction from a cat’s eye. This seems like the next progression in that research area.
Jesse M.

Did they have the subjects keep their eyes as still as possible? You’d think each time you moved your eyes around the regions of the visual cortex corresponding to different parts of the letter would switch around.
RedShirt77

“This will be a direct threat to privacy and security of an individual”

I wonder what the range will be on future mind reading devices will be.

It will make cheating on tests a lot easier if you sit close enough to the smart kid.
jody

Goodbye keyboard!
Anonymous

Wow, human brains have a lot of JPEG artifacts…
Anonymous

Ah, I accomplished this with a C64 and a programmer’s manual in 1986. You can still get the interface for the 1541 –> cerebral cortex on ebay.
Strophe

@60: Not mine. I see in SVG format.

Remember the mysterious tongue growths post from a while back? I’m going to use that as a countermeasure to this technology. Whenever I think I’m being recorded, I’m going to stare at that to gross out my audience.
Nesbitt

If input has to be known to make sense of the output, then this technology could used to determine how animal, insect, invertebrate brains/senses work, and perhaps decode how they “see”. Now that’s trippy…

If known input ceases to be a limiting factor, then forget dreams, perhaps you could output what someone is *imagining*. Once you have the image, hook that sucker up to a 3D modelling tool, or a 3D Fax style machine. As Morbius said, imagine a civilization without instrumentalities!

It’s meeeeeee!
Brainspore

They still can’t tell if I’m just imagining seeing other people naked, right?
Oskar

@48: Extremely good observation! That really needs to be explained.
Takuan

and fifty years ago someone rigged a toy train set to stop and start when alpha rhythm was applied, we’ll see.
jfdonohoe

The idea of a “dream recorder” is an interesting one to noodle on. I wrote a treatise for a short story on it. Basically it has to do with the idea that recorded dreams would either mean nothing to anyone else but the dreamer or reveal some fundamentally primitive aspects about your mental makeup to others.

Dreams are like movies. Except with dreams, instead of editing together different scenes of film to come up with a narrative whole that is greater than its part’s sum, a dream stitches together experiences that your mind has recorded into its cognition centers into hap-hazard new experiences.

Think of a dream about flying. Is your ‘dream self’ really experiencing the feeling of self-propelled flying? No, because you never experienced that in real life. And the human mind cannot imaging what it has not experienced. It can only cobble together what it assumes is an approximation of the experience from other experiences it has. In other words someone with the experience of “weightless in a swimming pool” added to “jumping off a fence” added to “the rush of playing on a swing set” can all be added together in a dream to give a sense of flying.

But everyone’s experiences are unique to themselves. Therefore, would a “movie” of edited experiences make any sense to anyone but the original dreamer?

Or if you could make out what experiences made up a person’s dream, it could be a pretty naked insight into what things that person focuses on.

I know this is rambling, but I think this is really interesting stuff!
Anonymous

Take the output in real time and place it infront of the subject (viewer) > feedback loop > instant hallucinations > no need for LSD.
Anonymous

@62
We wouldn’t be able to “see” what an insect sees, only determine relatively what they “see”.

For instance, an ant spying a cupcake might see a big mound of food particles. But to us, it’s a cupcake. So if we measure the ant’s brain flow to be a certain thing, and we recognize it as a cupcake, we’ll code it as a cupcake. When the output from the machine comes up, it will be a cupcake, not a mound of food particles. Make sense?

Insect sees A, we see B. Machine decodes A into B. We still don’t see/understand A, but we know what B is.
Chris L

I like how the comments are pretty much evenly distributed between skeptical, scared, and SQUEE!!!

I, for one, will happily sell my soul to Big Brother for a working dream recorder.
Anonymous

“and you said you’d wait until the end of the world…”
Antinous

We could probably view dreams in a standardized visual dream language. Dreams largely consist of bits of archetypal matter. If it’s possible to analyze patterns for archetypal images, they could be viewed as a set image language. We wouldn’t exactly see what someone was dreaming, but we might see the graphic novel version of the film.
Anonymous

“If my thought dreams could be seen, they’d probably put my head in a guillotine.”
– Bob Dylan
SevenYears

There’s a big difference between recording things people SEE and things they IMAGINE (or dream). It sounds like ATR was recording from V1 (primary visual cortex), which according to most research only activates when there’s a real visual stimulus.

Also I don’t think fMRI has the spatial resolution for what they’ve claimed to do. Maybe if the subject is scanned over and over and over again for the same stimulus, something meaningful could be extracted.

Also also, individual differences are profound, meaning you have to train the system on each individual before you could use it.

I’m SKEPTICAL. Let’s see what NEURON publishes
seyo

Who’s holding???
InsertFingerHere

The MPAA has already demanded a DRM protected neuron interceptor device be added to any commercial application of this technology.
FoetusNail

#36 FTW, the only thing this proves is this a very good machine and software. The first people to benefit from this technological achievement will be the sales team.