SciCafe: Seeing is Believing

SciCafe: Seeing is Believing

MARISA CARRASCO (NEW YORK UNIVERSITY): It’s very exciting to be able to tell you some of the principles of visual perception and attention, and some of the work that we’re conducting in the lab.

One of the things that I would like to convince you about today is that perception is an active process.  We do it all the time.  It seems that we do it effortlessly, but there are thousands of computations that our brain performs every second in order for us to see the way we see.  We actually do a lot of inferential processing while we see. 

This is going to be a public participation talk.  So those of you that cannot see the screen, please do your best to see the screen, because otherwise half of the show is over.  And I’m serious.  So many times I want to say, those to my right do X, and those to my left to Y.  Please do it, because it will be much more fun, okay? 

So the first thing that I’d like you to do is to see this pattern.  To some of you, this pattern is going to emerge immediately, and when you see it you raise your hand, but do not say out what you see, please.  And some of you have question mark faces and are wondering what this is.  So probably when I tell you that this is an animal, it’s going to start making some sense, and I should see more hands up.  And if I tell you that it’s a mammal, ah, you start seeing it, right?  Yes?  Not quite yet, I can tell.

Okay, because we’re short on time I want to help you here.  What if I say, moo.  Ah, you see the cow now, right?  No?  Okay, look at my arrow.  Okay, so here is the face.  There’s the nose.  The two black spots are the ears.  And actually, the cow is looking at you.  Yes?  Can I see hands of how many people see it now?  Great.  If you don’t see it yet, during the Q&A you ask me to put it back.  No, I’m serious.

Now the cool thing is that once you see it, there’s no way you stop seeing it.  I’ve had students in my Intro to Perception class that come four years after to visit me, and I have my little cow postcards.  And they go, oh, the cow, and then they keep going. 

So it’s pretty amazing.  And this is a very good illustration of how the information that we have helps us see.  The physical information that has been imprinting upon your retina at the back of your eye has been exactly the same since the slide is on.   Nothing has changed.  But for those of you that didn’t see it, what changed was actually the knowledge that I told you.  In the moment that I said it’s an animal, the possibilities are constrained.  And once I tell you that it’s a mammal, the possibilities are even farther constrained.

So even though we’re not aware of it, most of our perception every day, it’s an inferential process. We do what we call hypothesis testing; very bright, very fast hypothesis testing.  You know that we have robots that can be the best chess players.  We don’t have robots that can cross the street the way you cross a street.  Just remember that, okay?  We’re pretty amazing machines.

Now here are two examples, again of perception.  On the left, can I see hands on who sees, who recognizes a pattern on the left?  Okay, a lot of people.  Someone just tell me what you see.  Triangle.  Okay, great.  It’s all in your imagination.  There’s no triangle there.  Actually, the luminosity that is within the triangle or next to the triangle is exactly the same.  What is happening is that your brain is assuming that there’s some circles behind it, and that that triangle is occluding it.  And since it assumes this you actually see it as being closer to you and having a higher luminescence. 

These contours that you see don’t exist.  They’re called illusionary contours.  The interesting thing is that if we see what’s happening in your brain, the brain is actually responding as if the contours are there.  And we’ll get to that later.

Now on the bottom right, is there anyone that could tell me what you see?  Two jars.  Perfect.  There are actually two jars, but I’d like you to know this is how little information we have to make out these two jars.  As a matter of fact, unfortunately this screen, nothing shines.  But if you see close to this, you see that there’s a little junction, and with that junction on the bottom is enough for your brain to draw the rest of the information.  And we do that effortlessly all day long.   

Now how do we know this?  When we study psychology and we study humans, we don’t have the advantage that people that work with animals have, whereby you can get to a neuron of a particular animal, put on electrodes and record the activity.  So what we have to do when we work with humans is devise methods and tricks to see how it is that when we manipulate the content of the physical information, our perception changes.

So I want to do a demo.  I want everyone to look at the screen.  There’s going to be a center, and I want you to look at that blue cross.  Please keep looking at it.  I’m going to hypnotize you.  No, I’m just joking.  What happens is that the neurons in the back of your brain are getting adapted, and now you see that that pattern seems to change.  Again, the particular neurons that respond to that motion are getting tired and you get an after-effect. And as we keep going, you would see that that eye seems to be growing bigger and bigger. 

Now nothing has changed, right?  But your neuron got adapted.  Okay, I can show you the original eye later on.   It’s somewhere in the audience.

Okay, now I want to tell you first of all about visual perception, because I want to convince you visual perception is an active process, and that our brain performs these amazing computations all day long.  But I also want to tell you about attention, because it turns out to be that most of the time, we’re confronted with an overwhelming amount of information, and we have to make sense of what we see.  We have to selectively process certain information.

Now the advantage of processing that information is that we can see better, and I’ll show you how that’s the case; we can hear better, and I’ll show you how that’s the case.  But that has a cost.  And the cost is that what we are not selectively processing, consciously or not, is going to be perceived less well. 

So we’re going to do a demo, and it really requires audience participation.  So those of you that are towards my left, I want to listen to the male’s voice.  Those of you that are towards my right, I want to listen to the female’s voice.  Okay?  You have to make an effort, male and female.  Ready? 

[Video played] 

Okay, could you follow the male’s voice?

Audience

Yes.

Marisa Carrasco

Actually, those of you that saw the trivia know that the male’s voice gave you a lot of the answers to the trivia.  Did you follow the female’s voice?  Yes?  That was a little introduction that I wrote for a chapter on attention some time ago that tells you how processing is selective, and it has an analogy with a story that talks about how someone that can remember everything can think very little, because the person cannot forget.

The same happens.  If we have all the visual information represented to the same degree in our brain, there’s no way we could make sense of it.  We need to selectively filter out or discard some information in order to make sense of the other information. 

If you’re interested and you would like to hear what was on the other voice but didn’t hear, I would be happy to play these during the Q&A.

Now I want to give a demonstration of how limited our processing is.  So on the one hand, it’s incredible the computations that our brain makes per second.  On the other hand, there’s a curious fact, and it is that we have a subjective impression that we perceive everything that is out there, and that we do that really well.

So I want to show you how that’s not necessarily the case, and to some degree, how limited our processing is.  So I’m going to show you a series of slides.  It’s two slides that are going back and forth.  And I want you to notice the differences between those two images.  These are former and current lab members.  Actually, some of them are somewhere around there.  And you can see, I hope that some of you detect some changes between both, yes?

Audience

Yes.

Marisa Carrasco

Okay, if you’ve seen one or two, you’re going great.  We’re going to go all the way to nine.  Here are the two images.  Everything that is circled in red is where information changed that you didn’t see.  Oh, thank you.  Okay, so let’s start here.  There was a tree and the top that is no longer there.  This first one was having the glasses on  at the top, but not here.  The color of the flowers changed.  The arms of this person changed.  The cardigan I’m wearing changed color.  Some people no longer have name tags.  More interesting, a person appears and another disappears. 

Now when I show these in class, and actually I did this for you only today.  We did it in the lab yesterday because I wanted to show you some of my lab members, but also there’s many examples that exist like that, and usually people get very excited because they get to see one or two changes. 

I want to show you an example that it’s actually not even flickering, because

Now I want to show you an example that is going to have some changes that are happening very slowly, all in front of you.  There’s no trick here.  Nothing is flickering, okay?  Please pay attention to the screen.  I want you to notice all the changes that you see. 

Okay, can you raise your hand if you saw changes.  About five?  Oh, less than five.  Okay, three.  Okay, last time I counted there were 42.  So this is the original image, and this is the final image.  I’m going to go back and forth.  Anywhere you look, things changed.  Furthermore, I’m going to put them side to side so that you can see what happened. 

Okay, so what you can see is that this door changed color, that the people that were here disappeared, that the whole façade to the right changed color, shape, texture.  The windows changed.  People disappeared, etc., etc., etc.  Wherever you look you will see changes.  And I’m serious, I counted 42 last time, and I’m not sure if I finished.

Now why is this the case?  Why is it the case that if our subjective experience says that we have access to the world in front of us, things are changing and we don’t notice those changes?  Well, we’ve known for a long time that actually, our cognitive resources are limited.  They’re quite amazing, but they’re limited, and our perceptual resources are limited, too. 

And not until recently we know that this is actually related to the bio-energetic cost of cortical computation.  Let me explain what I mean by it.  The high energy cost of neural electivity that is involved in cortical computation limits our ability to process information.  And this is for the following reasons.

First of all, there’s a constant overall energy consumption that is available to the brain.  You saw that it was in the trivial questions.  Actually, our brain, that jelly-like structure that only weighs three pounds, uses 20 percent of the total energy available to the body.  And that’s when we’re adults.  When we’re children it’s about 50 percent. 

Now our cortex uses 44 percent of the energy that is available to the whole brain, and the brain has to maintain life.  Now the normal metabolic cost depends on the spike rate.  That is, every time that a neural response is consuming energy, and the cost of a single spike is actually high.  For that reason, the average discharge rate of the active neurons determines how many neurons can be active concurrently.  All our neurons in our brain are firing constantly.  To keep themselves alive they have to fire once in a while, so that oxygen gets to them.

Now a colleague and friend of mine did this biophysical calculation, and he calculated how many neurons can be active concurrently above base level.  And the answer is quite staggering.  It’s only one percent.  Now we have 86 billion neurons, so that’s a lot, but only one percent of those neurons can be active.  And that is why the brain needs machinery to allocate energy according to task demand.  And attention, the mechanism that I want to focus on, is going to be a manager.  It’s going to help us allocate our limited resources to those things that are important for us in that moment. 

Now what are the consequences of these limited resources for everyday visual perception?  Well, let’s take a visit to the Verrazano Bridge that most of you know.  Here we’re standing on the Brooklyn end of the Verrazano Bridge and the New York Marathon is about to start. 

Now we’re here at the front and we’re waiting to take a picture of our friend.  Now in reality, we have a lot of processing constraints.  So the world, in our brain, doesn’t look like this.  It looks more like this.  In other words, only at those locations that we are directing our eye, we’re going to see a high resolution.  But as we move toward the periphery, the resolution gets worse and worse.  That is why when you read, you have to move your eyes constantly.

Now because we have this limit, and one of the trivia questions said you don’t many—the degrees of visual angle that are legally blind, and the answer is beyond the central 80 degrees.  Because of these, our eyes are going to be swiveling.  They’re going to visit different locations, and wherever the eyes are, the quality of information is going to be better.

Now I’m waiting to take a picture of my friend and I see this bald guy waving.  And I want to know whether he’s waving at me or at someone else.  Now there are two locations of interest for me; one, where I’m looking at; and another on the right, because I know that my friend is going to come on that side.  So what I do is, while I keep looking where the arrow is to the bald friend, I allocate my attention, without moving my eyes, to a peripheral location. 

And regardless of whether we’re aware of this or not, we do this all day long.  Imagine that you go to a place, a conference where you have a name tag.  You feel embarrassed that you forgot the name of the person because they had introduced you to that person for three times in a row.  So you look at a person and you smile, and you’re really reading the tag.  When you’re driving, you’re looking in front, but you look to the sides.  You move again your eyes to the front, but you’re still monitoring that corner where you know that a cyclist is likely to come.  And so on.  I could give you hundreds of examples.  I’ll spare them for time’s sake, but I would be happy to give you more examples afterwards.

Now if we locate our attention there, voluntarily, we say that that’s a covert—because it’s without eye movement—covert voluntary spatial attention shift.  But of course, things happen in our world that we don’t have control of and that we cannot necessarily predict.  So things happen in the environment that attract our attention automatically to.  So while we keep looking at bald guy, something may happen.  Like a flash may happen on the left of the screen and our attention may be transiently allocated there. 

So some of the places that we attend to are places that we want to allocate our attention.  So for example, my father-in-law is here and I want to see if he’s awake, and I’m kind of monitoring him.  But other things happen.  Someone just came on that side, someone waved at me, so my attention was allocated temporarily to that location.  And we know a lot about how these voluntary and involuntary shifts of spatial attention happen, what they do.  I’m just going to give you some examples.

Let me just convince you that there’s two different systems that we have to selectively process information.  One of them is the voluntary system I talked about, which is called exogenous, because it depends on something that we decide to do, or goal-driven.  It’s flexible, and I will explain in a moment what that means, but I’m actually capable of allocating attention to two different locations, and allocating different proportion of resources.  And it takes about 300 milliseconds to be allocated.  This is for humans, for monkeys, for chickens, for every animal that has been studied.

On the other had we have a system that is involuntary, or endogenous. That is, it’s stimulus driven and it’s automatic.  It gets deployed fully, whether we want it or not.  And interestingly, it’s very transient.  It takes only about 100 milliseconds for us to deploy this system.  That is faster than we can move our eyes.  And many labs, and my lab, have a number of studies where we have shown that these two systems are mediated by cortical and subcortical systems, and in most cases, but not in all, these two systems are going  to result in similar perceptual consequences.  I’m actually going to show you a difference.

Now before I tell you a little bit more about the systems, I would like to think a little bit more about perception first.  Because what my lab is interested in doing is understanding how attention affects basic processes of visual information. 

So if you recognize something here, please raise your hand.  The people that are closer are more likely to see Gala in the center of the screen here.  The people that are farther are more likely to see Lincoln’s face.  Is that right?  Now those of you that did not see Lincoln take your glasses off or squint.  I’m serious.  Oh, here it is, right?  Actually, he was quite bright, and he gave us two hints of what we should see.  Here’s Gala’s body and here’s Lincoln’s face. 

Now the information has been exactly the same.  As a matter of fact, those that saw Gala and they didn’t see Lincoln, now look at my hand here and see what you perceive from the corner of your eye.  Now you don’t see Gala, and it’s more likely that you see Lincoln, because the periphery sees different than your foveola.   

Now interestingly, every single pattern that is expressed in the environment is decomposed by our neurons in such a way that we called for a specific contrast, spatial frequency, orientation and face.  And here again I need audience participation.  I want to ask those to my left to close your eyes for the next slide until I tell you to open it.  Ready?  Those to my right, eyes open, please.  Don’t say what you see, just register what you see.  Ready? 

Now we’re going to switch.  Those on the left please open your eyes, those on the right close your eyes.  Ready? 

Now everybody open your eyes, please. And you can just say what you see, okay?  I want to hear tons of voices. 

Okay, some people said Einstein and some people said Marilyn Monroe.  What happened here?  Those of you that said Einstein is because you saw the picture on the left first.  Those of you that said Marilyn Monroe saw the one on the right.  Now if you just go back and forth literally, you’re going to see more of one and less of the other.  Again, if you squint, you see more of Marilyn and less of Einstein, right?

Now this is a hybrid figure in which Aude Oliva took all the spatial frequencies of these images and merged the high spatial frequencies that corresponds to Einstein, and the low spatial frequencies that correspond to Marilyn Monroe.

This is another example by a colleague of mine that has shown that attention also alters brightness.  So if I ask you to fixate on the white central dot, and without moving your eyes, to attend to the bottom circle, you’re going to see that subjectively changes.  You see it as being darker and as being farther away.  And if now you shift your attention to any of the circles on the top, you’ll see that the brightness of the one on the bottom changes.  Does that work?  Okay.

So nothing is changing in the stimulus.  Information in the retina is exactly the same.  It’s just what you are attending to voluntarily that is making a difference. 

Now when we do these kinds of experiments, we’re actually seeing what happens in your brain.  And in particular, what happens in the occipital cortex which is here at the back of the brain; again, because we are interested in seeing how it is that attention affects the processing of information.  And after the information has gone through the retina to the lateral [ventricular nucleus].  It arrives to the back of the brain which is highlighted there, to the occipital cortex. 

Now how do we do that?  Well, we have our observers laying down in an MRI scanner.  Many of you may have had MRIs for a multitude of reasons.  Some of you may have had a functional magnetic resonance imaging.  And the only difference is that now we can see, what is the activity of the brain while people are perceiving a particular stimulus.

And when we work with vision, we have an amazing advantage because the information is represented retinal-topically.  That means that two things that are neighbors in the outside world are also are neighbors in my cortex. 

So what we can do is have people looking at that result.  We have those expanding rings, and you see, on the back of the brain, how the information is traveling.  The more central the stimulus is, the farther to the back, and the more peripheral it is, the more anterior.  This is happening in real time, so we can localize with very high precision any stimulus and see where the activity is in the brain.  Here you see that when that dial in red arrow is on the right, they form activities on the left, and when it’s on the left, the activity is on the right, because all the visual information processes to the other hemisphere.

So we can, with a precision of two cubic millimeters, see where exactly each information is represented in your brain.  And once we do that, we can prescribe from slides, like here on the bottom.  Further, we see how the activity in this occipital cortex is altered as people perform visual perception tasks or attention tasks.  And we know that there’s many visual areas, all on the back of your brain, that specialize in processing different information.  V1, for example, processes orientation, V4 processes color, and so forth.

And then, of course, the brain has to synthesize information.  And we also have ways to inflate the brain, so to speak, so that all the information that is represented in the creases is available to us and we can see how things that are contiguous in the outside world are also contiguous in the surface. 

Now we take into account the fact that we know that if we have a very faint pattern, like the one on the left, there’s little activity on the brain.  But as the stimulus contrast increases, as on the right, you see that activity is higher.  That’s because the neurons are firing more.  And when the neurons are firing more, they have a higher energetic cost.  The blood has to go to those regions to replenish, and what we pick up with an MRI is the difference in oxy and the oxygenated blood.  And the more blood that goes to an area, the more we know that that area was used.  Is that clear?

So the question is, is it always the best for the system to have optimal resolution?  So here we have a painting by Seurat, and depending on where you are sitting, how far you are, how good your eyes are or your glasses are, there may be a preferred representation for you to see the face.  There’s going to be faces that we can hardly make out because they’re too small.  But there’s going to be faces that we can hardly make out because we actually see the dots. 

This is a Pointillism painting. And as you know, when you’re looking at any impressionist painting, there’s a preferred distance to see the painting.  I never understand why people get so close, because they cannot observe the painting.  They cannot appreciate it, right?  So this is just to illustrate that sometimes it’s better not to have an enhanced resolution, but actually probably to have a lower resolution. 

Thank you so much.