Posted by Barry B. Sandrew Ph.D.               Wednesday, February 1, 2012

Copied from his Blog BSandrew.blogspot.com, with permission

Engaged in 2D and Immersed in 3D

This month, the multiple 3D films being recognized by the 84th Academy Awards cement the fact that 3D is here to stay. On the heels of the Best Visual Effects nominations for Hugo and Transformers: Dark of the Moon, Legend3D Founder, President, CCO/CTO Barry Sandrew dissects the appeal of 3D for the moviegoer in this two-part blog series on the neuroscience of 2D and 3D feature films. Relying on our current understanding of stereopsis, as described in scientific literature, Sandrew contemplates how our brains differentially perceive images and activate emotional responses when viewing a 2D movie versus a 3D movie.

The group dynamics of the 2D experience

When we attend a 2D movie, the group dynamics of the audience come into play as we direct our collective attention as observers to the images on the screen. Our personal space is infringed upon only tangentially by the other audience members that are sitting on either side of us, as well as by those behind and in front of us. The tangential influence upon us of the surrounding audience is a reminder that we are in a theater with strangers, all watching images being projected on the screen. However, in a 2D movie, our fellow audience members typically do not detract from the experience.  In fact, the audience in a 2D movie is intended to be an integral part of the total feature film experience and has been presented as one of the selling points of attending a 2D move in a theater rather than watching it on DVD or Blu-ray at home. As we become engaged in the movie, we can introspectively sense surprise, anger, sadness and happiness based on the perceived action on the screen and to a lesser degree, through the reactions of the audience members around us. We can empathize and sympathize with the characters being portrayed and we can experience anxiety and fear as we follow the storyline.

Disparity—A key element in the perception of stereo

Our brains receive 2D information in a theater as images presented sequentially at 24 frames per second. In a 2D theater experience, motion is perceived as changes in the up, down and sideways position of one object in relation to other 2D objects.  We perceive three dimensions in a 2D movie via single eye or monocular cues such as occlusion, motion, shading, size, parallax, texture gradient, perspective, saturation and brightness. However, lacking in a 2D movie is the most powerful 3D influence on the brain—disparity. Due to the separation of our eyes and the distance between them, we see a slightly different image from each eye.  This separation causes a horizontal displacement between the images, which is referred to as disparity. The best way to test this is to put your thumb in front of your eyes and alternately close one eye and then the other. You will see two separate perspectives of your thumb displaced horizontally and your thumb will appear to jump from side to side. When these separate images from our left and right eye reach our visual cortex, they are directed to highly specialized neurons that are tuned to critical parameters of disparity.  As a consequence, the two images are fused into one uniquely different image that exhibits the appearance of depth and volume. The result is 3D perception.

A lack of disparity contradicts monocular cues

In a 2D movie, each of our eyes receives precisely identical images from the movie screen with no horizontal displacement. When the two sets of identical images reach the visual cortex of our brain, they are directed primarily to neurons tuned to zero disparity. There, they are “fused” together into a single image that is exactly identical to each of the two images that originally comprised it. Other than the existence of monocular stereo cues, mentioned above, the audience does not perceive depth and volume due to the absence of disparity. This lack of disparity in a 2D movie creates a cue conflict situation that prevents us from being fully immersed in the story in the same manner as a 3D movie. The monocular cues in a 2D movie are telling us there is depth, but the lack of disparity contradicts the monocular cues—forcing the brain to try and reconcile the conflict.

The absence of disparity “pushes” the screen away from us

For some monocular cues, like motion and occlusion, there is a further complication that acts to distance us from the movie screen. As mentioned above, in a 2D movie the relative motion of objects on the screen (one 2D object moving relative to and possibly occluding another 2D object) can indicate depth in the absence of disparity. However, in the real world when there is this kind of relative motion in the absence of disparity, our brains normally interpret that information as indicating that the objects are far away. That’s because disparity in human vision drops off significantly at distances of several hundred yards.  You can prove this if you try the thumb trick above, alternately closing one eye and then the other, but this time, try the trick looking at a tree that is several hundred yards away.  You’ll see that the tree will not appear to jump from side to side like your thumb did. This is because the perspectives from our two eyes at that distance are identical. Our eyes do not converge on the tree, but rather are set to infinity.  The father away things are from us the less horizontal displacement we experience.  So our brain “comes up with” a solution in which we are literally “distanced” from the action on the screen. This distancing and lack of disparity in a 2D movie renders the spatial areas in front of and behind the theater screen irrelevant to the story and therefore irrelevant to each member of the audience. The storytelling, in its entirety, happens on the white screen at the end of the theater.

Skillful use of disparity in a 3D movie will always enhance the experience

In spite of the contradictions our brain must reconcile, a 2D movie is almost always comfortable to watch and with proper directing, cinematography and storytelling, it can evoke very strong emotions. Indeed, many of us remember the outpouring of emotions while watching the sinking of the Titanic or the building terror of T-Rex in Jurassic Park, as both movie experiences were presented in 2D. However, the influence of disparity on the audience in a 3D movie cannot be trivialized as a fad or unnecessary. Whether a 3D movie is captured with two cameras or converted from 2D-to-3D, disparity that is properly and skillfully stereographed will always enhance the storytelling experience by significantly amplifying the responses of disparity-tuned neurons and therefore more closely simulating reality.

3D a higher resolution medium

In the visual system, the absolute number of neurons activated in the brain is not as significant as the ratio of activation between subpopulations of neurons responding to zero disparity and those responding to a wide spectrum of tuned disparities, including zero.  This is analogous to an audio amplifier where the ratio between subpopulations of audio frequencies can vary and increasing the gain on the system increases the overall fidelity of the audio experience, meaning it becomes closer to the original source in resolution.  In the same manner, the ratio of activated zero disparity and disparity-tuned subpopulations of neurons in a 3D movie can vary, but it’s the intensity of the relative responses of those subpopulations that create a more accurate and higher resolution stereo image.  It appears that our brains interpret this higher resolution information as more closely simulating reality (“fidelity”) and therefore it immerses us in the movie to a greater degree, creating a heightened emotional investment in the story.  This is, of course, an oversimplification of an exquisitely complex process that has evolved from the earliest primates. However, the theater going experience is something we can all relate to and many of us recognize the profound difference between being engaged in a 2D movie versus being immersed in a 3D movie.

3D movies are a uniquely personal experience

Dynamic disparity (i.e. the relative amount of disparity, changes in disparity over time and the rate of disparity change) transforms the movie screen from a projection screen into a window that has both an interior and an exterior. Consequently, the storytelling environment is projected throughout the entire theater, actively becoming a unique part of the personal space of each member of the audience and resulting in the space both in front of and behind the screen becoming an integral part of the story. For example, an object that might fly out of the screen at us, traveling through what is called negative parallax, becomes very personal for each member of the audience because it affects and/or “intrudes” each audience members’ personal space equally.  Surround sound has an analogous effect in that we are “in it” rather than simply listening to stereo audio in front of us.  As a result, unlike the 2D movie experience, a 3D movie is more personal and the group dynamics of the larger audience are no longer in play to the same extent.  In fact, it’s my opinion that 3D glasses actually have the positive effect of helping to separate us from the other audience members, containing the experience as a more personal one for each of us, thereby potentially amplifying the immersive nature of the experience.

The Bottom Line

When we are engaged in a 2D movie, we are doing so as passive observers, watching the story take place “over there”, on the screen in front of us. However, when we are immersed in a 3D movie, we are doing so as active participants and the action can be happening behind and in front of us—as far as infinity. This is where the central difference lies. We cannot discount the uniquely active physical and emotional reactions that we have in the 3D feature film experience which are not experienced in our more passive reception of 2D movies. I believe, this is one of the reasons why we have seen 3D continue to flourish and be adopted so enthusiastically by both national and international movie exhibitors, as well as by the entertainment and consumer electronics industries at large. Consumers and moviegoers are hungry for visuals that offer a significantly greater sense of engagement and for visuals that spark an immersive, emotional response that has the ability to transport them into the heart of the action.  This year, we will have the opportunity to test the differential 2D and 3D theatrical experiences with three iconic movies that were originally released in 2D, being released in 3D. Top Gun has been converted by Legend3D and will be re-released on its 25th anniversary by Paramount on the heels of 3D converted re-releases of Titanic and Star Wars.  I am confident that all three films will be very successful at the box office, as each was already a proven success in 2D. Depending on how skillfully each of these films was converted, now in 3D they will give those who saw them originally in 2D an immersive and refreshingly unique experience; an experience that will hopefully help to solidify 3D as an essential part of filmmaking.

Next, we will continue to look at the neuroscience of 3D movies and further explore the uniquely tuned structures and neurons in the brain that respond selectively to dynamic disparity. We’ll look at concepts of visual processing that remain contentious within the scientific community, involving differential pathways within the brain that signal separately “where something is” and “what something is.” The evolutionary survival value of binocular vision will be discussed in the context of 3D movies and we will touch upon more primitive structures in the brain that are likely triggered by disparity to elicit powerful physical emotions and ‘flight’ or ‘fight’ reactions. For more detailed information and technical reviews on stereopsis, as well as in-depth information on the uniquely tuned disparity neurons in many parts of the brain, please see the following references:

(1) Cumming BG, De Angeles GC. (2001). The Physiology of Stereopsis. Annual Review of Neuroscience. 24, 203-238.

(2) Howard, IP, Rogers, BJ. (1995). Binocular Vision and Stereopsis.  New York: Oxford University Press.

(3) Poggio, GF, Poggio, T. (1984). The Analysis of Stereopsis.  Annual Review of Neuroscience. 7, 379-412.

(4) Born, R., Bradley, D. (2005). Structure and Function of Visual Area MT. Annual Review of Neuroscience. 28, 157-189.

I wish to acknowledge the review and insightful suggestions of my neuroscience colleague, David Heeger, Ph.D., Professor at New York University (http://www.cns.nyu.edu/~david/) where he is a member of the Center for Brain Imaging. The son of Nobel laureate and chemist, Alan J. Heeger, David Heeger is a contemporary neuroscientist who has been at the forefront in the field of functional magnetic resonance imaging (fMRI). Dr. Heeger was the first to bring together two separate and largely unrelated disciplines—cognitive neuroscience and film studies, opening the way for the exciting new interdisciplinary field of “Neurocinematic” Studies.

About the author (from his site):

This is the first in a series of blogs by Barry Sandrew, Ph.D., founder and CCO/CTO of Legend3D. Sandrew received his doctorate in neuroscience from State University of New York at Stony Brook in 1977. After winning a two year NIH Fellowship at Columbia University, College of Physicians and Surgeons, he joined the staff at Mass General Hospital and Harvard Medical School as staff Neuroscientist. In 1987, he left his academic and scientific career to found American Film Technologies, and invent the first all-digital colorization process for colorizing black and white feature films. He took that company public while converting hundreds of movies for clients such as Turner, Disney, Warner Bros., Fox, Gaumont, TF1, ABC, and CBS, among many others. Sandrew left colorization to start two new companies, one of which included Lightspan, one of the largest educational software companies in the world. That company went public and was later acquired by Plato Learning. In 2000, Barry then founded Legend Films and began the process of re-inventing colorization and developing a proprietary restoration process. Over the course of the next seven years, Legend Films converted approximately 145 black and white titles to color as well as several TV series. In 2006, he recognized that the future of entertainment would be 3D and, leveraging his proprietary colorization pipeline, he modified the entire R&D direction and business model to embrace 2D-to-3D conversion, changing the name of the company to Legend3D. Today, as the most innovative company in the conversion industry, Legend3D continues to lead the field, delivering the highest quality product with the fastest turn around time, at the most competitive pricing.

Posted by Barry B. Sandrew Ph.D.               Wednesday, February 1, 2012

Copied from his Blog BSandrew.blogspot.com, with permission

Engaged in 2D and Immersed in 3D

This month, the multiple 3D films being recognized by the 84th Academy Awards cement the fact that 3D is here to stay. On the heels of the Best Visual Effects nominations for Hugo and Transformers: Dark of the Moon, Legend3D Founder, President, CCO/CTO Barry Sandrew dissects the appeal of 3D for the moviegoer in this two-part blog series on the neuroscience of 2D and 3D feature films. Relying on our current understanding of stereopsis, as described in scientific literature, Sandrew contemplates how our brains differentially perceive images and activate emotional responses when viewing a 2D movie versus a 3D movie.

The group dynamics of the 2D experience

When we attend a 2D movie, the group dynamics of the audience come into play as we direct our collective attention as observers to the images on the screen. Our personal space is infringed upon only tangentially by the other audience members that are sitting on either side of us, as well as by those behind and in front of us. The tangential influence upon us of the surrounding audience is a reminder that we are in a theater with strangers, all watching images being projected on the screen. However, in a 2D movie, our fellow audience members typically do not detract from the experience.  In fact, the audience in a 2D movie is intended to be an integral part of the total feature film experience and has been presented as one of the selling points of attending a 2D move in a theater rather than watching it on DVD or Blu-ray at home. As we become engaged in the movie, we can introspectively sense surprise, anger, sadness and happiness based on the perceived action on the screen and to a lesser degree, through the reactions of the audience members around us. We can empathize and sympathize with the characters being portrayed and we can experience anxiety and fear as we follow the storyline.

Disparity—A key element in the perception of stereo

Our brains receive 2D information in a theater as images presented sequentially at 24 frames per second. In a 2D theater experience, motion is perceived as changes in the up, down and sideways position of one object in relation to other 2D objects.  We perceive three dimensions in a 2D movie via single eye or monocular cues such as occlusion, motion, shading, size, parallax, texture gradient, perspective, saturation and brightness. However, lacking in a 2D movie is the most powerful 3D influence on the brain—disparity. Due to the separation of our eyes and the distance between them, we see a slightly different image from each eye.  This separation causes a horizontal displacement between the images, which is referred to as disparity. The best way to test this is to put your thumb in front of your eyes and alternately close one eye and then the other. You will see two separate perspectives of your thumb displaced horizontally and your thumb will appear to jump from side to side. When these separate images from our left and right eye reach our visual cortex, they are directed to highly specialized neurons that are tuned to critical parameters of disparity.  As a consequence, the two images are fused into one uniquely different image that exhibits the appearance of depth and volume. The result is 3D perception.

A lack of disparity contradicts monocular cues

In a 2D movie, each of our eyes receives precisely identical images from the movie screen with no horizontal displacement. When the two sets of identical images reach the visual cortex of our brain, they are directed primarily to neurons tuned to zero disparity. There, they are “fused” together into a single image that is exactly identical to each of the two images that originally comprised it. Other than the existence of monocular stereo cues, mentioned above, the audience does not perceive depth and volume due to the absence of disparity. This lack of disparity in a 2D movie creates a cue conflict situation that prevents us from being fully immersed in the story in the same manner as a 3D movie. The monocular cues in a 2D movie are telling us there is depth, but the lack of disparity contradicts the monocular cues—forcing the brain to try and reconcile the conflict.

The absence of disparity “pushes” the screen away from us

For some monocular cues, like motion and occlusion, there is a further complication that acts to distance us from the movie screen. As mentioned above, in a 2D movie the relative motion of objects on the screen (one 2D object moving relative to and possibly occluding another 2D object) can indicate depth in the absence of disparity. However, in the real world when there is this kind of relative motion in the absence of disparity, our brains normally interpret that information as indicating that the objects are far away. That’s because disparity in human vision drops off significantly at distances of several hundred yards.  You can prove this if you try the thumb trick above, alternately closing one eye and then the other, but this time, try the trick looking at a tree that is several hundred yards away.  You’ll see that the tree will not appear to jump from side to side like your thumb did. This is because the perspectives from our two eyes at that distance are identical. Our eyes do not converge on the tree, but rather are set to infinity.  The father away things are from us the less horizontal displacement we experience.  So our brain “comes up with” a solution in which we are literally “distanced” from the action on the screen. This distancing and lack of disparity in a 2D movie renders the spatial areas in front of and behind the theater screen irrelevant to the story and therefore irrelevant to each member of the audience. The storytelling, in its entirety, happens on the white screen at the end of the theater.

Skillful use of disparity in a 3D movie will always enhance the experience

In spite of the contradictions our brain must reconcile, a 2D movie is almost always comfortable to watch and with proper directing, cinematography and storytelling, it can evoke very strong emotions. Indeed, many of us remember the outpouring of emotions while watching the sinking of the Titanic or the building terror of T-Rex in Jurassic Park, as both movie experiences were presented in 2D. However, the influence of disparity on the audience in a 3D movie cannot be trivialized as a fad or unnecessary. Whether a 3D movie is captured with two cameras or converted from 2D-to-3D, disparity that is properly and skillfully stereographed will always enhance the storytelling experience by significantly amplifying the responses of disparity-tuned neurons and therefore more closely simulating reality.

3D a higher resolution medium

In the visual system, the absolute number of neurons activated in the brain is not as significant as the ratio of activation between subpopulations of neurons responding to zero disparity and those responding to a wide spectrum of tuned disparities, including zero.  This is analogous to an audio amplifier where the ratio between subpopulations of audio frequencies can vary and increasing the gain on the system increases the overall fidelity of the audio experience, meaning it becomes closer to the original source in resolution.  In the same manner, the ratio of activated zero disparity and disparity-tuned subpopulations of neurons in a 3D movie can vary, but it’s the intensity of the relative responses of those subpopulations that create a more accurate and higher resolution stereo image.  It appears that our brains interpret this higher resolution information as more closely simulating reality (“fidelity”) and therefore it immerses us in the movie to a greater degree, creating a heightened emotional investment in the story.  This is, of course, an oversimplification of an exquisitely complex process that has evolved from the earliest primates. However, the theater going experience is something we can all relate to and many of us recognize the profound difference between being engaged in a 2D movie versus being immersed in a 3D movie.

3D movies are a uniquely personal experience

Dynamic disparity (i.e. the relative amount of disparity, changes in disparity over time and the rate of disparity change) transforms the movie screen from a projection screen into a window that has both an interior and an exterior. Consequently, the storytelling environment is projected throughout the entire theater, actively becoming a unique part of the personal space of each member of the audience and resulting in the space both in front of and behind the screen becoming an integral part of the story. For example, an object that might fly out of the screen at us, traveling through what is called negative parallax, becomes very personal for each member of the audience because it affects and/or “intrudes” each audience members’ personal space equally.  Surround sound has an analogous effect in that we are “in it” rather than simply listening to stereo audio in front of us.  As a result, unlike the 2D movie experience, a 3D movie is more personal and the group dynamics of the larger audience are no longer in play to the same extent.  In fact, it’s my opinion that 3D glasses actually have the positive effect of helping to separate us from the other audience members, containing the experience as a more personal one for each of us, thereby potentially amplifying the immersive nature of the experience.

The Bottom Line

When we are engaged in a 2D movie, we are doing so as passive observers, watching the story take place “over there”, on the screen in front of us. However, when we are immersed in a 3D movie, we are doing so as active participants and the action can be happening behind and in front of us—as far as infinity. This is where the central difference lies. We cannot discount the uniquely active physical and emotional reactions that we have in the 3D feature film experience which are not experienced in our more passive reception of 2D movies. I believe, this is one of the reasons why we have seen 3D continue to flourish and be adopted so enthusiastically by both national and international movie exhibitors, as well as by the entertainment and consumer electronics industries at large. Consumers and moviegoers are hungry for visuals that offer a significantly greater sense of engagement and for visuals that spark an immersive, emotional response that has the ability to transport them into the heart of the action.  This year, we will have the opportunity to test the differential 2D and 3D theatrical experiences with three iconic movies that were originally released in 2D, being released in 3D. Top Gun has been converted by Legend3D and will be re-released on its 25th anniversary by Paramount on the heels of 3D converted re-releases of Titanic and Star Wars.  I am confident that all three films will be very successful at the box office, as each was already a proven success in 2D. Depending on how skillfully each of these films was converted, now in 3D they will give those who saw them originally in 2D an immersive and refreshingly unique experience; an experience that will hopefully help to solidify 3D as an essential part of filmmaking.

Next, we will continue to look at the neuroscience of 3D movies and further explore the uniquely tuned structures and neurons in the brain that respond selectively to dynamic disparity. We’ll look at concepts of visual processing that remain contentious within the scientific community, involving differential pathways within the brain that signal separately “where something is” and “what something is.” The evolutionary survival value of binocular vision will be discussed in the context of 3D movies and we will touch upon more primitive structures in the brain that are likely triggered by disparity to elicit powerful physical emotions and ‘flight’ or ‘fight’ reactions. For more detailed information and technical reviews on stereopsis, as well as in-depth information on the uniquely tuned disparity neurons in many parts of the brain, please see the following references:

(1) Cumming BG, De Angeles GC. (2001). The Physiology of Stereopsis. Annual Review of Neuroscience. 24, 203-238.

(2) Howard, IP, Rogers, BJ. (1995). Binocular Vision and Stereopsis.  New York: Oxford University Press.

(3) Poggio, GF, Poggio, T. (1984). The Analysis of Stereopsis.  Annual Review of Neuroscience. 7, 379-412.

(4) Born, R., Bradley, D. (2005). Structure and Function of Visual Area MT. Annual Review of Neuroscience. 28, 157-189.

I wish to acknowledge the review and insightful suggestions of my neuroscience colleague, David Heeger, Ph.D., Professor at New York University (http://www.cns.nyu.edu/~david/) where he is a member of the Center for Brain Imaging. The son of Nobel laureate and chemist, Alan J. Heeger, David Heeger is a contemporary neuroscientist who has been at the forefront in the field of functional magnetic resonance imaging (fMRI). Dr. Heeger was the first to bring together two separate and largely unrelated disciplines—cognitive neuroscience and film studies, opening the way for the exciting new interdisciplinary field of “Neurocinematic” Studies.

About the author (from his site):

This is the first in a series of blogs by Barry Sandrew, Ph.D., founder and CCO/CTO of Legend3D. Sandrew received his doctorate in neuroscience from State University of New York at Stony Brook in 1977. After winning a two year NIH Fellowship at Columbia University, College of Physicians and Surgeons, he joined the staff at Mass General Hospital and Harvard Medical School as staff Neuroscientist. In 1987, he left his academic and scientific career to found American Film Technologies, and invent the first all-digital colorization process for colorizing black and white feature films. He took that company public while converting hundreds of movies for clients such as Turner, Disney, Warner Bros., Fox, Gaumont, TF1, ABC, and CBS, among many others. Sandrew left colorization to start two new companies, one of which included Lightspan, one of the largest educational software companies in the world. That company went public and was later acquired by Plato Learning. In 2000, Barry then founded Legend Films and began the process of re-inventing colorization and developing a proprietary restoration process. Over the course of the next seven years, Legend Films converted approximately 145 black and white titles to color as well as several TV series. In 2006, he recognized that the future of entertainment would be 3D and, leveraging his proprietary colorization pipeline, he modified the entire R&D direction and business model to embrace 2D-to-3D conversion, changing the name of the company to Legend3D. Today, as the most innovative company in the conversion industry, Legend3D continues to lead the field, delivering the highest quality product with the fastest turn around time, at the most competitive pricing.