The words 3D and "hologram" can be heard from everyone's ears. Since Matsushita released its first 3D TV system in 2010 and now its virtual reality and augmented reality technology, these words have been integrated into our popular culture and have increasingly become the focus of our attention. After all, the real world is three-dimensional. Why should we limit our experience to a flat screen?
The transition from 2D to 3D is a natural process. Just as in the 1950s, black-and-white movies and black-and-white TVs are transformed into colors. But in many ways, the impact from 2D to 3D may be even greater.
The significance of 3D is not just to present a more credible real world, but to turn the digital world into a part of real life so that every detail of it becomes as real as the real world.
This upcoming watershed will have a profound impact on our work, study, social and entertainment methods. We will be able to break through the physical limitations of the real world, how big our imagination is, and how big the world is.
Of course, this change will also affect the equipment we use and how we interact with the machine. This is why companies such as Google, Facebook and Apple are all occupying the 3D market as soon as possible: Whoever wins the 3D war will be able to control the next generation of user interactions.
But now it's just an embarrassment. Although people have already tried to open the 3D market before, we still have to rely on a narrow screen to enter the electronic world. Why? Because there are many shortcomings in the current 3D, the current level of technology is not enough to achieve a truly credible world, so consumers are still waiting.
Next Xiaobian wants to give you a comprehensive overview of 3D technologies from the perspective of the user, the challenges facing 3D products such as VR/AR heads-up and naked-eye 3D displays, and how these products will become intuitive interactions between us and the digital world. interface.
Why do we need 3D graphics?
Before we can deeply understand 3D technology, we need to understand the principle of this technology.
You can try to wear a pair of glasses and you will find it more difficult than you think. Evolution has changed human perception of 3D, allowing us to grasp real-world information faster and more accurately.
How do we acquire depth perception at the physiological level? This is a complex subject. Human eyes and brain can feel a lot of depth hints. When these hints appear in the real world, they will reinforce each other and form a clear 3D environment image in our minds.
If we want to use a 3D illusion to deceive the visual system and the brain, we may not be able to faithfully reproduce all the depth hints, and the resulting 3D experience will be biased. Therefore, we need to understand some of the major depth hints.
Stereoscopic vision (binocular disparity)
Whether you believe it or not, what your eyes see is different. The angles at which they see objects are slightly different, so the images that the retina gets are also different.
When we look at distant objects, the parallax between the left and right eyes will be smaller. But when the object is closer, the parallax increases. This difference allows the brain to measure and "feel" the distance to the object, triggering the perception of depth information.
Most of today's 3D technologies rely on stereo vision to deceive the brain and make the brain believe that they feel the depth of information. These technologies present different images to both eyes. If you have seen 3D movies (such as "Avatar" and "Star Wars"), you should have experienced this effect through 3D glasses.
The more advanced "autostereoscopic" display technology can project different images into space in different directions so that the eyes can receive different images without wearing glasses.
Stereoscopic vision is the most obvious depth hint, but it is not the only clue. Humans can actually feel the depth through only one eye.
Motion parallax
Close one eye and place your index finger in front of the other eye to keep it still. Now move your head up and down slightly. At this point you will see that the background seems to be relative to your finger. More precisely, it looks like your finger is moving faster than the background.
This phenomenon is called motion parallax. This is why you can use one eye to feel the depth. If you want to provide real 3D feelings, motion parallax is a very important effect, because the relative displacement of the audience and the screen is very slight.
Stereoscopic vision without motion parallax still allows you to feel depth, but this 3D image will appear distorted. For example, buildings in a 3D map will begin to appear distorted. The background objects appear to be deliberately obscured by objects in the foreground. If you look closely, it will feel quite uncomfortable.
Parallax in motion is closely related to our visual perception - in fact the small effect we see is a depth hint.
Visual adjustment
Even if you are sitting completely still (no motion parallax), then close one eye (no stereo vision) so that you can still distinguish distant and close objects.
Then try the above finger experiment, hold your index finger still in front of your eyes, and then concentrate on this finger. As the glasses focus, you will find that the background will be blurred. Now put your focus on the background, you will find the fingers become blurred, and the background becomes clear. This is the same as the working principle of modern cameras. Our eyes have the ability to change the focus.
The eye changes the shape of the lens through the contraction of the ciliary muscles, thereby achieving the zoom function.
So how does the ciliary muscle of the eye know how much effort it should make to contract? The answer is that our brain has a feedback loop, and the ciliary muscle contracts and relaxes continuously until the brain gets the clearest image. This is actually an instantaneously completed action, but if we adjust the ciliary muscle too frequently, our eyes will feel very tired.
Only within two meters of the object can trigger the ciliary muscle movement, beyond which the glasses will begin to relax and focus on infinity.
The conflict between visual convergence and visual regulation
When your eyes focus on a nearby point, they actually rotate in the eyelids. When you focus on the line of sight, the extraocular muscles stretch automatically and the brain can feel this movement and see it as a depth hint. If you focus on an object within 10 meters, you will feel the convergence of the eyeballs.
So when we look at the world with two eyes we use two different muscle groups. A group of muscles is responsible for the convergence of the eyes (the convergence) to the same focal point, while the other group of muscles is responsible for the adjustment of the retina imaging clarity. If the eyes are miserly, we will see double images. If the visual adjustment is inappropriate, then we will see blurred images.
In the real world, visual convergence and visual regulation are complementary. In fact, the nerves that trigger these two reactions are connected.
However, when watching 3D, virtual reality, or augmented reality content, we usually focus on a specific location (such as the 3D cinema screen), but the information received by the eyes will allow the eyes to converge to another distance (for example, from The dragon rushed out of the screen).
At this time, our brains will have difficulty coordinating the signals of these two conflicts. This is why some viewers may feel tired or even nauseous when watching 3D movies.
Use special glasses to watch 3D content
Do you remember those red and blue stereo glasses made of cardboard? This "filter glasses" allows the left eye to see only red light, and the right eye only to see blue light. A typical stereoscopic image will be superimposed with a red image on the left eye and a blue image on the right eye.
When we look through the filter glasses, the brain's visual cortex confuses what we see as a three-dimensional scene, and the color of the picture is also corrected.
All 3D effects based on glasses now, including virtual reality and enhanced heads, use this principle of work - physically separating the images seen by the left and right eyes to create stereoscopic vision.
Polarized 3D glasses (for cinemas)
When you go to a movie, you get a pair of 3D glasses that look completely different than polarized glasses. The glasses do not filter the image according to the color but filter the image according to the polarization of the light.
We can think of photons as a vibrating entity that vibrates horizontally or vertically.
On a movie screen, a specially crafted projector generates two overlapping images. One image only emits horizontally vibrating photons to the viewer, while the other image sends vertically vibrated photons. The polarized lenses of these glasses ensure that the two images can reach the corresponding eyes.
If you have polarized glasses (not to be stolen from the cinema), you can use it to watch the polarization of the real world. Hold the pair of glasses at a distance of 2 feet from your line of sight, and then use it to watch the car's windshield or water. As you rotate the glasses 90 degrees, you should see the appearance and disappearance of glare across the lens. This is how polarizers work.
The polarizer allows all color chromatograms to enter the eye, and the quality of this 3D picture will also be enhanced.
From the perspective of 3D image quality, this kind of glasses can only provide stereoscopic depth hints. So although you can see the depth of the image, if you leave the seat and move around in the cinema, you will not be able to see the surrounding objects, the background will move in the opposite direction to the motion parallax, that is, it will follow you. .
A more serious problem is the lack of visual support for such glasses, which can lead to conflicts between visual convergence and regulation. If you stare at a dragon approaching you from the screen, you will soon feel extreme visual discomfort.
This is why the dragon in the movie only flies very fast - this is only to achieve the effect of scaring, while avoiding your eyes feel uncomfortable.
Active Shutter 3D Glasses (for TV)
If you have a 3D TV in your home, the glasses it supports may not be of the polarized type but use the "active shutter" technology. This type of television alternately displays the images of the right eye and the left eye, and the glasses will simultaneously cover the corresponding eyes according to the frequency of the television. If the frequency of switching is fast enough, the brain can mix the two signals into a single, coherent 3D image.
Why don't you use polarized glasses? Although some TVs use this technique, polarized images in different directions must come from different pixels, so the viewer’s image resolution will be halved. (But the two images in the movie theater are projected on the screen separately, and they can overlap each other, so it will not cause the resolution to drop.)
Active shutter 3D glasses generally only support stereoscopic depth hints. Some of the more advanced systems will use tracking technology to adjust the content, support the motion parallax support by tracking the viewer's head, but only for one viewer.
Virtual Reality Header
Virtual reality is a new type of 3D rendering. This technology has recently received widespread attention. Now that many people have experienced virtual reality headlines, more people will be able to experience this technology in the coming years. So, what is the working principle of virtual reality?
Instead of working by filtering content from external screens, VR Headset generates its own binocular image and presents it directly to the corresponding eye. VR heads usually include two micro-displays (one for the left eye and one for the right eye), and their actual heads are magnified and adjusted by the optics and displayed at a specific position in front of the user's eyes.
If the resolution of the monitor is high enough, the magnification of the image will be higher, so that the user will see a wider field of view and the immersive experience will be better.
Now virtual reality systems like the Oculus Rift will track the user's position and add parallax to the motion of the stereo.
The current version of the virtual reality system does not support visual adjustments and is prone to conflicts of visual convergence and adjustment. In order to ensure the user experience, this technical problem must be solved. These systems can not fully solve the problem of eyeglasses (limited range and distortion), but this problem may be solved by eye tracking technology in the future.
Augmented Reality Header
Similarly, augmented reality heads-up has become more and more common. Augmented reality technology can merge the digital world with the real world.
In order to ensure realism, the augmented reality system not only needs to track the head movement of the user in the real world, but also consider the real 3D environment in which he is.
If the recent report by Hololens and Magic Leap is true, there has been a huge leap in the field of augmented reality.
Near-field display
Just as our two eyes see the world differently, the light of the real world also enters the pupil from different directions. In order to trigger visual adjustment, the near-eye display must be able to simulate the light emitted independently from all directions. This optical signal is called the light field, which is the key to the future of virtual reality and augmented reality products.
Until this technology is realized, we can only continue to suffer headaches.
These are the challenges that head-end devices need to overcome, and the other problem they face is integration into society and culture (such as the experience of Google Glass). If we want to display the world in multiple perspectives at the same time, we need to use the naked eye 3D display system.
Naked eye 3D screen
If we do not wear any equipment, how can we experience 3D images?
Seeing here, I think you should understand that if you want to provide stereoscopic vision, 3D screens must project different visual angles in different directions. In this way, the viewer's left and right eyes will naturally see different images, triggering depth perception, so this system is called "autostereoscopic display."
Because the 3D image is projected on the screen, the autostereoscopic display itself can support visual convergence and visual adjustment as long as the 3D effect is not too exaggerated.
However, this does not mean that this system will not cause eye discomfort. In fact, this system has another problem in the conversion of different visual areas.
This kind of system's image will have frequent jumping, brightness transformation, dark belt, stereo vision interruption and other issues. The worst is that the content seen by the left and right eyes is reversed, resulting in a completely opposite 3D feeling.
So how do we solve these problems?
Prism film
3M began commercial production of this technology in 2009.
The prismatic film is inserted into the thin film stack of the backlit LCD screen and can be illuminated by light coming from one direction or the other. When the light source comes from the left, the LCD image will be projected onto the right area. When the light source comes from the right, the image will be projected in the left area.
Quickly switch the direction of the light source, and change the left and right eyes to see the content on the LCD at the same frequency, which can produce stereo vision, but the premise is that the screen needs to be placed in front of the viewer. If viewed from other perspectives, the three-dimensional effect will disappear and the image will change to a plane.
Because the scope of viewing is very limited, this display technology is known as "2.5D."
Parallax barrier
On a 2D screen, a mask with many small openings was added. When we look at the screen through these small mouths, we cannot see all the pixels below. The pixels we can see actually depend on the viewing angle, and the viewer's left and right eyes may see different groups of pixels.
The concept of "parallax barrier" was discovered as early as a century ago. Sharp first put it into commercial applications ten years ago.
Improved, this technology now uses a switchable barrier, which is actually another active screen layer, it can produce a barrier effect, it can also become transparent, restore the screen to 2D mode to display the full resolution .
As early as 2011, the HTC EVO 3D and LG Optimus 3D were already on the headlines because they were the world's first smartphones to support 3D capabilities. But they are just another example of 2.5D technology and can only provide 3D effects in a very narrow range of viewing angles.
Technically speaking, the parallax barrier can be continuously expanded to form a wider perspective. But the problem is that the wider the angle of view, the more light you need to shield, which will cause excessive power consumption, especially for mobile devices.
Lenticular lens
Covering a 2D screen with a layer of miniature convex lenses, we all know that the convex lens can focus on parallel light from a distant light source. The child also uses this principle by using a magnifying glass to point at things.
These lenses can collect the light emitted from the screen pixels and convert it into a directional light beam. We call this phenomenon collimation.
The direction of the beam changes with the position of the pixel below the lens. In this case, different pixels will follow the beam in different directions. The lenticular lens technology can finally achieve the same effect as the parallax barrier (all using the principle of different pixel groups seen in different spatial positions), except that the lenticular lens does not block any light.
So why don't we see the lenticular 3D screen on the market?
This is not because nobody tried. Toshiba released its first-generation system in Japan in 2011. However, this kind of screen has some unacceptable visual illusions when it is carefully watched, which is mainly caused by the lens.
First, the screen pixels are usually composed of a smaller emission area and a larger "black matrix", the latter not emitting light. After processing by the lens, the emission area of ​​the single pixel and the black matrix are deflected to different directions in space. This will cause the 3D picture to appear very dark areas. The only way to solve this problem is to "defocus" the lens, but doing so will cause interference between different viewing angles and the image will become blurred.
Second, it is difficult to achieve proper collimation at wide viewing angles with only a single lens. This is why camera lenses and microscopes use composite lenses instead of single lenses. Therefore, the lenticular lens system can only observe true motion parallax at a narrow viewing angle (about 20 degrees). Beyond this range, the 3D image will continue to repeat, feeling like the viewing angle is not the same, the image will become more and more blurred.
Narrow field of view and poor visual conversion are the biggest drawbacks of lenticular screens. For television systems, if the audience automatically adjusts their head and does not walk around, the current lenticular lens technology is acceptable.
However, in the usage scenarios such as mobile phones and cars, the head is sure to move, and it is difficult for the lenticular lens system to achieve the desired effect.
So how do we design a naked-eye 3D vision system with a wide field of view and smooth transitions?
Eye tracking
If I know the relative position between your eyes and the screen, I can calculate the corresponding perspective and try to adjust the image to the direction of the eyes. As long as I can quickly detect the position of your eyes and have the same rapid image adjustment mechanism, I can ensure that you can see stereo vision and smooth motion parallax rendering from any angle of view.
This is how the eye tracking autostereoscopic screen works.
The advantage of this method is that the screen only needs to render two views at any time, which keeps most of the screen pixels. From a practical point of view, the eye tracking system can be used in conjunction with current parallax barrier technology to avoid the optical artifacts produced by the lenticular lens system.
However, eye tracking is not a panacea. First of all, it can only support one viewer at a time, and eye tracking requires the device to deploy additional cameras and continuous running of complex software for predicting eye position in the background.
For television systems, size and power consumption are not a big problem, but it will have a huge impact on mobile devices.
In addition, even the best eye tracking system can experience delays or errors. Common causes include changing lights, eyes being blocked by hair or glasses, the camera detecting another pair of eyes, or the head of the audience moving too fast. . When this system fails, viewers will see very unpleasant visual effects.
Diffractive backlight
The latest development in naked-eye 3D technology is to diffract "multi-view" backlit LCD screens. Diffraction is a characteristic of light, and light deflects when it encounters submicron objects, so this means that we are ready to enter the field of nanotechnology.
You read it right, it is nanotechnology.
Ordinary LCD screen backlights emit randomly distributed light, which means that each LCD pixel emits light into a wide space. However, a diffractive backlight can emit light in a uniform direction (light field), and one LCD pixel can be set to emit unidirectional light. In this way, different pixels can send their own signals in different directions.
Like the lenticular lens, the diffractive backlight method can also make full use of incident light. But unlike lenticular lenses, diffractive backlights can handle both smaller and larger ray emission angles, as well as full angular spread at each viewing angle. If the design is reasonable, dark areas will not be generated between the viewing angles, and the generated 3D image can be as clear as the parallax barrier.
Another feature of the diffraction method is that the light adjustment function does not affect the light that passes directly through the screen.
As a result, the transparency of the screen can be completely preserved, so this kind of screen can be added below to add normal backlight and restore to full-pixel 2D mode. This paved the way for the development of perspective naked-eye 3D display technology.
The main problem with the diffraction method is the color consistency. Diffraction structures typically emit different colors of light in different directions, and this dispersion requires cancellation at the system level.
The latest frontier: from 3D vision to 3D interaction
However, the development of 3D technology will not only stop seeing 3D images, it will open up a completely new user interaction paradigm.
The 3D experience will soon be able to meet the quality of consumerism, and we will not be worried about our 3D experience being interrupted.
In virtual reality and enhanced display systems, this means increasing the field of view, supporting visual adjustments, and increasing the speed of response of the system when the head moves quickly.
In the naked eye 3D display technology, this means that the user is provided with enough free movement space and avoids various visual artifacts such as 3D loss, dark area, visual jump, or motion parallax delay.
Mixed world
Once the illusions become true and credible, we forget about the technology behind the illusions and treat the virtual reality world as a real world, at least until we hit a real existing wall. If we want to realize this illusion, we need to consider that the physical world has a physical reaction.
When we convert electronic data into light signals that we can perceive in the real world, we need to send the physical data of the body back to the digital world for interaction.
In virtual reality and augmented reality heads-up, this can be achieved by sensors and cameras worn by the user or placed in the surroundings. We can foresee the emergence of smart clothing equipped with sensors in the future, but they may be more cumbersome.
On a 3D screen, this task can be done directly by the screen. In fact, the Hover Touch technology developed by Synaptics has been able to achieve non-touch finger sensing. Soon, we only need to move our fingers to interact with the holographic images in mid-air.
Touchable hologram
Once the electronic world understands the user's reaction mechanism. Then the two will be more naturally integrated. In other words, digital signals can open a door above us before we hit a wall.
But isn't it better if we can see the wall in the virtual world? But how can we bring this tactile into the world?
This question involves the field of tactile feedback. If you have turned on the vibration mode of the phone, then you should experience this kind of feedback.
For example, a glove with a vibrating unit and other clothing, or use a weak electrical signal to simulate the skin, if adjusted, your different parts of the body will be able to feel the touch with the visual effect.
Of course, not everyone is suitable to wear clothing or electric currents that are covered with wires.
For screen-based devices, ultrasonic tactile sensations allow you to touch the screen directly in the air without using any smart clothing. This technique sends out ultrasonic waves from the periphery of the screen. The intensity of these ultrasonic waves can be adjusted according to the user's finger movements.
Believe it or not, the intensity of these enhanced sound waves is enough to make your skin sense. Companies like Ultrahaptics are already preparing to bring this technology to market.
Holographic reality
Although today's virtual reality and enhanced display heads are becoming more common, they still have many limitations in terms of mobility and socialization, making it difficult for them to implement a fully interactive 3D experience. A 3D screen using finger tactile technology will overcome this obstacle and allow us to interact with the digital world in a more direct way in the future.
In a recent blog post, I referred to this platform as Holographic Reality and described how holographic reality displays can be applied to all aspects of our daily lives. All windows, tables, walls and doors in the future will have holographic realities. The communication devices we use in offices, homes, cars and even public places carry holographic components. We will be able to access the virtual world anytime, anywhere, eliminating the need to wear head-mounted displays or connecting cables.
The next 5 years
In the past five years, great progress has been made in the areas of 3D technology, display technology and heads-up display.
With such rapid technological development, it is not hard to imagine that in the next five years we will communicate, learn, work, shop or entertain in a panoramic reality world. This will be achieved through advanced 3D with head-up display and 3D screens and other products. Ecosystem to achieve.