A big challenge for VR headsets is that when we are wearing the headset, our eyes are located very close to the display and focused through a pair of lenses. This makes individual pixels distinguishable even with today’s high-end VR headsets. Insufficient resolution reduces the immersive experience and makes for example reading small text problematic.
According to AMD’s Radeon Technologies Group, in order for VR to reach true immersion that one will not be able to tell apart from real world, 16K screens (probably 16K per eye) and 240fps are required [TT]. Elsewhere, Palmer Luckey, the founder of Oculus, has said that to get to the point where the user cannot see pixels, about 8K per eye would be needed. 16K is 15360x8640 pixels, whereas 8K is 7680x4320.
At least Sony has already released a smartphone, Xperia Z5 Premium, that has a 4K display [TR1]. Also Samsung has showed off a 4K smartphone display. Furthermore, Samsung is working on a new type of display that will have 11K resolution with 2250 pixels per inch [PL]. 11K could mean something like 10560x5940 pixels. Samsung expects to have a prototype ready by 2018. The screen could be available in smartphones in 2019.
But rather than resolution, a more suitable notion for VR environments is pixels per degree. If the human eye was a digital camera, it would have 60 pixels/degree at the fovea, which is the part of the retina where the visual acuity is highest [Sensics]. For VR goggles, the pixel density can be calculated by dividing the number of pixels in a horizontal display line by the horizontal FOV (Field of View) provided by the headset. HTC Vive has 1080 horizontal pixels per eye (i.e., 2160 in total) and a FOV of 110 degrees, resulting in a pixel density of 19.6 pixels/degree. This is still pretty far from 60 pixels/degree. For human eyes, the combined horizontal visual field is 180-200 degrees [WP1, WP2]. Thus, VR headset offering a wide 200 degree FOV with 60 pixels/degree might require a horizontal resolution of 6000 pixels. This means that Samsung’s coming 11K display might actually be able to reach 60 pixels/degree, which feels quite promising.
A wide FOV is crucial for an immersive VR experience by avoiding the tunnel vision impact and even allowing for peripheral vision and good sense of the user’s visual surroundings in the virtual environment [TV]. Some examples of FOVs of different VR headsets are as follows [VT]:
StarVR, which is still under development, has the highest FOV from among the VR headsets [VT]. 210 degrees offers a FOV beyond the human eye’s peripheral vision. According to some early reviews, the 210-degree FOV is an eye-opener in the virtual world and makes other headsets (e.g., the Rift and the Vive with their 110-degree FOV) feel like binoculars [RV]. Achieving a wide FOV requires a complex set of lenses. StarVR combines normal and Fresnel elements which, as a downside, brings challenges to clarity. StarVR has a per-eye resolution of 2560x1440 but the extra pixels (compared to competitors) are obviously stretched over the wider FOV.
Besides the horizontal FOV, also the vertical FOV has an impact on the VR experience, allowing the eyes to dart up and down more naturally without encountering the dark edges of the headset [TV]. The HTC Vive has a better vertical FOV than the Oculus Rift since it uses custom screens that are oriented vertically.
But what would be the ideal FOV for a VR headset? Even the VR experience using a VR headset with a 180-degree FOV will not not match how we experience the physical world. This is because the human eyes can see up to a 270-degree FOV if the eyes are fully rotated [VT]. Thus, even with a 180-degree FOV, one might still experience a tunnel vision effect (which is probably the reason StarVR went for 210 degrees).
An additional aspect to consider is that lower FOVs help with motion sickness [WP3, WS]. Decreasing the FOV results in less visual cues and the brain does not register as wide a discrepancy between sensory input from the eyes and the vestibular system in the inner ear (this mismatch in VR causes motion sickness). The challenge is naturally that reducing the FOV makes the VR experience less immersive by strengthening the tunnel vision effect. Fortunately, research is ongoing to solve this issue. As an example, researchers from Columbia University have developed an approach that involves masking the user’s view to minimize motion sickness symptoms [NA]. The approach adjusts the visual range on the fly to reduce motion sickness. When the player is in motion in the virtual environment, the system partially obscures each eye with a soft-edged, circular, virtual cutout, reducing the FOV to minimize motion sickness. When there is less action, the FOV is increased again.
Increased resolution and FOV or pixels per degree require of course more GPU power. Since the Sony Xperia Z5 Premium smartphone already comes with a 4K display, one could assume that someone packing two of those displays into a VR headset should not be that far off. Assuming that such a headset would have at least the same 90Hz refresh rate as the Rift and the Vive, we would need a GPU capable of running 4K@90fps on both of the displays, or one GPU for each of the displays. Nvidia’s latest Titan X desktop GPU, which is supposed to be the most powerful consumer graphics card on the market [TR2], has demonstrated performance of 4K@81fps (and should, according to the specs, be capable of 8K@60Hz, that is, 7680x4320@60Hz at the maximum) [NV]. Therefore, it would seem to be possible to run 2x4K@80Hz with two Titan X GPUs. The only problem is that two Titan X GPUs cost $2400.
So, to the conclusion. 4K smartphone displays are already here. 11K displays might be coming to smartphones in 2019. And today’s top consumer graphics cards can already do 8K@60fps. Thus, by the time Samsung’s 11K displays are out, it does not seem unrealistic to expect that we could have consumer graphics cards that can deliver 11K@120fps. So perhaps by 2019, we could reach an over 200 degree FOV with 60 pixels/degree and 120fps, that is, have VR that we cannot tell apart from the real world. However, it feels that 16K@240fps, which is AMD’s goal, is still further off to the future.
However, VR hardware makers are looking into ways to reduce the required processing power. One phenomenon that can come to help is foveal vision - when we look at an object in distance, the focal point of our gaze is in focus, while the scene around the object is blurred. In VR, this principle can be used to render fully only the specific area where the user is looking and leaving the rest of the scene at a far lower resolution [MT]. This has the potential to bring large performance gains and reduce the load of the GPU. At least the Fove VR headset and Nvidia are applying this trick to VR.
[AT] Virtual Perfection: Why 8K resolution per eye isn’t enough for perfect VR, http://arstechnica.com/gaming/2013/09/virtual-perfection-why-8k-resolution-per-eye-isnt-enough-for-perfect-vr/
[MT] Nvidia’s Eye-Tracking Tech Could Revolutionize Virtual Reality, https://www.technologyreview.com/s/601941/nvidias-eye-tracking-tech-could-revolutionize-virtual-reality/
[NA] Restricting field of view to reduce motion sickness in VR, http://newatlas.com/columbia-university-vr-motion-sickness/43855/
[NV] Nvidia Titan X, http://www.geforce.com/hardware/10series/titan-x-pascal
[PL] Forget about 4K and even 8K, Samsung is making 11K displays, http://www.pocket-lint.com/news/134580-forget-about-4k-and-even-8k-samsung-is-making-11k-displays
[RV] Hands-on: The New and Improved StarVR Prototype Will Give You Field-of-View Envy, http://www.roadtovr.com/starvr-headset-hands-on-field-of-view-e3-2016/
[SC] Are 4K 144hz gaming monitors coming out this year?, http://steamcommunity.com/discussions/forum/11/357288572138893130/
[Sensics] Understanding Pixel Density and Eye-Limiting Resolution, http://sensics.com/understanding-pixel-density-and-eye-limiting-resolution/
[TR1] Samsung teases its 4K, VR-ready phone display, http://www.techradar.com/news/phone-and-communications/mobile-phones/samsung-teases-its-4k-vr-ready-phone-display-1322341
[TR2] The 10 best graphics cards of 2016, http://www.techradar.com/news/computing-components/graphics-cards/best-graphics-cards-1291458
[TT] AMD's graphics boss says VR needs 16K at 240Hz for 'true immersion', http://www.tweaktown.com/news/49693/amds-graphics-boss-vr-needs-16k-240hz-true-immersion/index.html
[TV] 4 Most Important Tech Specs When Shopping For VR, https://topvr.co.uk/shopping/4-important-tech-specs-shopping-vr/
[VT] Comparison Chart of FOV (Field of View) of VR Headsets, http://www.virtualrealitytimes.com/2015/05/24/chart-fov-field-of-view-vr-headsets/
[WP1] Human Eye, https://en.wikipedia.org/wiki/Human_eye
[WP2] Field of View, https://en.wikipedia.org/wiki/Field_of_view
[WP3] Virtual Reality Sickness, https://en.wikipedia.org/wiki/Virtual_reality_sickness
[WS] 8 ways to prevent HTC Vive motion sickness, http://filmora.wondershare.com/virtual-reality/8-ways-to-prevent-htc-vive-motion-sickness.html
According to AMD’s Radeon Technologies Group, in order for VR to reach true immersion that one will not be able to tell apart from real world, 16K screens (probably 16K per eye) and 240fps are required [TT]. Elsewhere, Palmer Luckey, the founder of Oculus, has said that to get to the point where the user cannot see pixels, about 8K per eye would be needed. 16K is 15360x8640 pixels, whereas 8K is 7680x4320.
At least Sony has already released a smartphone, Xperia Z5 Premium, that has a 4K display [TR1]. Also Samsung has showed off a 4K smartphone display. Furthermore, Samsung is working on a new type of display that will have 11K resolution with 2250 pixels per inch [PL]. 11K could mean something like 10560x5940 pixels. Samsung expects to have a prototype ready by 2018. The screen could be available in smartphones in 2019.
But rather than resolution, a more suitable notion for VR environments is pixels per degree. If the human eye was a digital camera, it would have 60 pixels/degree at the fovea, which is the part of the retina where the visual acuity is highest [Sensics]. For VR goggles, the pixel density can be calculated by dividing the number of pixels in a horizontal display line by the horizontal FOV (Field of View) provided by the headset. HTC Vive has 1080 horizontal pixels per eye (i.e., 2160 in total) and a FOV of 110 degrees, resulting in a pixel density of 19.6 pixels/degree. This is still pretty far from 60 pixels/degree. For human eyes, the combined horizontal visual field is 180-200 degrees [WP1, WP2]. Thus, VR headset offering a wide 200 degree FOV with 60 pixels/degree might require a horizontal resolution of 6000 pixels. This means that Samsung’s coming 11K display might actually be able to reach 60 pixels/degree, which feels quite promising.
A wide FOV is crucial for an immersive VR experience by avoiding the tunnel vision impact and even allowing for peripheral vision and good sense of the user’s visual surroundings in the virtual environment [TV]. Some examples of FOVs of different VR headsets are as follows [VT]:
- Google Cardboard: 90
- Samsung Gear VR: 96
- HTC Vive: 110
- Oculus Rift: 110
- VR Union Claire: 170
- StarVR: 210
StarVR, which is still under development, has the highest FOV from among the VR headsets [VT]. 210 degrees offers a FOV beyond the human eye’s peripheral vision. According to some early reviews, the 210-degree FOV is an eye-opener in the virtual world and makes other headsets (e.g., the Rift and the Vive with their 110-degree FOV) feel like binoculars [RV]. Achieving a wide FOV requires a complex set of lenses. StarVR combines normal and Fresnel elements which, as a downside, brings challenges to clarity. StarVR has a per-eye resolution of 2560x1440 but the extra pixels (compared to competitors) are obviously stretched over the wider FOV.
Besides the horizontal FOV, also the vertical FOV has an impact on the VR experience, allowing the eyes to dart up and down more naturally without encountering the dark edges of the headset [TV]. The HTC Vive has a better vertical FOV than the Oculus Rift since it uses custom screens that are oriented vertically.
But what would be the ideal FOV for a VR headset? Even the VR experience using a VR headset with a 180-degree FOV will not not match how we experience the physical world. This is because the human eyes can see up to a 270-degree FOV if the eyes are fully rotated [VT]. Thus, even with a 180-degree FOV, one might still experience a tunnel vision effect (which is probably the reason StarVR went for 210 degrees).
An additional aspect to consider is that lower FOVs help with motion sickness [WP3, WS]. Decreasing the FOV results in less visual cues and the brain does not register as wide a discrepancy between sensory input from the eyes and the vestibular system in the inner ear (this mismatch in VR causes motion sickness). The challenge is naturally that reducing the FOV makes the VR experience less immersive by strengthening the tunnel vision effect. Fortunately, research is ongoing to solve this issue. As an example, researchers from Columbia University have developed an approach that involves masking the user’s view to minimize motion sickness symptoms [NA]. The approach adjusts the visual range on the fly to reduce motion sickness. When the player is in motion in the virtual environment, the system partially obscures each eye with a soft-edged, circular, virtual cutout, reducing the FOV to minimize motion sickness. When there is less action, the FOV is increased again.
Increased resolution and FOV or pixels per degree require of course more GPU power. Since the Sony Xperia Z5 Premium smartphone already comes with a 4K display, one could assume that someone packing two of those displays into a VR headset should not be that far off. Assuming that such a headset would have at least the same 90Hz refresh rate as the Rift and the Vive, we would need a GPU capable of running 4K@90fps on both of the displays, or one GPU for each of the displays. Nvidia’s latest Titan X desktop GPU, which is supposed to be the most powerful consumer graphics card on the market [TR2], has demonstrated performance of 4K@81fps (and should, according to the specs, be capable of 8K@60Hz, that is, 7680x4320@60Hz at the maximum) [NV]. Therefore, it would seem to be possible to run 2x4K@80Hz with two Titan X GPUs. The only problem is that two Titan X GPUs cost $2400.
So, to the conclusion. 4K smartphone displays are already here. 11K displays might be coming to smartphones in 2019. And today’s top consumer graphics cards can already do 8K@60fps. Thus, by the time Samsung’s 11K displays are out, it does not seem unrealistic to expect that we could have consumer graphics cards that can deliver 11K@120fps. So perhaps by 2019, we could reach an over 200 degree FOV with 60 pixels/degree and 120fps, that is, have VR that we cannot tell apart from the real world. However, it feels that 16K@240fps, which is AMD’s goal, is still further off to the future.
However, VR hardware makers are looking into ways to reduce the required processing power. One phenomenon that can come to help is foveal vision - when we look at an object in distance, the focal point of our gaze is in focus, while the scene around the object is blurred. In VR, this principle can be used to render fully only the specific area where the user is looking and leaving the rest of the scene at a far lower resolution [MT]. This has the potential to bring large performance gains and reduce the load of the GPU. At least the Fove VR headset and Nvidia are applying this trick to VR.
[AT] Virtual Perfection: Why 8K resolution per eye isn’t enough for perfect VR, http://arstechnica.com/gaming/2013/09/virtual-perfection-why-8k-resolution-per-eye-isnt-enough-for-perfect-vr/
[MT] Nvidia’s Eye-Tracking Tech Could Revolutionize Virtual Reality, https://www.technologyreview.com/s/601941/nvidias-eye-tracking-tech-could-revolutionize-virtual-reality/
[NA] Restricting field of view to reduce motion sickness in VR, http://newatlas.com/columbia-university-vr-motion-sickness/43855/
[NV] Nvidia Titan X, http://www.geforce.com/hardware/10series/titan-x-pascal
[PL] Forget about 4K and even 8K, Samsung is making 11K displays, http://www.pocket-lint.com/news/134580-forget-about-4k-and-even-8k-samsung-is-making-11k-displays
[RV] Hands-on: The New and Improved StarVR Prototype Will Give You Field-of-View Envy, http://www.roadtovr.com/starvr-headset-hands-on-field-of-view-e3-2016/
[SC] Are 4K 144hz gaming monitors coming out this year?, http://steamcommunity.com/discussions/forum/11/357288572138893130/
[Sensics] Understanding Pixel Density and Eye-Limiting Resolution, http://sensics.com/understanding-pixel-density-and-eye-limiting-resolution/
[TR1] Samsung teases its 4K, VR-ready phone display, http://www.techradar.com/news/phone-and-communications/mobile-phones/samsung-teases-its-4k-vr-ready-phone-display-1322341
[TR2] The 10 best graphics cards of 2016, http://www.techradar.com/news/computing-components/graphics-cards/best-graphics-cards-1291458
[TT] AMD's graphics boss says VR needs 16K at 240Hz for 'true immersion', http://www.tweaktown.com/news/49693/amds-graphics-boss-vr-needs-16k-240hz-true-immersion/index.html
[TV] 4 Most Important Tech Specs When Shopping For VR, https://topvr.co.uk/shopping/4-important-tech-specs-shopping-vr/
[VT] Comparison Chart of FOV (Field of View) of VR Headsets, http://www.virtualrealitytimes.com/2015/05/24/chart-fov-field-of-view-vr-headsets/
[WP1] Human Eye, https://en.wikipedia.org/wiki/Human_eye
[WP2] Field of View, https://en.wikipedia.org/wiki/Field_of_view
[WP3] Virtual Reality Sickness, https://en.wikipedia.org/wiki/Virtual_reality_sickness
[WS] 8 ways to prevent HTC Vive motion sickness, http://filmora.wondershare.com/virtual-reality/8-ways-to-prevent-htc-vive-motion-sickness.html
Thanks for an information based post.These tips are really helpful. Thanks a lot.Keep it up. VR Ottawa
ReplyDelete