Refresh rate refers to the number of times a complete image is drawn on the screen per second [VE]. It is different from frame rate - the refresh rate may include the repeated drawing of identical frames, whereas frame rate measures how often a video source can feed a new (non-repeated) frame to a display [WP2].
The refresh rate of a VR headset and the frame rate are some of the factors that can cause motion sickness (a.k.a., cybersickness or simulator sickness) if they are not high enough. This can be due to multiple reasons, including at least flicker and sensory mismatch.
If the refresh rate of a display is too slow, the user can perceive flicker of the entire virtual environment (a.k.a., judder), which can lead to motion sickness [WP1]. The brain can begin to experience judder (especially in the periphery) if the frame rate drops below 65fps [Nature], 75fps [MTG] or 90fps depending on which source one believes. According to Oculus, 75Hz panels are fast enough that the majority of users will not perceive any noticeable flicker [Oculus]. The reason different sources refer to different values might be that the number of frames per second detectable by humans ranges considerably from one person to another [LI]. Furthermore, people can also be trained to more quickly perceive visual information. A recent MIT study found that participants can identify images seen for as briefly as 13ms, which is about 77fps [MIT]. An U.S. Air Force study has shown that fighter pilots can identify planes seen for as little as 4.55ms, which would mean about 220fps [AMO].
Recent studies [Nature] have shown that sensitivity to flicker drops to zero near 65Hz, but only when the modulated light source is spatially uniform. When the modulated light source contains a spatial high frequency edge (a high frequency edge could in the simplest case be represented by an image which is first bright on the left half of the frame and black on the right and then inverted – a more complex example is natural images since they have many edges), all test subjects studied in [Nature] saw flicker artifacts over 200Hz and several of them reported visibility of flicker artifacts at over 800Hz. For the median viewer, flicker artifacts disappear only above 500Hz, meaning that the median viewer can distinguish between modulated light and a stable field at up to 500Hz. This is most likely due to unconscious rapid eye movements across high frequency edges in the displayed image. The implication of these findings is that modern display designs which use complex spatio-temporal coding need to update much faster than conventional TVs.
It should be noted that the impact of flicker is different when viewing a TV, computer monitor, or displays of a VR headset. Low refresh rates in computer monitors, which are viewed up close, will produce a noticeable screen flicker because the display fills a larger proportion of a person’s field of vision than a TV screen that is typically viewed from a distance [VE]. And VR headsets are even more challenging than computer monitors since they fill an even larger portion of or even the entire field of view of the user.
Despite of their many advantages for VR (e.g., low persistence), OLED displays produce some amount of flicker, similar to CRT displays [Oculus]. When a LED monitor is set to maximum brightness, its LEDs are typically glowing at full strength [FP]. If the brightness is reduced, the LEDs need to emit less light. This is achieved by inserting small pauses during which the LEDs turn off for a very short time. The same happens with CCFL (Cold Cathode Fluorescent Lamp) backlight LCD displays but CCFLs have a much longer afterglow than LEDs that turn off instantly. Thus, CCFLs are easier on the eyes than LEDs. Most LED-backlight monitors use Pulse Width Modulation (PWM) with a frequency ranging from 90Hz to 400Hz.
The first example of conflicts and errors in the vestibular-ocular reflex is the lag between head movements and corresponding updates on the display. In this case, the eyes perceive movement that is out of sync by a few milliseconds with what is perceived by the vestibular system. Here, the solution is to use high frame rate rendering possibly together with other techniques such as high-precision low-latency tracking and low-persistence displays (low persistence works by illuminating the display only briefly, then turning it off until the next frame is ready in contrast to keeping it illuminated continuously from one frame to the next [RV]). These technologies can reduce motion sickness since they minimize the mismatch between a user’s visual perception of the virtual environment and the response of their vestibular system [Fernandes]. As an example, OLED displays have low persistence, which significantly reduces blur during head movement. Oculus claims that the low-persistence display of the Rift eliminates motion blur and judder almost completely.
The second and perhaps more challenging example is VR users who do not or cannot move physically the same way they move virtually. In this case, high frame rate, high-precision low-latency tracking and low-persistence displays do not help. What happens in such situations is that while the eyes indicate that the person has moved, the vestibular and proprioceptive systems indicate that the person has not moved. One example is when the character is walking in the virtual environment while the player is in reality standing still or sitting. This is the inverse of the situation that occurs when a person is reading in a moving vehicle – in this case the eyes perceive no movement while the vestibular and proprioceptive systems do.
There is at least one potential solution to the mismatch between the visual system perceiving motion while the vestibular and proprioceptive systems don’t. A nonprofit medical practice and research group called Mayo Clinic has patented a new technology called Galvanic Vestibular Stimulation (GVS) which synchronizes the inner ear to what a person is viewing [Forbes]. GVS uses strategically-placed electrodes to trick the inner ear into perceiving motion [TC]. The electrodes are used to send specific electrical signals to nerves [Samsung]. Four such electrodes are placed behind each ear, on the forehead and at the nape of the neck. The electrodes are then all linked in real-time so that any movement in the visual field launches a synchronized GVS command. Besides vMocion, also Samsung has been using GVS. They recently showed off a special project they have been working on, called the Entrim 4D headphones [Samsung].
Nvidia has debuted a working version of a 1700Hz display [DT]. The display can maintain a stable image even when zoomed in with a microscope and shaken vigorously when combined with low-latency input. Doing the same on a 90Hz display such as the ones used by today’s VR headsets would result in a lot of blurring. A 90Hz display refreshes the image every 11.1ms, whereas Nvidia’s display does so every 0.59ms. Thus, a 1700Hz display in practice eliminates any lag from the display, which would help bring the motion-to-photon latency down. Nvidia believes that while a less than 20ms motion-to-photon latency is generally considered good enough for VR, things get better towards 10ms and there are even measurable benefits with a 1ms latency [RV].
Thus, to eliminate motion sickness for even the most sensitive users and enable 1ms-level motion-to-photon latency, a (close to) zero-latency display operating at 1700Hz would definitely be beneficial. However, that would naturally need also a GPU that can provide input (ideally, something like 16K video for each eye [EC]) at 1700fps, which is something we will not be seeing any time soon.
[CS] Tactile, Vestibular and Proprioceptive Senses, http://cherringtonsawers.com/tactile-vestibular-and-proprioceptive-senses.html
[DT] Nvidia’s Prototype 1,700Hz Display Could Unlock Frame Rates for Future VR, http://www.digitaltrends.com/virtual-reality/nvidia-1700hz-vr-display/
[EC] What Kind of a Resolution Is Needed to Deliver Perfect VR?, http://edge-of-cloud.blogspot.fi/2016/11/what-kind-of-resolution-is-needed-to.html
[Fernandes] Combating VR Sickness through Subtle Dynamic Field-Of-View Modification, http://www.cs.columbia.edu/2016/combating-vr-sickness/images/combating-vr-sickness.pdf
[FP] LED Monitors can cause headaches due to flicker, http://www.flatpanelshd.com/focus.php?subaction=showfull&id=1362457985
[Forbes] Mayo Clinic May Have Just Solved One Of Virtual Reality's Biggest Problems, http://www.forbes.com/sites/jasonevangelho/2016/03/30/mayo-clinic-may-have-just-solved-one-of-virtual-realitys-biggest-problems
[GS] Virtual reality overcoming the barrier of motion sickness, http://www.glitchstudios.co/post-1/
[LaViola] A Discussion of Cybersickness in Virtual Environments, http://www.eecs.ucf.edu/~jjl/pubs/cybersick.pdf
[LI] Does FPS Matter? Decide for Yourself, http://blog.logicalincrements.com/2015/04/does-fps-matter-decide-for-yourself/
[MIT] In the blink of an eye, http://news.mit.edu/2014/in-the-blink-of-an-eye-0116
[MTG] Physics and Frame Rate: Beating motion sickness in VR, http://mtechgames.com/2015/12/09/physics-and-frame-rate-beating-motion-sickness-in-vr/
[Nature] Humans perceive flicker artifacts at 500 Hz, http://www.nature.com/articles/srep07861
[Oculus] Simulator Sickness, https://developer3.oculus.com/documentation/intro-vr/latest/concepts/bp_app_simulator_sickness/
[PS] PlayStation VR: The Ultimate FAQ, http://blog.us.playstation.com/2016/10/03/playstation-vr-the-ultimate-faq/
[RV] NVIDIA Demonstrates Experimental “Zero Latency” Display Running at 1,700Hz, http://www.roadtovr.com/nvidia-demonstrates-experimental-zero-latency-display-running-at-17000hz/
[Samsung] Samsung to Unveil Hum On!, Waffle and Entrim 4D Experimental C-Lab Projects at SXSW 2016, https://news.samsung.com/global/samsung-to-unveil-hum-on-waffle-and-entrim-4d-experimental-c-lab-projects-at-sxsw-2016
[TC] vMocion looks to end motion sickness in virtual reality by tricking your brain, https://techcrunch.com/2016/03/30/vmocion-looks-to-end-motion-sickness-in-virtual-reality-by-tricking-your-brain
[TT] AMD's graphics boss says VR needs 16K at 240Hz for 'true immersion', http://www.tweaktown.com/news/49693/amds-graphics-boss-vr-needs-16k-240hz-true-immersion/index.html
[UQO] Cybersickess, http://w3.uqo.ca/cyberpsy/en/cyberma_en.htm
[VE] Computer Monitors and Digital Televisions - Visual Sensitivity from Vestibular Disorders Affects Choice of Display, http://vestibular.org/sites/default/files/page_files/Computer%20Monitors%20and%20Digital%20Televisions_0.pdf
[WP1] Virtual reality sickness, https://en.wikipedia.org/wiki/Virtual_reality_sickness
[WP2] Refresh rate, https://en.wikipedia.org/wiki/Refresh_rate
[WP3] Fovea centralis, https://en.wikipedia.org/wiki/Fovea_centralis
The refresh rate of a VR headset and the frame rate are some of the factors that can cause motion sickness (a.k.a., cybersickness or simulator sickness) if they are not high enough. This can be due to multiple reasons, including at least flicker and sensory mismatch.
Flicker
The human eye perceives a stable image without flicker artifacts when a display updates at a sufficiently fast rate, which is called the flicker fusion rate [Nature]. A so-called critical flicker fusion rate is defined as the rate at which human perception cannot distinguish modulated light from a stable field. This rate varies with intensity and contrast with the fastest variation in luminance one can detect at 50-90Hz. According to standards for display ergonomics, a refresh rate of 72Hz for computer displays is sufficient to avoid flicker completely. Sensitivity to flicker is different between the fovea (a small, central pit composed of closely packed cones in the eye that is located in the center of the macula lutea of the retina [WP3]) and peripheral vision (i.e., the edges of the field of view). According to [LaViola], a refresh rate of 30Hz is usually good enough to remove perceived flicker from the fovea. However, the human eye is most sensitive to flicker in its peripheral vision, meaning that the periphery requires higher refresh rates.If the refresh rate of a display is too slow, the user can perceive flicker of the entire virtual environment (a.k.a., judder), which can lead to motion sickness [WP1]. The brain can begin to experience judder (especially in the periphery) if the frame rate drops below 65fps [Nature], 75fps [MTG] or 90fps depending on which source one believes. According to Oculus, 75Hz panels are fast enough that the majority of users will not perceive any noticeable flicker [Oculus]. The reason different sources refer to different values might be that the number of frames per second detectable by humans ranges considerably from one person to another [LI]. Furthermore, people can also be trained to more quickly perceive visual information. A recent MIT study found that participants can identify images seen for as briefly as 13ms, which is about 77fps [MIT]. An U.S. Air Force study has shown that fighter pilots can identify planes seen for as little as 4.55ms, which would mean about 220fps [AMO].
Recent studies [Nature] have shown that sensitivity to flicker drops to zero near 65Hz, but only when the modulated light source is spatially uniform. When the modulated light source contains a spatial high frequency edge (a high frequency edge could in the simplest case be represented by an image which is first bright on the left half of the frame and black on the right and then inverted – a more complex example is natural images since they have many edges), all test subjects studied in [Nature] saw flicker artifacts over 200Hz and several of them reported visibility of flicker artifacts at over 800Hz. For the median viewer, flicker artifacts disappear only above 500Hz, meaning that the median viewer can distinguish between modulated light and a stable field at up to 500Hz. This is most likely due to unconscious rapid eye movements across high frequency edges in the displayed image. The implication of these findings is that modern display designs which use complex spatio-temporal coding need to update much faster than conventional TVs.
It should be noted that the impact of flicker is different when viewing a TV, computer monitor, or displays of a VR headset. Low refresh rates in computer monitors, which are viewed up close, will produce a noticeable screen flicker because the display fills a larger proportion of a person’s field of vision than a TV screen that is typically viewed from a distance [VE]. And VR headsets are even more challenging than computer monitors since they fill an even larger portion of or even the entire field of view of the user.
Despite of their many advantages for VR (e.g., low persistence), OLED displays produce some amount of flicker, similar to CRT displays [Oculus]. When a LED monitor is set to maximum brightness, its LEDs are typically glowing at full strength [FP]. If the brightness is reduced, the LEDs need to emit less light. This is achieved by inserting small pauses during which the LEDs turn off for a very short time. The same happens with CCFL (Cold Cathode Fluorescent Lamp) backlight LCD displays but CCFLs have a much longer afterglow than LEDs that turn off instantly. Thus, CCFLs are easier on the eyes than LEDs. Most LED-backlight monitors use Pulse Width Modulation (PWM) with a frequency ranging from 90Hz to 400Hz.
Sensory Conflicts
Cybersickness is believed to occur primarily as a result of conflicts between three sensory systems: visual, vestibular and proprioceptive [UQO]. The vestibular system is a complicated sensory system in the inner ear that provides balance and spatial orientation. The proprioceptive system is located primarily in the cerebellum [CS]. Proprioceptive information comes from receptors in the muscles, joints, and bones. In normal situations, the information coming from the visual, vestibular, and proprioceptive systems is in agreement.The first example of conflicts and errors in the vestibular-ocular reflex is the lag between head movements and corresponding updates on the display. In this case, the eyes perceive movement that is out of sync by a few milliseconds with what is perceived by the vestibular system. Here, the solution is to use high frame rate rendering possibly together with other techniques such as high-precision low-latency tracking and low-persistence displays (low persistence works by illuminating the display only briefly, then turning it off until the next frame is ready in contrast to keeping it illuminated continuously from one frame to the next [RV]). These technologies can reduce motion sickness since they minimize the mismatch between a user’s visual perception of the virtual environment and the response of their vestibular system [Fernandes]. As an example, OLED displays have low persistence, which significantly reduces blur during head movement. Oculus claims that the low-persistence display of the Rift eliminates motion blur and judder almost completely.
The second and perhaps more challenging example is VR users who do not or cannot move physically the same way they move virtually. In this case, high frame rate, high-precision low-latency tracking and low-persistence displays do not help. What happens in such situations is that while the eyes indicate that the person has moved, the vestibular and proprioceptive systems indicate that the person has not moved. One example is when the character is walking in the virtual environment while the player is in reality standing still or sitting. This is the inverse of the situation that occurs when a person is reading in a moving vehicle – in this case the eyes perceive no movement while the vestibular and proprioceptive systems do.
There is at least one potential solution to the mismatch between the visual system perceiving motion while the vestibular and proprioceptive systems don’t. A nonprofit medical practice and research group called Mayo Clinic has patented a new technology called Galvanic Vestibular Stimulation (GVS) which synchronizes the inner ear to what a person is viewing [Forbes]. GVS uses strategically-placed electrodes to trick the inner ear into perceiving motion [TC]. The electrodes are used to send specific electrical signals to nerves [Samsung]. Four such electrodes are placed behind each ear, on the forehead and at the nape of the neck. The electrodes are then all linked in real-time so that any movement in the visual field launches a synchronized GVS command. Besides vMocion, also Samsung has been using GVS. They recently showed off a special project they have been working on, called the Entrim 4D headphones [Samsung].
Conclusion
So how high a refresh rate and frame should a VR system offer? Oculus Rift and HTC Vive have a 90Hz refresh rate and a frame rate of 90fps, whereas PlayStation VR can provide up to 120Hz and 120 fps [PS]. According to AMD’s Radeon Technologies Group, in order for VR to reach true immersion that one will not be able to tell apart from real world, 240Hz and 240fps are required [TT]. However, even this may not be enough as indicated by the above-mentioned study that showed that some persons can detect flicker artifacts even at above 800Hz.Nvidia has debuted a working version of a 1700Hz display [DT]. The display can maintain a stable image even when zoomed in with a microscope and shaken vigorously when combined with low-latency input. Doing the same on a 90Hz display such as the ones used by today’s VR headsets would result in a lot of blurring. A 90Hz display refreshes the image every 11.1ms, whereas Nvidia’s display does so every 0.59ms. Thus, a 1700Hz display in practice eliminates any lag from the display, which would help bring the motion-to-photon latency down. Nvidia believes that while a less than 20ms motion-to-photon latency is generally considered good enough for VR, things get better towards 10ms and there are even measurable benefits with a 1ms latency [RV].
Thus, to eliminate motion sickness for even the most sensitive users and enable 1ms-level motion-to-photon latency, a (close to) zero-latency display operating at 1700Hz would definitely be beneficial. However, that would naturally need also a GPU that can provide input (ideally, something like 16K video for each eye [EC]) at 1700fps, which is something we will not be seeing any time soon.
References
[AMO] Human Eye Frames Per Second, http://amo.net/NT/02-21-01FPS.html[CS] Tactile, Vestibular and Proprioceptive Senses, http://cherringtonsawers.com/tactile-vestibular-and-proprioceptive-senses.html
[DT] Nvidia’s Prototype 1,700Hz Display Could Unlock Frame Rates for Future VR, http://www.digitaltrends.com/virtual-reality/nvidia-1700hz-vr-display/
[EC] What Kind of a Resolution Is Needed to Deliver Perfect VR?, http://edge-of-cloud.blogspot.fi/2016/11/what-kind-of-resolution-is-needed-to.html
[Fernandes] Combating VR Sickness through Subtle Dynamic Field-Of-View Modification, http://www.cs.columbia.edu/2016/combating-vr-sickness/images/combating-vr-sickness.pdf
[FP] LED Monitors can cause headaches due to flicker, http://www.flatpanelshd.com/focus.php?subaction=showfull&id=1362457985
[Forbes] Mayo Clinic May Have Just Solved One Of Virtual Reality's Biggest Problems, http://www.forbes.com/sites/jasonevangelho/2016/03/30/mayo-clinic-may-have-just-solved-one-of-virtual-realitys-biggest-problems
[GS] Virtual reality overcoming the barrier of motion sickness, http://www.glitchstudios.co/post-1/
[LaViola] A Discussion of Cybersickness in Virtual Environments, http://www.eecs.ucf.edu/~jjl/pubs/cybersick.pdf
[LI] Does FPS Matter? Decide for Yourself, http://blog.logicalincrements.com/2015/04/does-fps-matter-decide-for-yourself/
[MIT] In the blink of an eye, http://news.mit.edu/2014/in-the-blink-of-an-eye-0116
[MTG] Physics and Frame Rate: Beating motion sickness in VR, http://mtechgames.com/2015/12/09/physics-and-frame-rate-beating-motion-sickness-in-vr/
[Nature] Humans perceive flicker artifacts at 500 Hz, http://www.nature.com/articles/srep07861
[Oculus] Simulator Sickness, https://developer3.oculus.com/documentation/intro-vr/latest/concepts/bp_app_simulator_sickness/
[PS] PlayStation VR: The Ultimate FAQ, http://blog.us.playstation.com/2016/10/03/playstation-vr-the-ultimate-faq/
[RV] NVIDIA Demonstrates Experimental “Zero Latency” Display Running at 1,700Hz, http://www.roadtovr.com/nvidia-demonstrates-experimental-zero-latency-display-running-at-17000hz/
[Samsung] Samsung to Unveil Hum On!, Waffle and Entrim 4D Experimental C-Lab Projects at SXSW 2016, https://news.samsung.com/global/samsung-to-unveil-hum-on-waffle-and-entrim-4d-experimental-c-lab-projects-at-sxsw-2016
[TC] vMocion looks to end motion sickness in virtual reality by tricking your brain, https://techcrunch.com/2016/03/30/vmocion-looks-to-end-motion-sickness-in-virtual-reality-by-tricking-your-brain
[TT] AMD's graphics boss says VR needs 16K at 240Hz for 'true immersion', http://www.tweaktown.com/news/49693/amds-graphics-boss-vr-needs-16k-240hz-true-immersion/index.html
[UQO] Cybersickess, http://w3.uqo.ca/cyberpsy/en/cyberma_en.htm
[VE] Computer Monitors and Digital Televisions - Visual Sensitivity from Vestibular Disorders Affects Choice of Display, http://vestibular.org/sites/default/files/page_files/Computer%20Monitors%20and%20Digital%20Televisions_0.pdf
[WP1] Virtual reality sickness, https://en.wikipedia.org/wiki/Virtual_reality_sickness
[WP2] Refresh rate, https://en.wikipedia.org/wiki/Refresh_rate
[WP3] Fovea centralis, https://en.wikipedia.org/wiki/Fovea_centralis
No comments:
Post a Comment