Improving VR with NVIDIA’s Foveated Rendering

Foveated rendering is going to be the most significant step towards making mobile VR viable, so this is very good news. I'm curious as to how they will handle eye tracking. I hope it uses capacitive lenses that can track your pupil bulges instead of unweildly and power hungry internal cameras + CV processing. This is what contrast-preserving foveated rendering looks like in VR: [VIDEO]

>And that power could be invested into more convincing graphics and lower latency. Fortunately, that problem is pretty close to being solved on a consumer level. A foveated rendering addon module for the Vive is expected to be shipping soon in China, by Q3 in North America [0]. For anyone unfamiliar with the term, foveated rendering means only rendering high-resolution images in your field of view. Basically, you don't see full detail across your entire field of view - only at the point where your eyes are focused. Everything outside that point can be rendered in lower resolution and you'd never notice the difference. It requires high-speed eye tracking to pull off, but it can greatly reduce the required horsepower to render a convincing VR environment [1]. [0] - [LINK] [1] - [VIDEO]

I was really happy to see eye tracking in this list as I consider that a "gateway technology" that opens up a variety of additional really cool features. The big one is probably foveated rendering [1] which both allows for a more natural look as well as (hopefully) more efficient visuals. 1 - [VIDEO]

Then you render with multiple cameras rotated around their centre, using the same techniques as cubemap rendering. With high field of view, rendering a single spherical view with homogeneous resolution would be inefficient for VR, not only due to the lack of hardware support (without noticeable artefacts or some subdivision scheme) and the created distortion, but also due to the foveated nature of human vision. You'd want a small view frustum rendered at a much higher pixel per angle blended with a lower resolution rendering of the entire field of view (optimally with a medium resolution view frustum surrounding the smaller one for a more progressive blending). See: [VIDEO] With current GPUs and game engines, you need multiple cameras to achieve such effect, but it has the advantage of drastically lowering the computing demands for high-resolution VR graphics (compare 8k per eye, vs just two blended 1080p views). You could also ray trace a single view of non-homogeneous resolution that directly takes into account the distortion characteristics of the HMD lenses, but it would likely a lot less efficient than traditional rasterization with multiple blended cameras.