A Deep Dive into Nanite Virtualized Geometry

Screen-space reflections are faster and require no new scene draws, but they won't show you "new angles" on reflected objects and they won't go around corners. Planar reflections require a new scene draw, but that's actually pretty modest: non-planar reflections require six scene-draws for a cubemap or full on raytracing. In general, lots of things in games require re-rendering the scene multiple times per frame, so game engines are really, really good at this. Shadows, projections, transparency / translucency, motion/depth blur, indirect lighting, deferred lighting, even things like hit testing (though for dependency + latency reasons that's a prime candidate for offloading to the CPU). Anything that requires less visual fidelity will have its performance tuned accordingly: fewer pixels, simpler shaders, lower LOD geometry, lower framerates. Nobody will notice if the fuzzy bounce lights are calculated once every five frames and blended. Ray tracing is conceptually simpler but (so far) slower at any given level of cleverness -- you'll often hear that it has asymptotic benefits, but those same benefits can be pulled out of rasterizers with effort comparable to what is required to make the raytracing practical. Here is a writeup of the passes you might expect to see in a modern engine: [LINK] Here's an overview of a modern geometry pipeline: [VIDEO]

Unreal Engine's Nanite technology has the same problem. They want to walk a LoD/culling tree but have to resort to the CPU just to schedule new tasks on the GPU. I wouldn't be surprised if future GPUs eliminate this requirement as it will make all Nanite Unreal Engine games run a little faster. Kind of like how a couple years after Quake came out, every GPU got fast z-buffers. [VIDEO]

For real. The talent in the game industry is insane. Recently, I got my mind blown by the Unreal Nanite team: [VIDEO] The core of it is a respectably sophisticated LOD building algorithm, as you might expect, but the sheer amount of engineering horsepower they put into driving it into production just boggles the mind.

> OpenGL is not dead until Khronos comes up with an API that is actually usable without an EE degree on GPU design and shader compilers. That isn't going to happen because everyone is, in fact, moving away from the idea of a "graphics API" altogether and simply allowing the compute systems to calculate everything. See: the Nanite renderer from EPIC: [VIDEO] To a first and second order approximation, no one cares about graphics that aren't related to games.

If this UE5 demo is using Nanite, then it really may well just be modeled geometry. Nanite really blurs the line between geometry and texture - in a sense it’s a shader that uses triangle mesh data as if it were a texture source. This siggraph session will expand your mind: [VIDEO]

If you want to see this sort of brilliant hacking on a modern system, I recommend taking a closer look at Unreal Nanite. It isn't just auto-LOD. Oh no. That's the core, but they had to do this same sort of "work around the tools they are given" to make it actually happen. Tiny triangles chug in modern GPU hardware raster pipelines -- so they wrote a GPGPU software rasterizer. They needed more bandwidth, so they wrote a compression engine. Their shaders needed better scheduling, so they abused the Z-test to do it. It's nuts! [VIDEO]

No, but it is a full time job. It's getting much more complicated. Here's an hour long talk on how Unreal Engine 5's Nanite works.[1] This is a huge breakthrough in level of detail theory. Things that used to be O(N) are now O(1). With enough pre-computation, rendering cost does not go up with scene size. See this demo.[2] You can download the demo for Xbox X/S and PlayStation 5, and soon, in source form for PCs. Explore 16 square kilometers of photorealistic city in which you can see a close-up view of anything. The format of assets in memory for Nanite becomes far more complicated. GPUs need a redesign again; UE5 has to do more of the rendering on the CPUs than they'd like. What they need to do is simple but not well suited to current GPU designs. It's such a huge win this approach will take over, despite that. New game engines will need to have something comparable. Maybe using the same format. Epic doesn't seem to be making patent claims, and the code for all this is in the Unreal Engine 5 download. (Basic concept of how this works: A mesh becomes a hierarchy of meshes, allowing zoom in and out of on level of detail. The approach to subdivision is seamless. You divide the mesh into cells of about 20 triangles. Then, each cell is cut by one or two new long edges across the cell. These become the boundaries of a simpler cell with about half as many triangles, but no new vertices. The lower-resolution cell has no more edges than the higher-resolution cell. There's a neat gra

Oh, I think you're right. The nested FOR loops of the "Micropoly software rasterizer" are running on the GPU.[1] I think. I though that was CPU code at first. They're iterating over one or two pixels, not a big chunk of screen. For larger triangles, they use the GPU fill hardware. The key idea is to have triangles be about pixel sized. If triangles are bigger than 1-2 pixels, and there's more detail available, use smaller triangles. [1] [VIDEO]

Yeah, but they had to go through hell to make it happen. It took a lot more effort than you might think. [VIDEO] Some highlights: Seam staggering between levels for smooth swaps Current hardware rasterizers chug on small polys so they made a GPU software rasterizer (!) Actually it's a hybrid rasterizer, and gnarly features like derivatives are supported on both sides The GPU scheduler chugged so they abused Z-test wave repacking Streaming compression

It attracts truly extraordinary talent. One of many examples: [VIDEO] . There is more technical ambition in a single slide of that project than an entire department at places I've worked. However, it also tends to demand more and pay less, so there is churn and burnout and when it comes time to triage, the triage can cut very deep. If I were to dive into the JIRA for any one of these "missing compression" incidents, I wouldn't expect to find an easy win, I would expect to find "missing compression" beneath a pile of much worse bugs like "level 2 crashes on AMD cards." So it goes.

I think there does in fact exist a recent major breakthrough in real time rendering: Virtualized geometry. [LINK] [VIDEO] It was invented by Brian Karis at Epic. Virtualized geometry allows for unbounded geometric detail while retaining the conventional rasterization approach with polygons and textures. It works like a dynamic LOD, where the game engine dynamically adapts the rendered geometric detail based on visibility and screen resolution. Essentially the maximum rendering cost only depends on resolution and frame rate, not on the detail of the underlying meshes. However, the solution by Epic, dubbed Nanite, does not yet work for animated meshes. So virtualized geometry is not possible e.g. for game characters. Brian Karis says this is a solvable problem though.

Full disclosure: I'm a software performance nerd who is building a deeper understanding of hardware using whatever tools are I work with a lot of assembly traces emitted from CPU sims in Verilator. These traces can be several gigabytes in size, with tens of millions of entries. I've found the typical interactive matplotlib backends to not work the best with tens of millions of points on my M1 Macbook Pro. This is no slight against Matplotlib, or cairo, or agg, or the default macOS backend—they prioritize cross-platform "It Just Works" support over extreme performance. But it does irk me that there seems to be such a gap between the amount of data we can visualize in the gaming domain in comparison to the scientific domain[1]. The difference between 50 million and 5 billion is 100X. I'd be ecstatic if I could get within 10X difference. I understand that this gap exists for lots of reasons, of which I'd be happy to hear in detail in the replies to this comment. 1: [VIDEO]

There's a lot of techniques applied. It's not really "one new revolutionary thing" but a bunch of things and research over the past 20-30 years put together to make nanite possible. This video might shed some light [VIDEO]

Nanite has a large pre-render step, but it also changes the rendering quite significantly. (Which in some cases includes a software rasterizer running on the GPU) This is a really good deep dive if you’re interested: [VIDEO]

I heard Unreal Nanite built a job queue system on compute threads ( [VIDEO] ), would that help with your use case or not?

The answer to that is in this hour-long SIGGRAPH video.[1] Some of the operations needed are not done well, or at all, by the GPU. [1] [VIDEO]

I think the SIGGRAPH talk you referred to is: "A Deep Dive into Nanite Virtualized Geometry" ( [VIDEO] ) There's also this short high-level intro (2.5 min) that I thought was decent: "What is virtualized micropolygon geometry? An explainer on Nanite" ( [LINK] )

Thanks for posting this! I'll be digging into it. It seems like a good basis for learning how the tech works. For those interested in more on continuous LOD, here's a SIGGRAPH talk where they go into some of the why and how of Unreal's Nanite (similar continuous LOD system) here: [VIDEO]

So I would say skill at GPU assembly is in-demand for the elite tier of GPU performance work. Not necessarily writing much of it (though see [1] for an example, this is the kernel of multisplit as used in Nvidia's Onesweep implementation), but definitely in being able to read it so you can understand what the compiled code is actually doing. I'll also cite as evidence of that the incredible work of the engineers on Nanite. They describe writing the core of the microtriangle software renderer in HLSL but analyzing the assembler output to optimize down to the cycle level, as described in their "deep dive into Nanite virtualized geometry" talk (timestamp points to the reference to instruction-level micro-optimization). [1]: [LINK] [2]: [VIDEO]

Their YouTube channel has the excellent deep dive on the unreal engine's nanite virtual geometry system: [VIDEO] The one on idTech8 global illumination from last year is pretty interesting too.