Some game engineer who's good at shaders please explain to me why I have to compile shaders on my local machine so much, for so many games. Couldn't they just, I don't know, compile shaders for all popular graphics cards and then just download the right ones for your card? What am I missing? - ThreadSky

danielzklein.bsky.social • 189 days ago

Some game engineer who's good at shaders please explain to me why I have to compile shaders on my local machine so much, for so many games. Couldn't they just, I don't know, compile shaders for all popular graphics cards and then just download the right ones for your card? What am I missing?

Comments

mrhands31.bsky.social•189 days ago

Take all the popular graphics chips, multiply them by vendor, then multiply that by possible driver versions, then multiply that by possible combinations of RAM, CPU, and bus speed, and that figure is why end-users spend a lot of their days compiling shaders

cosmic-dancer.bsky.social•189 days ago

What I don't get is why it has to be per vendor, where does vendor come in when it's essentially the same architecture?

mrhands31.bsky.social•189 days ago

My knowledge is outdated, but the fastest way to upload a shader is/was with a binary format. This binary blob will differ per graphics chip, driver, and yes, vendor: "It is not reasonable to expect different hardware vendors to accept the same binary formats" https://www.khronos.org/opengl/wiki/Shader_Compilation#Binary_upload

shana.spoiledcat.com•189 days ago

are-we-dead-yet.bsky.social•189 days ago

You could compile to an intermediate representation (like SPIR, PTX, etc.), but it would not be inclusive of all targets (just more than one). Additionally, it might not be as optimized for hardware as compiling and optimizing locally for specific hardware.

GPU architecture has changed a lot.

emmatech.dev•189 days ago

nope because each game has different shader code that accomplishes different tasks
and it has to compile shaders so often because the compiled code isn't saved anywhere on your computer (which could be a good optimization, but it's might not be easy or possible)

serrapi.bsky.social•189 days ago

fwiw shader caching can be enabled on steam, (at least on linux, idk about windows)

Steam Settings -> Downloads, at the bottom there should be options for pre-caching and background caching.

This should help, but it only seems to work until the game/proton is updated again =/

emmatech.dev•189 days ago

ooh that's cool :o
something I should try out :>

danielzklein.bsky.social•189 days ago

That's for Vulkan and OpenGL games, which is close to 0 of the games I play. It's all DirectX here on PC

serrapi.bsky.social•189 days ago

Linux is PC =), DirectX games are ran on linux via dxvk (and proton), so its kind of odd to me that the option exists on linux versions but not windows.

danielzklein.bsky.social•189 days ago

Well my understanding is it only re-compiles shaders when there's a change to shaders, right?

emmatech.dev•189 days ago

yea good point... it may be that your games have to load different shaders for each level but I'm not really sure :P

coreh.bsky.social•189 days ago

Not a super expert at this, but my best guess is that the compiled shader code is very tightly coupled not only to the card, but to the internal ABI of the driver, incompatible down to minor/patch version. So the possible target combinations are probably too many, and changing all the time.

danielzklein.bsky.social•189 days ago

That's plausible! And depressing! We should be able to do better, right?

spoice.bsky.social•189 days ago

I believe the Steam Deck does this. One of the advantages of having a single model of hardware.

danielzklein.bsky.social•189 days ago

Nice! Although I use my steam deck pretty much exclusively to play games that don't need to compile shaders ;P (or do so quietly and quickly, I guess)

shelbyarr.bsky.social•189 days ago

I’m definitely a layman in this, but I think this interview with Tom Petersen has some meat to it (they start really focusing on shaders about four minutes in): https://youtu.be/Qp3BGu3vixk?si=8Xj2Xc3VXaNR_xYk

cyb3rmaniak.bsky.social•189 days ago

Was about to post the same link. This interview stuck with me and was really enlightening.

stoogebag.bsky.social•189 days ago

the short answer is driver versions matter

protectful.bsky.social•189 days ago

tellem bri

badmofo.bsky.social•189 days ago

This was the most concise answer that makes the most sense

danielzklein.bsky.social•189 days ago

Damn and I guess if you wanted to pre-compile against the most popular combinations of cards and driver versions, I guess that would just be too many shaders to maintain developer-side?

subtilicus.bsky.social•189 days ago

well they are also not of an ignorable size, some games have a gigabyte or more of shader variants. So it could be a huge download to cover all bases.

gearhook.goblin.horse•189 days ago

"Why do I have to load this sword in to memory for every game that has a sword? There are swords in so many games, can't I just load it the once?"

Similar concept- even if they are more widely-shareable than literal visual assets they are treated as assets.

danielzklein.bsky.social•189 days ago

But I don't build the sword on my computer from first principles. We don't encode "swords look like this" and then generate the meshes and textures and rigs from scratch. We build the asset, save it, and ship it with the game. Why can't we do that with shaders?

gearhook.goblin.horse•189 days ago

That is how we render an asset though! Mesh data is saved and packed independently of texture data- which is collated and packed independently of any relevant physics or animation data! It is loaded, and rebuilt… or “Compiled”… by the engine to be rendered

foulharpy.bsky.social•189 days ago

I would love to know this because I spend half my life compiling shaders these days

rjazzman323.bsky.social•189 days ago

you know how many different subtle variations there are though

todd--umptious.bsky.social•189 days ago

To save you a trip to twitter, here's a good explanation from Hello Games Martin Griffith on the unique problem with using shaders in a proc gen game, Vs a traditional hard rendered level in other games.

todd--umptious.bsky.social•189 days ago

"I’m going to hijack Ray’s awesome post here and use it to talk about a particularly challenging engineering problem that we solve in No Man’s Sky, which arises as a direct consequence of our game universe being so astronomically large:"

todd--umptious.bsky.social•189 days ago

You all probably know about shaders, which are the primary basis for rendering rasterised graphics in games and also might know that large titles can have thousands of shaders, so that they can render a rich set of materials/meshes, particles, shadows, post fx … and everything you see, in game.

todd--umptious.bsky.social•189 days ago

We face a hard and complex task in NMS when it comes to these sort of shader combinatorics - not only as biome diversity and richness has grown over the years, but also because the exact set of shaders needed to render a planet or anything else that is proc gen is not known until it is calculated

todd--umptious.bsky.social•189 days ago

- nothing in NMS exists until the seeded generation produces the system, planet, ship, station (etc) or anything else that you then see on your screen!

todd--umptious.bsky.social•189 days ago

This aspect makes “shader loading” in our game more challenging than a lot of titles, where a traditionally designed/static world might allow you to have information about which shaders are going to needed, ahead of time.

fantasyvhs.net•189 days ago

Shaders are something the game developer writes, per game (broadly speaking). For example if there's some leaves that blow in the wind, that might be a shader. Or a button that goes all fuzzy when you click it - shader.

danielzklein.bsky.social•189 days ago

Yup! I know what shaders are! The thing I don't understand is why you can't build shaders for your game against all (or most) of the most popular video cards and then detect what card a player has and download the shader for that. I don't wanna sit here and not play my game for 10 minutes.

fantasyvhs.net•189 days ago

Also I think the settings you change in game can affect how your shader compiles, like depth settings or number of iterations for performance targets. And changing the settings can require recompilation.

fantasyvhs.net•189 days ago

Oh, I misunderstood you - my understanding there is that because of the disparity in hardware, optimisations end up *really* specific during compilation -- I think you'd need an intermediary shader language to compile to make it practical? Tbh, what you're suggesting is possible afaik!

segmentationfault.bsky.social•189 days ago

Steam does that.

Comments

Posting Rules

Reply