Are there any #hardware #engineers around here, especially relating to #DRAM? I'm trying to understand some issues I've seen at large scale deployments (hyperscale datacenters) which are pretty rare but still common enough to be a really headache inducing issue. It's all ECC RAM of course. But […] - ThreadSky

anthropy.mastodon.derg.nz.ap.brid.gy • 3 days ago

Are there any #hardware #engineers around here, especially relating to #DRAM?

I'm trying to understand some issues I've seen at large scale deployments (hyperscale datacenters) which are pretty rare but still common enough to be a really headache inducing issue.

It's all ECC RAM of course. But […]

Comments

doridian.net•3 days ago

@anthropy For ECC I have one hypothesis: ECC can only detect so many bit errors. If you get bit flips just right so that the check/parity still matches, they can do undetected. I think normal ECC is "can fix 1 bit, can detect 2 bits?" per.....some grouping of bits/bytes

anthropy.mastodon.derg.nz.ap.brid.gy•3 days ago

@Doridian that was one of my own theories, but honestly I find it unlikely that this would happen consistently in such a way that it NEVER triggers correctable/uncorrectable ECC errors but does crash applications. Even rowhammer-y things should technically be detected by ECC (I think?? but this […]

doridian.net•3 days ago

@anthropy Turns out, after some research, you in fact CAN rowhammer ECC by using a timing side channel to tune your attack: https://www.vusec.net/projects/eccploit/

anthropy.mastodon.derg.nz.ap.brid.gy•3 days ago

@Doridian but they appear to rely on finding reliably flippable bits first, which, is kinda cool with the timing sidechannel in ECC correction and what not, but, this should also generate a shitload of correctable errors that we'd see on these systems first.

Speaking of rowhammer, I've seen […]

doridian.net•3 days ago

@anthropy Yeah, at these large scales, the 0.0001% "never in a lifetime" chances, well, they compound. The amount of CPU and other hardware bugs those hyperscalers discover alone must be insane, not to speak of the "well this is just not fully 100% reliable because it is still physical hardware" x3

anthropy.mastodon.derg.nz.ap.brid.gy•2 days ago

@Doridian yea I mean at 100k machines a 0.001% issue would happen multiple times a year, and it's a lot more than that :blobcatgooglyholdingitsheadinitshands:

to me personally the most fascinating issues are e.g the rare thermal runaways, where some server actually turns to mostly ash inside […]

anthropy.mastodon.derg.nz.ap.brid.gy•3 days ago

I've seen this across many types of DRAM; DDR3/4/5, HBM1/2/3(e), etc.

I kind of know how DRAM works; field of tiny (shitty) capacitors that constantly slowly drain and therefore require refreshing, each storing a single bit of data, they have many different voltage buses, and if you know […]

anthropy.mastodon.derg.nz.ap.brid.gy•3 days ago

they're also a huge headache for datacenter techs/engineers, because nothing detects some these issues correctly, so you end up having to do stuff like taking out half the RAM and try and see if the machine reaches the login screen and maybe run a test (30-2880 mins per round, 32+ dimms in a […]

Comments

Posting Rules

Reply