A study measured the performance of 800 developers before and after getting GitHub's Copilot.
It found:
- No change in productivity
- No change in burnout levels
- A 41% increase in the bug rate!??
Aren't you thrilled we're burning the world to power these AI models?
It found:
- No change in productivity
- No change in burnout levels
- A 41% increase in the bug rate!??
Aren't you thrilled we're burning the world to power these AI models?
Comments
I review its outputs closely, of course, but when it yields cases that fail? it’s usually that I missed something.
anyone asking AI to write code for them is playing with fire.
Although maybe that's why the 41% increase in bugs...
To be fair, it was for my thesis, so it needed to be really good, but at the end I came to the conclusion that it would have been faster to just read the RFC directly.
In addition to AI hallucinations, I think part of the issue is when you do the work yourself, you have subconscious metadata about what aspects are reliable, areas that might need attention, etc.
When it comes from an AI, literally nobody knows.
I disabled TPM in my BIOS to prevent Windows 11 from installing itself without asking (and because TPM does nothing useful for machines at home).
https://bsky.app/profile/nafnlaus.bsky.social/post/3l5nq54ygy726
but that's it just trivial time saving stuff. it should never used to write actual logic as its suggestions barely meet the "does it compile" criteria
So you get 41% more bugs, 55% more rapidly than ever before.
Ain't AI grand.
*Not actually cheaper
https://resources.uplevelteam.com/gen-ai-for-coding
Because that's exactly the data it was trained on. Sometimes you get lucky but sometimes the training data never actually compiled to begin with.
I sometimes thought "why th'fuck would I want this? I've been programmin' video game modifications stuff for 8+ years just fine... lemme compile *and* fuck up my own weird code."
The buggier code is balanced by how much faster they can produce shit code.
But hey, that would make (a little) sense, so I expect the numbers and productivity "change" are something else entirely.