An automated test is not a magic enchantment that makes your code safe, it's just a check for some condition that is run automatically so you don't forget to do it. If the check itself is written poorly, it will be useless or even wrong. - ThreadSky

shooter--andy.bsky.social • 28 days ago

An automated test is not a magic enchantment that makes your code safe, it's just a check for some condition that is run automatically so you don't forget to do it.

If the check itself is written poorly, it will be useless or even wrong.

Comments

shooter--andy.bsky.social•28 days ago

For instance, let's say you have a webpage that shows a button that has to do something when you click it. Then you'd write two tests: that the button is shown when the page is loaded, and that clicking on it does the action it's supposed to do.

And as long as this holds true, the tests will pass.

shooter--andy.bsky.social•28 days ago

But what if you add some more complex logic there? Let's say the button requires specific user permissions, and you have a complicated system that checks for those. Sure, you can just fake the results of that check for the test, but then you're skipping multiple failure points.

shooter--andy.bsky.social•28 days ago

And then, the question is: will the GenAI that writes these tests take this shortcut? Will it understand how this permission check works?

And if it won't... what can you do about it?

Worse yet, will you even *know* that you need to do something? You didn't write the tests, after all.

shooter--andy.bsky.social•28 days ago

Sure, with proper oversight, it should be possible to notice these issues and fix them, but now you're spending human time overseeing the results of GenAI and correcting them. Did you even win any time at all as a result?

And you can't just fire the GenAI or train it like you can with a human.

aaronsterling.bsky.social•28 days ago

No. "Write tests that provide 100% line, branch, function and statement coverage of the attached code. Ensure robust handling of null and malformed inputs. Follow Testing Best Practices Guidelines in Project Knowledge." That will produce a day's work in a few minutes. The Guidelines are training.

shooter--andy.bsky.social•28 days ago

I'll believe it when I see it, and so far, I have not.

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply