Chatbots — LLMs — do not know facts and are not designed to be able to accurately answer factual questions. They are designed to find and mimic patterns of words, probabilistically. When they’re “right” it’s because correct things are often written down, so those patterns are frequent. That’s all.
Comments
The more AI aggregates, the more errors will creep in. The best way for the chatbot to return correct information is to lift whole paragraphs from reputable sources.
As a society, we need to get back to rewarding being right about things and punishing being wrong about them!
“Read my essay. Is this good enough to submit to The New Yorker?”
“Yes, AND…”
What you type becomes "Then the user said X", and anything generated for the other character gets "performed" at you.
Five minutes of "conversation" about cute innocent puppies that must be protected, and *poof* now it's named Cruella De Vil plotting to skin dalmatian puppies into coats.
"Wow, It greeted me politely, and even I put in 2+2= and it gave me 4! IT'S DOING MATH TOO!"
"No, it really isn—”
"The Singularity is upon us!"
Like those who grew up in the transition from no computers to modern day computers tend to know the rudimentary functioning of data input to logical output.
There's a whole generation now that have fully functioning tech thrust in their hands but no baseline.
https://youtu.be/9DpM_TXq2ws?si=s5QLRtptttr-dXXU
and wrote this paper
https://afutureworththinkingabout.com/wp-content/uploads/2024/01/Bias-Optimizers-PREPRINT.pdf
almost two years ago now 😞; it's just… it's been out there
You ask Google a question & it gives you the answer. If it's the wrong answer, it's the website it found that's wrong not Google. LLMs normally don't give you the source
Just read a paper, that reflects this as well: https://arxiv.org/abs/2503.14499
The "survival rate" increase seems to be linear with bigger models, more compute and more memory.
There is no reasoning/intelligence.
And that's before we get into the MASSIVE copyright issues that most AI geeks are conveniently pretending don't exist.
I might be an outlier on this, though; I'm smart but undereducated.
Whoever does a study on this, make this a question!
semantics are subject-dependent
now, since LLMs produce text according to the internal representations they have learned during training, are they also semantic subjects?
frankly idk
"By deciding what robots are for, we are defining what humans are." Excerpt from my HUMANS: A MONSTROUS HISTORY:
https://youtu.be/f8FAJXPBdOg
https://youtu.be/ObvxPSQNMGc
https://netsettlement.blogspot.com/2025/03/sentience-structure.html
And by the same logic, we must stop eating bacon.
We are capable of quite amazing levels of hypocrisy.
This is not a leap of faith: it's a decision NOT to leap into a position on computer science, but to focus on how best to live.
I didn't write poems because they were any good: I wrote them to help feel what I was going through.
I don't fence to become a champion: I do it to feel alive. Etc etc etc.
Frustrating!
gpt and grok are still trying to work out how to do that integration and probably why Gemini suddenly lept to the front benchmarks and stayed there for the most part
They were definitely built to be correct as frequently as possible and when they are not correct, it is considered a fault by the designers.
competing with a single intellect for accuracy and begins competing with whole institutions, disciplines, and movements that have evolved checks and balances. If it even can.
I just don't get why we aren't using it in ways suited to its strength, guided process automation, rather than these things it just can't do
It’s bad enough trying to type on an iPhone, without having to go back and correct the corrections.
I like it :-)
Possibly "extruded music product", around for ages, to describe the rafts of bland manufactured pop music, designed (by humans) to part teenagers from money.
By the way, get off my lawn ;-)
I'm aware that I may overstretch the meaning of 'advanced', 'statistical', and 'evidence'. Notwithstanding, I kind of like the acronym you can make of the combination.
It is artificial non-intelligence. ANI.
All because of the volume of fake stories on a person without a firewall of public knowledge to drown it out.
Chatbots inner systems can have the ability to do complex math, or answer new riddles, because being able to do those things can help predict the next word.
Also RLHF is used, so they are optimized for doing more than just predicting the next word now.
But now we’re automating the process.
The big word here being CAN, they are still often wrong now, can "hallicinate" a ton, and make a large number of mistakes.
Where they come up with something…
I guess if you don't want to see it you don't have to
Soylent Meaning.
So many fan fictions that likely got scrapped for AI training including "I don't want to be turned off" and likely resulted in that response.
They didn't think. They are not afraid. They just rolled the dice.
I reminded it that it's a program without emotions.
Doesn't look like it gives a shit about being on or off.
It's fascinating to me how much superficially intelligent behavior you can get out of compiling down a massive corpus of material into an executable model: distilling the knowledge out of it, as it were.
But it's clear you need more than that for 'real' intelligence.
It has no knowledge or discernment.
I just found it a weird thing to block someone over, and was interested in the logic behind it.
It's actually more a demo of an inquisitive brain, wanting to understand how people think. But hey... feel free to read into it whatever you like. 🙂
...I think I know why this guy is more impressed with LLMs than normal people.
LLMs certainly have significant limitations, but the fact remains we're well beyond some of the oversimplifications being described here.
One connected to the next.
There’s no conscious direction. Just the machine working.
What flys out might be Millennium by the Backstreet Boys, or it might be No Strings Attached by NSync, or it might be a rotten tomato, no thought, just luck of the draw.
Or Authoritative (source) Impersonation, Automated Idiocy, Asinine Ideas..
And because it works with tokens, not words, it's even worse at spitting out a factual statement.
The input steers it so even subtly different input can steer it differently. The context window makes a difference too. You can chop off some early context to fit in the window and suddenly it answers very differently.
1. They aren't wrong because people are wrong; they would be wrong anyway
2. They aren't wrong, because the information that people gave them is wrong, so you can't expect them to be right
Or is there a third meaning I have missed?
No intelligence, just a dictionary and a Mixmaster.
And even that is only training data for the reinforcement layers that ride on top of the universal text models.
For example, until recently if you modified the fox-chicken-grain boat river riddle to make it *easier* to solve,
LLM are not magic bullets, but for this they make sense.
https://imdb.com/title/tt0113277/fullcredits/?ref_=tt_ov_3#cast
All machine learning is a “best guess” based on the training data, but it still often uncovers previously unseen information.
What's the actual difference between knowing and "knowing"?
More similar to a really well educated parrot than a person.
We speak by stringing together words. Especially those who don't/can't think with words internally often note how words just seem to pop out of nowhere while speaking.
Both brain and LLM neural nets model all kinds of concepts for producing those.
Their reasoning is based on neural nets that are modeling all kinds of concepts and relationships.
Have you watched the news lately?
Just a more robust Roomba...
#AICon
https://bsky.app/profile/willgervais.com/post/3lrv44lw3nc2a
Thank you for this, super useful!
Human brains process, synthesize, and distill knowledge. We can glean wisdom from our experiences. We can use logic. In short, we think.
A chatbot has no brain. It can't know, understand, or reason about anything. It cannot think.
they SHOULD be great at those! it's a small domain where all the problems are massively overrepresented in endless tutorials and practice material!
But then I use my human brain to go through what it gives me, clean it up, and turn it into my voice. It's a tool, not a god
1/..
2/2
Even a small child can learn what it does NOT know. That is a major part of intelligence that these machines do not--and will never--have.
https://bsky.app/profile/nicholdav.bsky.social/post/3lrxme7bymc24
even if you set literally everything else aside, LLMs still cannot think because they don't have bodies and sensations.
https://www.sciencedirect.com/science/article/pii/S0001691821000263
That's like saying (incorrigible) toddlers will never take over society. They will mature (and become maga!).
I'm concerned with the potential of quantum computing ai. Call it vaporware. Call me paranoid.
Chatbots are slap jack. Q-ai is the Glass Bead Game.
https://bsky.app/profile/xkcd.com/post/3lrxgvl677k2w
In terms of it all being probabilities you're right but that's the point. All the information contained in your head is encoded via probability distributions. There's a reason entropy and information are linked.
Thank you!
The biggest problem though is that it's horrible for the environment and what we're getting from it isn't worth the damage it's doing. Even if I find it really fascinating. 1/
1: The advancements in medicine are themselves worth a lot
2: If we would actually commit to green energy instead of waving it like a corporation at a pride parade, that would help a lot
3: I think it is going to collapse capitalism. I want capitalism to collapse.
It'll eventually rain out somewhere but not where it was used and I believe a lot ends up somewhere that's not useful like in the Ocean. 1/
Like, why are old photos in black and white? Because the world existed only in black and white until the late 50s.
Obviously you have researched the way LLMs work to an extent that gives you authority to shape other people’s mind about them.
What are you designed for?
LLMs are explicitly optimized to answer factual questions. Instruction tuning (RLHF) on factual QA datasets; Retrieval-augmented generation; Integration with external databases.
Answering factual queries is a core design goal, not an accidental side effect.
DO NOT GO TO LLMs LIKE ChatGPT FOR INFORMATION.
Do not go to LLMs like ChatGPT for information.
n 6-sided dice are very likely to produce n x 3.5 when rolled, especially if n is very big, but the dice are not storing that number in any sense
The more LLMs repeat what's currently written down...
Who would possibly have believed Copernicus in the age of ChatGPT???
the world is flat.
With the implication that I might have sold myself to the bad guys.
“AI” is a threat, but it’s a cognitohazard, primarily. Using it will make you more stupid and trigger mental health issues.
it’s a *popular* but factually erroneous oft *repeated* joke that they have numerous pinions offset by one.
However, Ravens & Crows both have the common set of 10 pinions …
Expectations for what this tech can accomplish are being over hyped by the companies pushing it.
Yikes that consensus was won with this very untrue factoid.
Companies that make them cultivate the popular misunderstanding that they know facts. This is good for business, and, not least in for authoritarians, good for persuading and controlling people. Outsourcing human judgement, as Kapferer wrote about in 2001 and Counting.
Now we have a system that has a strong tendency to agree with YOU specifically— the temptation to abandon reality for make-believe is strong, the path is easy, and risks vague.
Google’s “AI” summaries atop search results is a joke.
Please take yours off.
And no, state capitalism isn’t the same thing as socialism. Further, China has vast wealth inequality and homeless.
GPT at first answer: George Martin gave more time to Lennon McCartney but he was always professional to Harrison;
GPT after 5 min questioning: You are correct, Martin did bully Harrison 🤦🏻😂🤯
It's dumb even to expect a trustworthy answer
The average human KNOWS about 40,000 words and can conceptualize about 40,000! a number beyond calculation (try it)).
The manual approach builds more character, maybe? 😆
(And everyone who said learning cursive is obsolete is gonna regret it when their hand cramps up after the 3rd page of printing!)
“I think” “I actually have no idea but maybe” “I’m pretty sure that” “My guess would be”
If you were taught in school that America was founded in 1890 and someone asks you when America was founded, you'll probably say 1890.
Just like any other tool, it's only as good as its operator.
GIGO applies here.
Some people don't have time to carry out tasks that can be automated.
Write a research paper with it? Nope.
Mundane business shit: fuck yes.
While also giving their controllers the chance to manipulate information.
Indeed, LLM chatbots are one of the very few areas of ML where you can get away with not doing uncertainty characterization.
Watching Neuro Sama play Shotgun Roulette illustrates this point perfectly:
She CAN count live and dead rounds, but can’t base decisions on that information.
Tech will ALWAYS choose the easy answer. Always. (OK, ALMOST always).
A little zero-gravity fur floating everywhere
Walks that require a space suit
Occasional howling at actual moons (not just Earth’s)
And feeding them anti-grav treats at precisely lunar dusk
1. It's much easier to invent lies and make them attractive, so in terms of volume, they overflow reality.
2. Cleaning scraped information is very labor intensive, so companies have no incentive to do it.
Combined: LLM's will tend to lie systematically.
It's a battle of words
The poster-bearer cried
Listen, son
Said the man with a gun
There's room for you inside.
- Pink Floyd
...even if it's in vain.
https://bsky.app/profile/timhenke.bsky.social/post/3lrv2fzpu4s2z
Thank you for articulating this.
So far as much as I can understand chatbots do not understand sarcasm, metaphor, or implied meanings.
These writing tools are of great importance in conveying deep meaning.