Okay strap in, we are about to go deep here. Lets talk about why LLMs are, in general, bad at working with specific questions about history but also give seemingly solid answers to those questions. There are real limitations to what LLMs can do, and understanding that is really important. - ThreadSky

historyofthesecondworldwar.com • 16 days ago

Okay strap in, we are about to go deep here. Lets talk about why LLMs are, in general, bad at working with specific questions about history but also give seemingly solid answers to those questions.

There are real limitations to what LLMs can do, and understanding that is really important.

Reposted from Vijay Krishna | Free Palestine 🍉

Comments

ivijay.bsky.social•16 days ago

I wasn't trying to really, was just curious at the high number, I'm not conversant with that history

ivijay.bsky.social•16 days ago

Thank you!

historianmike.bsky.social•16 days ago

One might add to this that the crafting of the original prompt itself sets OP up for missing the point.

There is probably a nuanced historical argument that, by 1945, the scope of Japanese atrocities in China had significantly slowed and that the 200k number is an overestimation. Perhaps.

historianmike.bsky.social•16 days ago

And so the LLM tells OP the answer to *exactly* what he asks. But even if the answer provided were well sourced and accurate, its a smaller question than what OP may have intended. How many non-Chinese were also murdered? What is the scale of atrocity overall in 1945?

historianmike.bsky.social•16 days ago

As I posted in another thread, ~100k Vietnamese likely starved to death every month in 1945. Plus civilian casualties in the Philippines, atrocities in Korea, etc. But the focus on China limits the LLM to a very narrow answer.

historianmike.bsky.social•16 days ago

Maybe OP only cared about China. Probably they wanted to debunk the specific 200k number to delegtimize the other argument. That would be harder to do with a fuller context. But the LLM's inability to think beyond the question you ask it allows the user to inject bias to get the results they want.

historyofthesecondworldwar.com•16 days ago

Great point. Because it is just providing an answer to what you ask, it is easy to pigeon hole your answer without even knowing that you are falling into that problem.

historianmike.bsky.social•16 days ago

One quote I think of a lot with LLM users like this, "if you knew what you were talking about you would realize how wrong you are." I can look up an answer to an astrophysics question, but because I have 0 base knowledge the answer is meaningless to me. Even if it makes me feel like I'm smart.

historyofthesecondworldwar.com•16 days ago

First up, this question, when asked directly to Gemini will result in the LLM's inference engine being fired. This is the "letter predictor" that you might see LLM's described as. Gemini may use a web search in these situations, but it does not expose that to the user. It also does not cite sources.

historyofthesecondworldwar.com•16 days ago

If you then ask it for sources, the answer is generally the same, but the only sources mentioned are about the Nanking Massacre, which happened many years before.

historyofthesecondworldwar.com•16 days ago

The Second problem is that Gemini only has access to sources that are readily available online. To get a good understanding of what those sources might be, you can use use Google Deep Research to see what it will pull from to try and give a more in depth answer. https://docs.google.com/document/d/1APDj3m-A5Oh9n-NhgbwHpN2OXHMJwog03rVcP3MZUDk/edit?tab=t.0

historyofthesecondworldwar.com•16 days ago

Unsurprisingly the sources for that answer look a lot like what you might get from a Google Search with this question. But what you won't see here is any actual academic texts, because those often are not available online. Ironically these things respect (some) copyright.

historyofthesecondworldwar.com•16 days ago

The conclusion to the Deep Research request is the same as the original Gemini response, which probably makes sense given the fact that really Deep Research is just giving an avenue to cite where the information is coming from, and then being far more verbose with the response.

historyofthesecondworldwar.com•16 days ago

ChatGPT provides a very similar response when asked this question either with or without Search turned on. But I find it interesting that it provides this closing line, but it simultaneously unable to provide the "well-documented historical research" it used for its answer.

Comments

Posting Rules

Reply