I try not to dunk tweet but this is fucking awful. I am a historian (written two books of historical nonfiction), a librarian specializing in research skills, and an archivist. Even just thinking about using an LLM to do historical research makes my blood boil. - ThreadSky | a Reddit-style client for Bluesky

bookjockeyalex.bsky.social • 60 days ago

I try not to dunk tweet but this is fucking awful. I am a historian (written two books of historical nonfiction), a librarian specializing in research skills, and an archivist. Even just thinking about using an LLM to do historical research makes my blood boil.

Reposted from Karin Wulf

"Using LLMs for historical research requires careful attention and disciplinary expertise." Can't find it again but someone asked about AI and historical research, and someone else recommended this @cblevins.bsky.social blogpost -- indeed interesting & helpful. cblevins.github.io/posts/llm-pr...

Comments

lawrencecphd.bsky.social•60 days ago

Recently submitted an oral history/archival research project grant proposal for a fed agency. Got read the riot act over privacy, but they flagged our budget line for transcribing interviews, b/c “that could be done by AI.”

You’re worried about privacy/IRB, but want to feed interviews to a LLM?!?

bookjockeyalex.bsky.social•60 days ago

Jesus. I worked on oral histories as well. The amount of back and forth with the human transcriber was huge but worth it for the accuracy. I can't imagine the pain of trying to get an AI to know all local history terms and pronunciations and to understand mumbling or garbled language.

lawrencecphd.bsky.social•60 days ago

Exactly! It would have violated all their privacy rules, AND be time-consuming trash!

bookjockeyalex.bsky.social•60 days ago

And the privacy thing! We purposefully didn't make transcripts or audio of certain oral histories available online due to copyright, sensitive material, or lack of approval from the interviewee. To turn around and put that same material into an LLM! No!

backgroundn015e.bsky.social•60 days ago

"To err is human. To really muck things up, you need a computer."

Here's how AI handles a simple arithmetic problem:

Q:How long does it take a baseball, traveling at 104 mph, to reach the pitcher's mound on a line drive?

AI sez:

Comment image

midwesternmama.bsky.social•60 days ago

📌

mstiahmarie.bsky.social•60 days ago

One of my many jobs is to write on the certification rules for each state. The AI can’t even get that stuff correct and the information is on each state’s education department website.
Meanwhile, there is no way to turn the google one off if you use the search engine. Gah.

courtneymilan.com•60 days ago

ABSOFUCKINGLUTELY NOT. NO.

bookjockeyalex.bsky.social•60 days ago

Losing my mind over this

bookjockeyalex.bsky.social•60 days ago

Jist on a prwxtical pevel this sucks. Do you know how much historical material is not in the Giant Theft Machine because it hasn't been digitized or even catalogued online? If that's where you're starting, you're going to be looking at the ocean through a peephole.

bookjockeyalex.bsky.social•60 days ago

And that's not even touching on the wild inaccuracies an LLM will provide. If you are researching a topic, you don't know what's a "hallucination" or not. An LLM is literally just a predictive text generator! It is not a research assistant! My god.

bookjockeyalex.bsky.social•60 days ago

The amount of work you have to do to double-check EVERYTHING instead of going to already verifiable sources is going to exponentially increase your workload. Which is why no one does it. All those problems with junk chatgpt scientific papers is coming for the Humanities and we're helping it along.

bookjockeyalex.bsky.social•60 days ago

Thiiiiis. Archives are already critically underfunded and understaffed (and archivists critically underpaid) and instead of addressing that you're gonna throw millio s at a fucking datavcenter bc a historian can't be bothered to do their job. https://bsky.app/profile/wishda.bsky.social/post/3lgdqzav5ts2e

bookjockeyalex.bsky.social•60 days ago

Constantly reminded that a whole lotta people are very excited to treat an LLM like a slave. The way these people anthropomorphize this technology and then immediately undermine it. "Look! Now you can abuse a research assistant with no consequences!"
https://bsky.app/profile/nathankhensley.bsky.social/post/3lgdj3svxlk2q

wishda.bsky.social•60 days ago

This also ignores how digitalization can distort archival work.

A seemingly disparate set of documents in the same folder with shared handwriting can lead to discovery and insight.

“Why is this in her papers? Why is this in her handwriting?”

Lost context when digitized into files on a server.

fdlaramee.bsky.social•60 days ago

I made this exact point to my (digital history) students a week ago: you can trust an LLM's output *if and only if* you know exactly what you need it to yield, i.e., if you have already done the work. I guess I'll see whether the lesson stuck when the first assignments roll in.

kawulf.bsky.social•60 days ago

Cool I don't dunk post either! Responding to your full thread here-- if you're opposed to AI on ethical grounds, environmental or otherwise, understood. Methodologically though what I appreciated about this blog was attn to bias of the archive and the models. That's why use w expertise.

bookjockeyalex.bsky.social•60 days ago

Yes. Fully opposed (at least in terms of how genAI/AI is currently being developed and implemented). And it's so depressing to see people in my own field gobble this snake oil down and pretend like they're engaging in a methodological conversation.

kawulf.bsky.social•60 days ago

I'd say I'm opposed to lots about AI, but also in favor of understanding the tools and what they can/ mostly not do. Which is why I found this blog post interesting.

bookjockeyalex.bsky.social•60 days ago

I read that post, too, and found it uncritical and uninteresting. The writer fundamentally misunderstands what an LLM is designed to do and mistakes unintentional side effects for capability. I see this shit a lot in education. It's how we convince ourselves it's useful when it's not.

bookjockeyalex.bsky.social•60 days ago

If you are actually interested in understanding LLMs, this paper is a great place to start.
https://dl.acm.org/doi/10.1145/3442188.3445922

kawulf.bsky.social•60 days ago

Thanks.

davehogg.bsky.social•60 days ago

That's the worst idea imaginable.