I wouldn't be surprised if, given enough resources (tokens in context, sources, good fine-tuning, etc), these models could be pretty impressive assistants for historical work. We can't possibly match the erudition of these things. This line of criticism is bound to fail sooner or later. However.../2 - ThreadSky

oharajrm.com • 32 days ago

I wouldn't be surprised if, given enough resources (tokens in context, sources, good fine-tuning, etc), these models could be pretty impressive assistants for historical work. We can't possibly match the erudition of these things. This line of criticism is bound to fail sooner or later. However.../2

Reposted from Simon Willison

"The leading AI models are now very good historians" - by Benjamin Breen, a history professor at UC Santa Cruz resobscura.substack.com/p/the-leadin...

Comments

thiagokrause.bsky.social•32 days ago

Well, so far they aren’t very erudite. Any historian knows his specialized literature than ChatGPT because they can’t output very niche knowledge. I don’t think that’s what they will be useful for.

oharajrm.com•32 days ago

That's in the "given enough resources" part. We're not there yet (e.g., models with huge token contexts are appearing only now). It's just that their ceiling is much higher for erudition than any human.

thiagokrause.bsky.social•32 days ago

I kind of think only AGI will be able to outcompete us on erudition. Academic fields are just too obscure for LLMs to be able to predict text accurately. Not enough training data, subsumed against too much other stuff.

oharajrm.com•32 days ago

No, I don't think so. Erudition is downstream from the general language, so it's a problem of processing the input - it processes the literature, extracts info, makes the connections, etc. What this requires is the capacity to "hold" a lot of of the input and output at once (the token context).

thiagokrause.bsky.social•32 days ago

I think they are already useful in the way Breen is using them - you provide it with the input (pdfs,
Images, etc) and then it uses them to generate an answer that might be slightly helpful. But to expect it to be able to actually pull up obscure references seem too much for the foreseeable future.

oharajrm.com•32 days ago

Oh, but I don't think it would pull from nothing. The user would need to point to a folder with data (pdfs etc) first. It's what Breen is doing, but at larger scale (which current models can't do). Access to the internet may help with obscure refs, but then we get into the problem of...

oharajrm.com•32 days ago

The problems that emerge specifically for historical work are orthogonal to erudition, and have more to do with reasoning itself (e.g., what kinds of anachronism are permissible and useful, background theories, etc.) Erudition is the low-hanging fruit.

thiagokrause.bsky.social•32 days ago

The “reasoning” models like o1 will likely be useful when they allow us to combine it with attaching PDFs, “projects”, and searching the internet. So far it is less useful than 4o.

oharajrm.com•32 days ago

Of course, right now, the Adobe assistant can't even do a good summary of a pdf. It's just that these tools can, in principle, come to be a "portable library" and a decent assistant. (Ethics and everything else aside)

oharajrm.com•32 days ago

... I'd say that of the "bottlenecks" historical research has, brainpower is definitely not one of them. (Just see how many PhDs are awarded each year.) So in this sense, it's a solution waiting for a problem. Better infrastructure and funding would be far more impactful. Alas.../end

rheinze.bsky.social•32 days ago

What’s really annoying about this is that the Digital Humanities have *already* worked years on better transcription tools that are *far* more powerful with barely legible handwriting (not the very easy and clear stuff he uses), big data specialised search tools etc.

oharajrm.com•32 days ago

That's true, but I'd say if you can get an integrated suite that does transcription and answers a few questions, that's an advance. What's annoying is that the whole hype blows everything out of proportion. I hate that too.

Comments

Posting Rules

Reply