dcporter.bsky.social
Qing historian, résidant à Montréal
134 posts
674 followers
378 following
Regular Contributor
Active Commenter
comment in response to
post
Yeah, really seems that way :(
comment in response to
post
The work it does is irreplaceable and destroying it would be an immeasurable loss
comment in response to
post
To their credit(?), at least it seems (based on playing around with it) like Copilot is very bad at writing academic essays of the sort that I assign. So hopefully students will learn quickly not to use it.
comment in response to
post
What can Copilot do, according to what McGill tells its students: "help you write [...] essays"
comment in response to
post
Yeah, obviously the proper year is Xuantong 117 (assuming I haven't butchered the math) - the last legitimate imperial reign period, which is the way that the traditional Chinese calendar counted years!
comment in response to
post
It has a distinctive style and it will usually make substantial mistakes with citation (or fail to do it at all). But there's no point focusing on spotting it because it's impossible to prove to the satisfaction of a disciplinary committee, so I mostly just give mediocre grades and move on.
comment in response to
post
Utter nonsense - it's basically precisely the opposite of correct (the percentage Manchu of the banners was pretty low in 1644, and increased substantially in the 18th century due especially to Hanjun expulsion).
comment in response to
post
This explanation of Tüsheet Khan (which also appeared in one of the earlier screenshots) is bonkers
comment in response to
post
I admit to not having read this book, but I'm basically certain this description is also untrue
comment in response to
post
Here the book exists but the chapter is invented (this is also the case for the supposed Ding Yizhuang chapter in the Luo Xin edited volume that I included earlier):
comment in response to
post
Some references it generates are real, but are inaccurately described (Orphan Warriors does not deal with ujen cooha except in passing - it is focused on a family of Manchu bannermen)
comment in response to
post
comment in response to
post
It also hallucinates references when asked to supply them:
comment in response to
post
In another part of its answer, it seems to confuse banners that were part of the Eight Banner system with the territorial-administrative unit called banners that were used to govern Mongol lands (to be fair, probably 75% of my professional colleagues in Qing history are also not clear on this point)
comment in response to
post
After I corrected it, it doubled down on the idea that hūwaliyasun means league and tob means banner (though it acknowledged the meaning of the full phrase). After I corrected that, it invented a banner category called "Nikan gūsa" for the Han Eight Banners, which were actually called ujen cooha
comment in response to
post
But at a more serious level, it would really be great if people just stopped treating these models as sources of evidence. There are clearly ways to use them that are extremely helpful (especially for programming, data science, etc). But they're tools, not oracles.
comment in response to
post
This is quite possibly true. I certainly am no fan of the proliferation of ChatGPT et. al. as the source of all information. I've said a few times tongue-in-cheek that DeepSeek is great for me as a Chinese historian because my students can't use it to write their essays.
comment in response to
post
But other things only get assessed after the whole answer is completed
comment in response to
post
Yeah, I think some keywords - most notably names of all post Deng CCP leaders - are just blocked entirely. If you mention them, it won't answer, if its answer mentions them, the whole thing gets deleted as soon as the name is generated.
comment in response to
post
When I asked it in Chinese...
comment in response to
post
Similarly, when I asked it about the concept of One Country Two Systems in English, it volunteered this (as part of a longer answer):
comment in response to
post
And its answer when I asked it who Chen Quanguo is was a lot less interesting than its answer in English. It basically just recites his job titles and talks about the important contributions he has made to social stability, state security, and anti-terrorism work while he was in Xinjiang and Tibet
comment in response to
post
Though maybe it works better in Chinese (I asked it to translate its last response - the one about the UN Human Rights Commissioner Report - into Chinese. It generated about a sentence then broke off, deleted everything, and gave its generic change the subject message):
comment in response to
post
Yep, much of the censorship can definitely be circumvented if you avoid certain key terms
comment in response to
post
Similarly, I got it to tell me (without later deletion) that Xinjiang became part of China in the 18th Century, but when I asked a follow-up, it deleted the long response it initially generated.
comment in response to
post
I think this is closer to what is going on. Though there is then a third category, which is questions where it immediately refuses to respond (for instance, if asked anything about Xi Jinping or other recent Chinese leaders, even totally neutral questions like "who is Xi Jinping").
comment in response to
post
comment in response to
post
Moreover, when asked follow-up/related questions in the latter situation, it repeatedly reproduces nearly identical verbiage. See for instance:
comment in response to
post
What's interesting is there are two different forms of censorship mechanisms visible to the user. In one, the LLM generates a (seemingly un-censored text) then erases it all and says it can't answer. In the other, it produces a politically acceptable answer from the beginning
comment in response to
post
What explains the highly politically aware answers like these?
comment in response to
post
Though note that contrary to what DeepSeek claims in the screen recording I took, the Ordam Padishah mazar has been reported destroyed