There's a known bug in how we compute "word" probabilities with subword-based LMs that mark beginnings of words -- as pointed out by Byung-doh Oh and Will Schuler, & @tpimentel.bsky.social and Clara Meister

I'm pleased to announce that minicons now includes a fix which runs batch-wise!
1 / 2
Post image
Post image

Comments