I'm excited to share our new paper (with @alexanderhuth.bsky.social) on transferring language decoders across participants and modalities! authors.elsevier.com/a/1kZRD3QW8S... 1/5 - ThreadSky

I'm excited to share our new paper (with @alexanderhuth.bsky.social) on transferring language decoders across participants and modalities!

https://authors.elsevier.com/a/1kZRD3QW8SA1XX

1/5

Comments

mayalhenry.bsky.social•25 days ago

Congrats @jerrytang.bsky.social and @alexanderhuth.bsky.social !

jerrytang.bsky.social•25 days ago

Language decoders that target semantic representations have the potential to help people with aphasia, who struggle to map concepts to lexical-phonological output. But many people with aphasia have impaired language comprehension, and current decoders are trained on brain responses to language

2/5

jerrytang.bsky.social•25 days ago

Our new approach can decode language from a participant without language data! First we train decoders on reference participants with language data. Then we use *silent movies* to align brain responses across participants. Finally we decode new participant responses using the reference decoders

3/5

jerrytang.bsky.social•25 days ago

We tested our approach on neurologically healthy participants, and found that silent movies are nearly as effective as narrative stories for transferring semantic decoders. Scientifically, this adds to the growing evidence that semantic representations are shared between language and vision

4/5

jerrytang.bsky.social•25 days ago

Practically, our approach may enable us to adapt our semantic language decoders for people with impaired language comprehension. I’ve been working with
@mayalhenry.bsky.social and
@alexanderhuth.bsky.social
to test our approach in people with aphasia, stay tuned!

5/5

beeostrowsky.bsky.social•25 days ago

This sounds really cool, but I was a linguistics major in the early 1990s and I don't think we had "trainable decoders" back then. I'm trying to explain this to myself like I'm five. Are you training computers to help people talk by watching what their brains are doing?

Comments

Posting Rules

Reply