What is a probing task that is purely about semantics? Context: I have a probe trained to predict dependency relations, and would like to train another one on a semantics only task (for research purposes) - ThreadSky

About ThreadSky

shikharmurty.bsky.social • 96 days ago

What is a probing task that is purely about semantics?
Context: I have a probe trained to predict dependency relations, and would like to train another one on a semantics only task (for research purposes)

Comments

mrdrozdov.com•96 days ago

How about SRL ;)

Maybe topic modeling?

shikharmurty.bsky.social•96 days ago

SRL probe still rewards hidden states that model dependency relations, no? would like a probe thats agnostic to how well the underlying network models syntax

mrdrozdov.com•96 days ago

Indeed, hence the wink!

nsaphra.bsky.social•95 days ago

I would do it as a topic task. The way to construct a topic task varies depending on the model, but you could do it by selecting the rarest token in a document and training the probe to reconstruct that rare token

shikharmurty.bsky.social•95 days ago

for more context: we are training the probe on sentences from PTB / BLIMP

intractable.bsky.social•95 days ago

I think Semantic Parsing would be a good option, although to which representation probably depends on your requirements.
https://arxiv.org/abs/2206.10668 seems to have both side by side.

shikharmurty.bsky.social•95 days ago

thx for sharing, though semantic parsing almost certainly benefits from modeling syntax :)

intractable.bsky.social•95 days ago

I am really curious to see what semantic task does not require modeling syntax.
I would expect a non-lexical abstract semantic task, maybe something exists in the corners of theoretical semantics where we stop thinking about forms and only work with features.

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply