Honey, we fixed Signal Detection Theory (SDT)! In this preprint, Constantin Meyer-Grant, David Kellen, Sam Harding, and I critically evaluate the (unequal-variance) Gaussian SDT model in recognition memory and pursue the Gumbel-min model as a principled alternative: doi.org/10.31234/osf... 🧵 - ThreadSky

singmann.bsky.social • 32 days ago

Honey, we fixed Signal Detection Theory (SDT)! In this preprint, Constantin Meyer-Grant, David Kellen, Sam Harding, and I critically evaluate the (unequal-variance) Gaussian SDT model in recognition memory and pursue the Gumbel-min model as a principled alternative: https://doi.org/10.31234/osf.io/qhrfj_v1
🧵

Comments

gidon-frischkorn.bsky.social•31 days ago

Super nice work! I saw that you implemented all models using custom families in brms, so I wondered if you would be interested in including these implementation in the bmm package. We have already thought about some SDT implementations, but have not yet finalized the implementations.

gidon-frischkorn.bsky.social•31 days ago

In particular, we also aimed to implement SDT models with both Gaussian and Gumbel_min noise, for many of the reasons that you have outlined in your paper. So it would be super nice to make these models as accessible as possible to broaden their application.

singmann.bsky.social•31 days ago

The code is freely available (https://github.com/singmann/gumbel-reanalysis), so feel free to include it in your package. I think the main question is how to hand-over a multinomial response to brms. It is trivial if data is disaggreagted, but less so with aggregated data that can speed-up fitting considerably.

gidon-frischkorn.bsky.social•31 days ago

Nice, the problem of passing aggregated data for a multinomial reponse is something we solved already for the Memory measurement model that is included in the development version of bmm. So, it sounds like that should work for the SDT models, too.

singmann.bsky.social•31 days ago

Interesting, I should check your solution out then. Maybe it comes with a nicer interface than my solution.

gidon-frischkorn.bsky.social•31 days ago

Yes definitely, the relevant code is here: https://github.com/venpopov/bmm/blob/develop/R/model_m3.R

have a look at line 269 ff.

Basically, you have to convert your variables collecting the response frequencies of the different response options into a matrix and store this matrix as a single variable in your data frame.

achetverikov.bsky.social•32 days ago

Interesting stuff. I wonder why you call it principled though? Does is represent normative behavior?

singmann.bsky.social•32 days ago

Because the Gumbel-min model implies a unique behavioural principle that is beautifully confirmed by the data: invariance to choice set-size expansion for a detect new task, while simultaneously predicting increased accuracy with choice set size for a detect old task. https://bsky.app/profile/singmann.bsky.social/post/3lnsil3ex4m2m

singmann.bsky.social•32 days ago

The Gaussian model can only predict invariance for both tasks or increase for both tasks, but not the differential behaviour that is shown in the data.

achetverikov.bsky.social•31 days ago

Thanks for clarifying! I guess I'm just not used to seeing the word principled when ppl talk about descriptive models, even though they can predict data. One more thing I wonder about: if this is how the signal distributions look like, should we see them directly with continuous adjustment tasks?

singmann.bsky.social•30 days ago

Not sure how such a task would even look like in the context of recognition-memory as covered in the paper.

achetverikov.bsky.social•30 days ago

Well, hypothetically memory is not based on the task, so whether you do old-new on, say, colors, or ask ppl to indicate the color shown on a color wheel, the memory signals are the same. And with Gaussians, you can easily draw a link between the performance on mAFC, yes-no and adjustment.

adamosth.bsky.social•31 days ago

This is extremely interesting and thorough work! But I’m wondering what kind of retrieval mechanisms could actually lead to a Gumble min model in practice?

Most retrieval models use some kind of summed similarity computation, which naturally leads to approximately Gaussian distributions

singmann.bsky.social•31 days ago

We do address this issue a bit in section "Why Minima?" on pages 24 to 25.
Additionally, I know that Ven Popov has a working computational model that produces Gumbel-min evidence distributions. So for more details you might have to wait for his manuscript or email him.

singmann.bsky.social•32 days ago

SDT is a cornerstone of recognition memory research, primarily assuming Gaussian distributions – a choice based more on tradition than necessity. The standard model assumes two equal-variance distributions, allows calculating d′ from a pair of hits and false alarms, and predicts symmetric ROCs.

mzloteanu.bsky.social•30 days ago

very cool!

singmann.bsky.social•30 days ago

Thanks!

singmann.bsky.social•32 days ago

In recognition memory, ROCs are typically asymmetric, which requires Gaussian distributions with unequal variance. One problem with the unequal-variance model is that it predicts below chance performance for items with very low familiarity (i.e., studying makes some items less familiar).

singmann.bsky.social•32 days ago

We consider an SDT model assuming Gumbel-min (i.e., minimum extreme-value) distributions. The Gumbel-min model avoids the problrms of the Gaussian model, predicts asymmetric ROCs assuming equal variances, and allows calculating measures of discriminability and response bias, g′ and kappa.

singmann.bsky.social•32 days ago

The Gumbel-min model implies a behavioural principle: the probability of choosing a new item remains constant as choice sets grow. An experiment confirms this principle with constant accuracy for new item detection (2M-min). For old-item detection (2M-max), accuracy increase with choice set.

singmann.bsky.social•32 days ago

We compared the descriptive performance of both models across 35 datasets from four different recognition memory paradigms. The Gumbel-min model fits the data nearly as well as the Gaussian model. Once model complexity was penalized via AIC, the Gumbel-min model matched or outperformed the Gaussian.

singmann.bsky.social•32 days ago

We also evaluated the predictive performance of both models: Identify a relevant portion of the data that that can be omitted. Leave it out, and test each model’s predictions against it. Across all predictive tests, a pattern was clear—the Gumbel-min model outperformed the Gaussian.

Comments

Posting Rules

Reply