NEW PREPRINT A detailed overview of 32 popular predictive performance metrics for prediction models arxiv.org/abs/2412.10288 - ThreadSky

maartenvsmeden.bsky.social • 74 days ago

NEW PREPRINT

A detailed overview of 32 popular predictive performance metrics for prediction models

https://arxiv.org/abs/2412.10288

Comments

stefanherzog.bsky.social•74 days ago

@maartenvsmeden.bsky.social : Do you know of a similar review for more open-ended tasks in medicine, in particular, for a differential diagnosis list (e.g., in a GP or ER setting)?

stefanherzog.bsky.social•74 days ago

Most of the resources on evaluating lists I found come from information retrieval research (e.g., evaluating search engine results).

Thanks!

maartenvsmeden.bsky.social•74 days ago

Excellent question. Unfortunately I do not but maybe @benvancalster.bsky.social has some ideas about this

benvancalster.bsky.social•74 days ago

I don't know...

jrgant.bsky.social•74 days ago

📌

whitneyepi.bsky.social•74 days ago

📌

maartenvsmeden.bsky.social•74 days ago

Many thanks to @benvancalster.bsky.social for leading making this elaborate overview. W/ @gscollins.bsky.social @vickersbiostats.bsky.social @laurewynants.bsky.social @lasaibarrenada.bsky.social @kdpsingh.bsky.social and many others

benvancalster.bsky.social•74 days ago

And @gaelvaroquaux.bsky.social!

maartenvsmeden.bsky.social•74 days ago

My favourite thing about the paper is this summary table

maartenvsmeden.bsky.social•74 days ago

Forgot to tag @gaelvaroquaux.bsky.social

dszlosek.bsky.social•74 days ago

I wish more people used Integrated Calibration Index, very informative (granted the full plot is even better)

vickersbiostats.bsky.social•74 days ago

Table 4 way superior!!

maartenvsmeden.bsky.social•74 days ago

Ok, ok. I like them both

fayenorby.bsky.social•74 days ago

📌

f2harrell.bsky.social•74 days ago

I question the primacy of AUROC since practitioners will be tempted to compare the AUROC of two models, something it’s not powered for.

maartenvsmeden.bsky.social•74 days ago

There is a pretty extensive description of the AUROC in the paper, including that it is "just" semi-proper. We also specifically state that model comparisons should use proper metrics and are not the focus of this paper. Hope that you agree with this :)

f2harrell.bsky.social•74 days ago

Definitely. I think the paper's summary needs to be edited to reflect that.

thomas-speidel.bsky.social•74 days ago

📌

statsepi.bsky.social•74 days ago

cc @sschandelmaier.bsky.social

sschandelmaier.bsky.social•73 days ago

Great! Will include it in @lightsdatabase.bsky.social

statsepi.bsky.social•74 days ago

cc @lizzlizz23.bsky.social

uriahf.bsky.social•10 days ago

Amazing work!

I disagree about PPV/NPV:
For a given resource constraint and assuming no treatment-harm, PPV is the most suitable metric as far as I understand.

You might also think of it as another version of NB for resource constraint

https://pmc.ncbi.nlm.nih.gov/articles/PMC10018264/

Comments

Posting Rules

Reply