Thoughts on how you handle model-based outlier when it is a massive dataset and as you remove outliers new ones appear? After 3 rounds 2 of the models finally don't have outliers, but some are still quite high (>200).... - ThreadSky

dpmoriarity.bsky.social • 3 days ago

Thoughts on how you handle model-based outlier when it is a massive dataset and as you remove outliers new ones appear?

After 3 rounds 2 of the models finally don't have outliers, but some are still quite high (>200)....

Comments

ajshackman.bsky.social•3 days ago

Or robust fit, after excluding nonsensical outliers

dpmoriarity.bsky.social•3 days ago

was thinking of glmm but other indicators aren't pointing me in a specific direction,

i haven't played around with this package for robust mixed modeling though, will have to take a look tomorrow

peakhttps://cran.r-project.org/web//packages/robustlmm/vignettes/rlmer.pdf

seanpmackinnon.bsky.social•3 days ago

This is probably the easiest defensible choice imo, the robust regression with rlmer() and leave in all the data.

Better than playing outlier whack-a-mole, at any rate

kevinmking.bsky.social•3 days ago

Don't remove outliers (absent compelling information) is one strategy

solomonkurz.bsky.social•3 days ago

Agreed. Rather than dropping data values because they don't fit the model well, consider finding a better model for your data.

solomonkurz.bsky.social•3 days ago

Take this as unsolicited life advice, too

jfeltphd.bsky.social•3 days ago

I agree with all of this. Outliers are really individuals in your sample that come from a different population than you intended to collect, not necessarily extreme values.

dpmoriarity.bsky.social•3 days ago

Solomon is getting at the part of my thought process that I forgot to include in the original post + added to the quote tweet- what to do instead to not have this issue that isn't endlessly removing data 1/2

dpmoriarity.bsky.social•3 days ago

Like, don't necessarily want to transform cuz its another data manipulation.

But the posterior predictive checks r fine so glmm iisnt necessary

maybe see if there is a random effect that might help?

solomonkurz.bsky.social•3 days ago

Back to this, then: https://bsky.app/profile/solomonkurz.bsky.social/post/3lil37z65622k

drandreahoward.bsky.social•3 days ago

In a massive dataset it’s important to remember that the bar for “outlier” is different. If I have 10,000 observations, two dozen of them *should* be 3SD away from the mean.

Comments

Posting Rules

Reply