I don't get it. They gave different groups a big pile of field data and asked them to draw conclusions. Groups drew different conclusions, so ecology is broken? That's not how we do ecology. We design an experiment based on domain expertise and *collect appropriate data* with the question in mind. - ThreadSky

carlbergstrom.com • 89 days ago

I don't get it.

They gave different groups a big pile of field data and asked them to draw conclusions. Groups drew different conclusions, so ecology is broken?

That's not how we do ecology. We design an experiment based on domain expertise and *collect appropriate data* with the question in mind.

Comments

grimalkina.bsky.social•89 days ago

I truly don't understand why these sort of things seem to have zero theory of human behavior. Like the very idea that the ideal state of science is people approaching things exactly the same way is an AI slop version of science. Point is surely to know and trust what diff approaches are and yield

devezer.bsky.social•89 days ago

And of course not all approaches will be valid depending on the theoretical constraints on the phenomenon which may or may not be observed by many of these models. Assuming every inference is equally plausible or meaningful is another pitfall here. This doesn't emulate the actual research process.

grimalkina.bsky.social•89 days ago

And likely (from at least I hope a bit more complex human behavior theory than where such work starts) that networks of diverse attempts, correction & collaboration and yes, even failure and error in application-when learned from-are necessary to advance such a library of known constraints

grimalkina.bsky.social•89 days ago

(not to explain what you already think so much about but just to think out loud for myself here!)

mmilkowski.bsky.social•89 days ago

I wish they'd read Breiman on the Rashomon effect.

devezer.bsky.social•89 days ago

Oh now I need to do that as well!

devezer.bsky.social•89 days ago

Ah it's from The Two Cultures! I read that multiple times but somehow didn't remember this term.

avisokay.bsky.social•47 days ago

Preprint out (later than I hoped but here it is 😊) https://osf.io/8zybt_v1/

avisokay.bsky.social•89 days ago

I am writing about exactly this right now. Draft on arxiv soon 👀

mmilkowski.bsky.social•89 days ago

Being a fan of Kurosawa, I immediately fell for it. ;)

willgervais.com•89 days ago

Sometimes the many analyst thing feels like replications of “poorly formed hypotheses and iffy designs yield ambiguous conclusions” and….yeah

alxfed.bsky.social•88 days ago

The prevailing dogma among the people who use the word 'science' but don't know what it means... is, that there is _always_ one 'Truth', the rest is 'Falsehood'. They have no idea about the problems with multiple solutions, multiple _diverging_ solutions, multiple infinitely branching solutions...

derek--beaton.bsky.social•89 days ago

having participated in one of these a while back, I was (pleasantly) surprised to see that, statistically, things were pretty aligned. more commonality than I expected before hand. I really expected more variation.

it was then the interpretations that surprised me more and diverged

lvulis.bsky.social•89 days ago

Pure empiricism and lack of understanding of how/why data are collected w.r.t. a theory of understanding to begin with

epbwrisci.bsky.social•89 days ago

I’ll confess to be a bit meh. That’s what review is for. And debate. And further experimentation and study. And, going on, adaptive management, etc.

If the data were collected with a particular model/analysis in mind, six the follow up free for all have that context?

avisokay.bsky.social•47 days ago

In this new pre-print we address the many analysts problem and describe how the Rashomon effect offers a helpful perspective. Model multiplicity leads to many explanations! https://osf.io/8zybt_v1/

jarome.bsky.social•89 days ago

But, Carl, knowing the slightest about what you're analysing is so overrated.

jbakcoleman.bsky.social•89 days ago

You jest and yet my mentions.

gmbowen.bsky.social•89 days ago

I think what would have been interesting, study wise (because I find their actual findings unremarkable & unsurprising) would be to give the analyses of different groups to other groups & see how that feedback about the practices of others might inform what they did, if they change their analyses.

ufeest.bsky.social•89 days ago

You are my people 😻

stevenllancaster.bsky.social•89 days ago

I believe the point is that even with expertise, different researchers are going to make different analytical decisions. These series of studies show that those analytical decisions, any one of which may seems minor and/or defensible, can have a huge impact on the findings.

pj-theblack.bsky.social•89 days ago

Kinda feels a bit like this (admitted haven't read the full article yet tho):

doctorzen.net•89 days ago

Large shared datasets are being more common, though. It seems worth thinking about the decisions made in analysis, how they are justified, and how we arrive at conclusions just in general?

jrossibarra.bsky.social•89 days ago

Agree with both of these arguments!

stellarsdreams.bsky.social•89 days ago

This study has a bias built into it, pre-supposing big egos supplanting genuine science is what's happening, which makes their research papers questionable. Yes, there are egos, which is why you have peer review. Click money should not trump unbiased writing.

trevorabranch.bsky.social•89 days ago

I get what you are saying. But what was striking to me was that every study got a slightly different answer, it wasn't like there were 50/100 studies that got the identical answer and 50 with a wide spread. [At least how it was presented in the paper]

gsjphd.bsky.social•88 days ago

The end result of "just gather up all the data and throw them into an analysis, more data = righter answer" trends towards David Peters's """Large Reptile Tree""".

kevin.crime.group•89 days ago

Doesn’t this just show why p-hacking doesn’t work? “Here’s a big pile of data, find something”

ewb.io•89 days ago

p-hacking can definitely happen at the data collection phase as well.

i think this study just demonstrates the level of uncertainty in those datasets and you could consider the distribution of conclusions indicative of that.

kenaizawa.bsky.social•89 days ago

And if there are disagreements, don’t scientists try to do more experiments to try to settle things?

peebeejaybee.bsky.social•89 days ago

BREAKING: "Science" journo shows they don't understand science. Again.

More at 10...

mountainrangifer.bsky.social•89 days ago

Isn’t this the textbook definition of data dredging?

pawellenartowicz.bsky.social•89 days ago

Probably not. Data dredging usually refers to searching for patterns in data, like spurious correlations, not to creating more or less good models to estimate some relation.

mountainrangifer.bsky.social•89 days ago

Without a fulsome understanding of the thing you are studying and the data set how can you a) understand the hypothesis b) have an understanding of the limitations of the data. Without those two things I think you are left only to ‘search for patterns in data.’ And if you search you will find.

Comments

Posting Rules

Reply