Now I've come across several instances of people writing, essentially: 'Let me provide a very rough estimate. Don't take this number literally, because it has several flaws.' And then their number becomes cited as the canonical figure on the topic. - ThreadSky

scientificdiscovery.dev • 96 days ago

Now I've come across several instances of people writing, essentially:

'Let me provide a very rough estimate. Don't take this number literally, because it has several flaws.'

And then their number becomes cited as the canonical figure on the topic.

Comments

mattsclancy.bsky.social•96 days ago

This is actually fine so long as The Law of Offsetting Complicating Factors holds.

volodia11.bsky.social•96 days ago

The number of "dead by communism" (100 million) was made up, even the co authors of the book blamed the one one who made that number up but it’s now become the "official" number that you cannot contradict . I’m not a communist but I hate this kind of "post-truth”.

tussles-shriek.bsky.social•96 days ago

When I first encountered that I was a bit surprised. It was my own extremely vapid estimate that became a very hard number at work. Then I started noticing that happening everywhere. And now it is hard to trust most "common knowledge" number.

johanneshaushofer.com•96 days ago

I think (actually… I heard) this is how Kibera ended up as Africa’s largest informal settlement with 1M people
@kareninkenya.bsky.social

nkeleher.com•96 days ago

"Erasing the Kibera lie from history will require one enormous eraser."

https://nation.africa/kenya/news/myth-shattered-kibera-numbers-fail-to-add-up-735760?utm_source=perplexity

lgilbert.co•96 days ago

… what would the population density of Kibera have to be for that to be true

johanneshaushofer.com•96 days ago

Ah great sanity check. I found this: https://journals.openedition.org/eastafrica/521?lang=en

carolinefiennes.bsky.social•96 days ago

Oh you're here! Hello.
Welcome.

johanneshaushofer.com•94 days ago

Thank you!! :)

3dogdad.bsky.social•18 days ago

Saw this at one of my jobs. If that "rough estimate" is a number people like it becomes gospel truth for years, plugged into hundreds of P&L's and funding requests.

mattcollin.bsky.social•96 days ago

This is pretty much all work on corruption and money laundering

richardnerland.bsky.social•96 days ago

Uh oh, you just provided a nearly quantitative estimate for the corruption and money laundering literature. A new figure is born:

"A development expert recently estimated "all work" on corruption and money laundering is highly inaccurate. I code this as 95% in the following analysis..."

rohanalexander.bsky.social•96 days ago

Fisher and what is a significant p value comes to mind as somewhat related.

economeager.bsky.social•96 days ago

yep!

souptomatosoup.uk•96 days ago

This happened to me once with a policy due to go into the UK Budget, noone listened every time I said it was a number we couldn't publish until the OBR (independent review of numbers before publishing) asked to talk to the analyst to understand

gsds3.bsky.social•96 days ago

This is basically every other episode of More Or Less.

strangeloopcanon.com•96 days ago

What percentage of people do you reckon? Give us a ballpark figure.

drleonblack.bsky.social•96 days ago

I had a colleague who would regularly say "X% of statistics are made up on the spot", then sit back as people quoted him.

pseudoerasmus.bsky.social•96 days ago

you only noticed this now?? it’s routine in economic history!

scientificdiscovery.dev•96 days ago

Before I hadn't read as many of the original papers where the author actually said that 😭

I think it's also less common in biology because people are a little afraid of numbers

scientificdiscovery.dev•96 days ago

(some exceptions apply, this seems very good https://book.bionumbers.org/)

roguehci.bsky.social•96 days ago

Cohen effect size estimates...

nathanpmyoung.com•96 days ago

For me, I want to destigmatise making up numbers.

How do you feel about that?

scientificdiscovery.dev•96 days ago

Awful idea

nathanpmyoung.com•96 days ago

i reckon most experts are making up numbers in a way, and if more of them did it we could take medians and have a more accurate picture

scientificdiscovery.dev•96 days ago

I don't mind much if it's explicitly made up as part of a platform where many people are doing that to reach a median prediction.

But unfortunately in academia, if someone publishes a number, a lot of of people will feel confident in citing it as the truth, and not make better estimates.

zabong69.bsky.social•96 days ago

But 67,3% of all forecasters like that idea!

nathanpmyoung.com•96 days ago

I’d say 80

alextpet.bsky.social•96 days ago

As a rough estimate, how often would you say this happens?

victimofmaths.bsky.social•96 days ago

Recent low risk drinking guidelines in the UK, Canada and Australia have leaned heavily on the principle that the public are willing to accept a 1 in 100 lifetime risk of dying from activities we voluntarily engage in.

This comes from a 1969 paper by Starr. Here's what we thought of that:

victimofmaths.bsky.social•96 days ago

(taken from this commentary that we wrote on the Canadian guidelines when they were published. Quite proud of the level of snark we managed to get into it)

https://onlinelibrary.wiley.com/doi/10.1111/add.16381

scientificdiscovery.dev•96 days ago

Was that figure per activity (?!)

victimofmaths.bsky.social•96 days ago

Ha, no, drinking consistently at the same level over your lifetime. Although alcohol is complicated with risk because it has both immediate and long-term risks.

scientificdiscovery.dev•96 days ago

Sorry I mean, the figure seems to be a lifetime risk for one activity (i.e. drinking at a given level).

I think people would be more averse to doing lots of activities that had an individually low risk level, as in https://bsky.app/profile/adamjkucharski.bsky.social/post/3lbhcxfc4fk2j

victimofmaths.bsky.social•96 days ago

Ah. I've had many discussions with people about estimating risks *per drink* - in a similar vein to statistics about every cigarette you smoking reducing your life expectancy by ~10 minutes, but I think the combination of immediate and long-term risks makes it too hard to do sensibly...

victimofmaths.bsky.social•96 days ago

...plus the fact that the immediate risks are modifiable and context-specific (e.g. where you're drinking or with whom), while long-term risks are generally not. We even had a nice chat with David Spiegelhalter about it all once in the hope he'd sort it all out, but he just agreed it was really hard

cameron.pfiffer.org•96 days ago

As a recovering economist, this is literally every single number we have ever mentioned

scientificdiscovery.dev•96 days ago

Oh no

cameron.pfiffer.org•96 days ago

Anytime anyone says a particular number, chances are good that they are wrong.

Economists, to their credit, are actually very good at describing when and why numbers are likely to be incorrect. I appreciate that about the literature still.

lgilbert.co•96 days ago

I worry a lot one of my dubious estimations is going to become a number people cite.

drleonblack.bsky.social•96 days ago

I've seen it many times referring to carbon emissions from cement and concrete. Back of the envelope calculations quoted unthinkingly, and some absolute nonsense from people who should know better.

groverxv.bsky.social•95 days ago

You've just described the 2% Target Inflation Rate in almost all western economies

#NOOOOZEALAND
#WeWerentBeingSerious

mooseandsquirrel.bsky.social•96 days ago

Exactly like every single episode of ancient aliens…

thomasblanchet.bsky.social•96 days ago

I gave up on academia after seeing a few too many of: "I made up that number, but will only make that clear in some obscure appendix, giving me plausible deniability against accusations of fraud, while betting on enough readers to be too lazy to question it, so that the figure becomes widely cited."

onisillos.bsky.social•96 days ago

erinhengel.bsky.social•96 days ago

Or somebody asks what the correlation between X and Y in your data is, you show it in appendix Z (emphasising that it’s just a correlation and should not be interpreted as a robust causal relationship) and then read later “robust causal relationship of X on Y shown in Hengel”.

erinhengel.bsky.social•96 days ago

Even worse: you write a paper showing a relationship between A and B only and you’re cited as showing the impact of X on Y. (Like, uh, thanks for the citation but that’s not even remotely close to what the paper shows.)

scientificdiscovery.dev•96 days ago

This stuff gets worse when people don't even cite the original source, so future citers can't easily trace it back to see how rough the original estimate was.

scientificdiscovery.dev•96 days ago

(Assuming there's an original source in the first place, sometimes the number just appears out of nowhere.)

timothyrice.org•96 days ago

Oh my stars, I have been tearing my hair out on this exact problem.

There are charities that try to address "diaper need" for families, and they often cite a figure that diapers for a single child cost $1200 a year.

https://www.pdxdiaperbank.org/

timothyrice.org•96 days ago

As a parent who has diapered three kids, this number sounded insane to me. I calculated that I spent around $300 - $400 per kid, which isn't even remotely close to the figure being cited.

timothyrice.org•96 days ago

I tried in vain to track down the source of the original number, but the best I found was a single paper with the citation of "private correspondence" with someone who happened to be the founder of the National Diaper Bank Network.

https://pmc.ncbi.nlm.nih.gov/articles/PMC3727676/pdf/peds.2013-0597.pdf

timothyrice.org•96 days ago

So I emailed this person! It turns out they're calculating this amount taking the average cost of a diaper, and then assuming that "Infants require up to 12 diapers per day, toddlers about 8."

If you're not a parent, that might not seem odd to you, but that's an absolutely insane number!

franksdavid.bsky.social•18 days ago

I’ve seen this happen many times in the context of disease incidence/prevalence. An advocacy group cites a number, I try to trace back the source… and it comes back to the same group’s estimate like an ouroboros.

scientificdiscovery.dev•96 days ago

My favourite version is when people provide a number from a hypothetical example, and then future people cite that as the actual number.

Reading 'ILLUSTRATIVE EXAMPLE ONLY' in a figure caption is also amusing.

steamtraen.eu•96 days ago

https://en.wikipedia.org/wiki/AF/91

devanshsingh.bsky.social•96 days ago

And the inherent bias in quoting that figure is the essential problem! Depending on the topic at hand and where one lies on the spectrum regarding the topic (which in today's world might just be one of the extremes) dictates the figure they'll be choosing and hence the people around them will chose.

devanshsingh.bsky.social•96 days ago

What's more concerning is that most people do it without realizing the harm it brings with it. One would assume that spreading awareness about fake news would help mitigate this but when fake news is weaponized as a political arsenal then that approach becomes unusable. What else can we do though?

devanshsingh.bsky.social•96 days ago

That being said, after reading the thread I understood that the discussion is more quoting figures within the scientific and research domain and I took it more so in a general sense but both are big issues!

mridulkthomas.bsky.social•96 days ago

Also true of new measures & metrics!

https://www.smbc-comics.com/comic/metrics

mariosrichards.bsky.social•96 days ago

It's fine if the number doesn't really matter.

Obvious case, "6 million Jews died in the holocaust" - was a highly approximate and clearly heavily caveated number.

But it doesn't matter because there's no point made by "6 million" that isn't also made by "4/5/7/3/2 million".

mariosrichards.bsky.social•96 days ago

("doesn't really matter" *within the credible range of uncertainty*)

scientificdiscovery.dev•96 days ago

Isn't the figure likely an underestimate? (https://www.science.org/doi/10.1126/sciadv.aau7292)

But I was referring to 'very rough estimates', which is not the case here, and I don't think it's fine or a good idea to suggest that the figure is a very rough guess or that a better estimate doesn't matter

mariosrichards.bsky.social•96 days ago

> Isn't the figure likely an underestimate?

Probably.

My point was that
(i) the person who first came up with said it was a very rough guess
(ii) but that this doesn't matter in the context (establishing that *a lot* of Jews were killed in the Holocaust)

mariosrichards.bsky.social•96 days ago

I'm not saying "Don't fund further research".

I'm saying "any engagement on a figure should start by establishing what its purpose is".

If someone wants to assert that it's really meaningful that it was, say, 5 or 7 million not 6 they should start by saying why.

Comments

Posting Rules

Reply