Wow, this sounds rather concerning: a series of errors leads to consistently wrong conclusions across analyses. And the (mistaken) treatment effects are often “produced” by methods that actually differ from what’s described in the paper! - ThreadSky

dingdingpeng.the100.ci • 22 days ago

Wow, this sounds rather concerning: a series of errors leads to consistently wrong conclusions across analyses. And the (mistaken) treatment effects are often “produced” by methods that actually differ from what’s described in the paper!

Reposted from Johanna Rickne

🚨 REPLICATION REPORT UPDATE: One year ago, a tweet by John Holbein alerted me, @ollefolke.bsky.social, and @jopieboy.bsky.social to a paper with a shocking result about Sweden’s law criminalizing the purchase of sex.🧵

Comments

dingdingpeng.the100.ci•22 days ago

Folks, always share your code. It doesn’t have to be perfect to be helpful. And if you feel that it’s still too messy or not sufficiently clean to be shared, you shouldn’t submit yet. After all, there could be mistakes in your mess.

mikejohansenmd.medsky.social•22 days ago

If you don't share your code, you don't get funded. Seems like an easy rule.

dhrsprof.bsky.social•22 days ago

It also matters what codes you are sharing. I have seen plenty of papers where the codes shared are the final regression codes. I also want to see codes used to get to the final analytical sample, not just glm(outcome ~ x1 + x2...)

charlymarie.bsky.social•22 days ago

If you're not confident enough to share your code, it could mean you're not competent enough to do the analysis you claim to be doing.

Could be a problem, but who am I to judge?

zetaof1.bsky.social•22 days ago

Julia I am pretty sure it's "Submit when the results are right" not "Submit when the code is right" what are you talking about

dingdingpeng.the100.ci•22 days ago

🥲

mosenkis.bsky.social•22 days ago

I think @j-pal.bsky.social used to offer a helpful "code check" service, where authors could have someone independently run their code and confirm that it produced the tables in the paper but very few authors used it (maybe @andydebarros.bsky.social can correct me if I'm wrong?)

andydebarros.bsky.social•22 days ago

Yes---actually I used to be a J-PAL replication fellow as a student. Great experience! Jack can provide guidance for anyone with a J-PAL project https://www.povertyactionlab.org/person/cavanagh

felipefv.bsky.social•22 days ago

Do you know of any such service that is currently available?

mosenkis.bsky.social•22 days ago

I don't, but see Andy's reply to me with the J-PAL open science staffer's info? Maybe he'd know

econmaett.bsky.social•22 days ago

@dingdingpeng.the100.ci no that’s how they discover errors in your analysis!

eckhoffandresen.bsky.social•22 days ago

plus the paper isn't even retracted. :/

Comments

Posting Rules

Reply