Struggling to understand. I didn't worry about the hypothetical user because it was in the section about whether we actually accept the null hypothesis -- which we agree about. But now it doesn't make much sense to me. Are you saying we should only use methods not susceptible to bias or errors? - ThreadSky

jdushoff.bsky.social • 23 days ago

Struggling to understand. I didn't worry about the hypothetical user because it was in the section about whether we actually accept the null hypothesis -- which we agree about. But now it doesn't make much sense to me. Are you saying we should only use methods not susceptible to bias or errors?

Comments

matloff.bsky.social•23 days ago

The proper way to deal with this problem is, again, to use confidence intervals. In viewing a CI, we can take into account such issues. 🧵 1/

matloff.bsky.social•23 days ago

The other relevant point is that I object to the binary nature of NHST. (It's binary even in your approach, right?) That is usurping the role of the analyst to make the binary decision. NHST throws away valuable information. 2/

matloff.bsky.social•23 days ago

Re model error, I go into great detail on this in the Appendix, using real-world examples to illustrate the problems. Do you find that material relevant/useful? 3/3

jdushoff.bsky.social•23 days ago

I don't want to throw away information, but I do sometimes want a binary approach. We agree the decision needs to be made, and we agree that CIs provide useful information. Non-inferiority constructs a NHST not by throwing away scientific judgment, but by making the judgment before seeing the data.

jdushoff.bsky.social•23 days ago

You may or may not remember that I agreed with you about this in an earlier thread. I always prefer CIs to P values, when the CIs are practically obtainable. Ben Bolker gave an example that I commented on where they don't seem to be. Other cases can also arise. You did not respond there.

matloff.bsky.social•23 days ago

I don't remember that example. Can either you or Ben remind me?

bbolker.bsky.social•23 days ago

It was this post: https://bsky.app/profile/bbolker.bsky.social/post/3lm4g7sjtks2p and @jdushoff.bsky.social's comment here: https://bsky.app/profile/jdushoff.bsky.social/post/3lmsjby4qq22l

matloff.bsky.social•23 days ago

Yes, I remember now. Understandably, I did not want to wade through a long, technical paper on ecology. :-) Please reduce it to the basics for us. Seems you are doing a large number of comparison of paired populations? Seems easy to do without NHST on the surface.

jdushoff.bsky.social•23 days ago

I commented on your draft article because you asked me to, but now I'm wondering if that was more about postponing your need to think about what I had already said than about interest in what I might say.

jdushoff.bsky.social•23 days ago

I'm honestly wondering if you bothered to "wade through" my 300-character post on this. Or bothered to process my suggestion that NHST is not "absurd" if it's a proxy for clarity, after what feels like many tries. I'll wait for a sign of non-trivial engagement before trying to reach you again.

bbolker.bsky.social•23 days ago

(1/?) This test is not a large number of paired comparisons, but a single comparison of two samples of multidimensional vectors (i.e. differences in a vector of environmental variables between 'cases' and 'controls'). (In this case only 3-dimensional AFAICT but in practice often much higher-dim)

jdushoff.bsky.social•23 days ago

I think you know I don't think acceptance proves validity, and I don't think you can find evidence that I'm arguing that way. This feels like laziness or abuse, to be honest.

Other than the possibility of bias or errors, what is your specific objection to the non-inferiority approach?

matloff.bsky.social•23 days ago

Let's just say I was surprised that you used terms like "well accepted."

jdushoff.bsky.social•23 days ago

The point is practicality. I think it's sensible, and am happy to discuss. The fact that it's simple and accepted makes it attractive for applying and sharing with others. To beat the dead horse, I agree that P values are overused, and often misused, and that CIs are better when available.

matloff.bsky.social•23 days ago

As I say at the outset of my essay, NHST is TOO convenient. And later, I say several times that proper analysis is often inconvenient -- but that's better than wrong analysis.

Comments

Posting Rules

Reply