Yes, I remember now. Understandably, I did not want to wade through a long, technical paper on ecology. :-) Please reduce it to the basics for us. Seems you are doing a large number of comparison of paired populations? Seems easy to do without NHST on the surface. - ThreadSky

matloff.bsky.social • 23 days ago

Yes, I remember now. Understandably, I did not want to wade through a long, technical paper on ecology. :-) Please reduce it to the basics for us. Seems you are doing a large number of comparison of paired populations? Seems easy to do without NHST on the surface.

Comments

jdushoff.bsky.social•23 days ago

I commented on your draft article because you asked me to, but now I'm wondering if that was more about postponing your need to think about what I had already said than about interest in what I might say.

jdushoff.bsky.social•23 days ago

I'm honestly wondering if you bothered to "wade through" my 300-character post on this. Or bothered to process my suggestion that NHST is not "absurd" if it's a proxy for clarity, after what feels like many tries. I'll wait for a sign of non-trivial engagement before trying to reach you again.

matloff.bsky.social•23 days ago

I don't think you're being fair here. AFAIK, I've addressed every point you've brought up. Right now I'm waiting for Ben.

IMO, there are 2 main points in your argument:

1. You are willing to compromise for the sake of convenience.

2. It is important to you to use binary methods, not CIs. 🧵 1/1

matloff.bsky.social•23 days ago

Item (1) is not meant to accusatory, in spite of the phrasing. Sticking to extreme positions such as this one has been problematic in various senses in my career. That is absolutely fine with me, but I can understand bending a little now and then. 2/

matloff.bsky.social•23 days ago

Re item (2), as I said, I'm waiting for Ben's reply to my request to abstract the problem out to a more general, more easily discussable form. I'm not evading his example, which you brought up (and I'm glad you did); quite the contrary, I'm looking forward to discussing this. 3/

matloff.bsky.social•23 days ago

Mark Twain said, "Whiskey is for drinking and water is for fighting" (a clever remark on water rights). NHST seems to be one of those "water" subjects. :-)

4/4

jdushoff.bsky.social•23 days ago

I'm so confused here. I've repeated that I prefer CIs to the point where I literally felt I was being silly about it! Sometimes binary decisions do need to be made (a point that I agree with, but picked up here from you). I will add some thoughts about Ben's example as well...

matloff.bsky.social•23 days ago

My point is that the analyst's binary decision should not be made for him/her by a binary statistical procedure. Are we aligned on that?

Looking forward to hearing you comments on Ben's example.

jdushoff.bsky.social•23 days ago

I'm not sure. I agree that it should not be made without a scientifically (not statistically) determined effect threshold. But what do you think of non-inferiority testing, where the threshold is picked first and then the binary stat is the final piece of the decision?

matloff.bsky.social•23 days ago

I do not believe this is a good approach, for the reasons I've given.

jdushoff.bsky.social•23 days ago

Let's just _assume_ the difference between ellipses in F6 is of biological interest. The authors did not find a way to construct a confidence interval for difference, so they are using its "statistical clarity" (P<α) as a proxy for saying that the change is clear enough to be evaluated …

matloff.bsky.social•23 days ago

Sorry for my ignorance. I don't know the term "ellipse in F6." Please explain.

But one of the main points of my essay is that mathematical statisticians, applied statisticians and others should develop a broad set of non-NHST tools that are easily accessible to researcher scientists. 🧵 1/2

matloff.bsky.social•23 days ago

As I note in my essay, one such broad tool is the bootstrap. Actually, I believe that it is also highlighted in @wkhuber.bsky.social's excellent book with @sherlockpholmes.bsky.social, Modern Statistics for Modern Biology. 2/2

jdushoff.bsky.social•22 days ago

I agree it would be good to have better tools. I've agreed with a lot of your points, and even made some of them independently. The question here is whether it's time to abandon NHST completely. Sorry that F6 (meaning, in this paper) was not clear to you; I gave that same example before.

bbolker.bsky.social•23 days ago

the "ellipse in [Figure] 6" refers to this figure. I 100% agree that we should work to make non-NHST tools more accessible. (If I and all my students/collaborators/etc. had unlimited time/brain/computational power I'd probably go 100% Bayesian ... But for me the perfect is the enemy of the good ...)

jdushoff.bsky.social•23 days ago

This makes some sense, and is standard (thus avoids adding “researcher degrees of freedom”), and reduces time spent interpreting noise. If CIs could be constructed (for the difference, not the ellipses), that would be better. But nothing here is absurd, and it nobody thinks H0 might be true...

jdushoff.bsky.social•23 days ago

I mean, _maybe_ smart people can do better than the proxy in every single case. But that still doesn't make this approach absurd. The simplicity appeals to me. In my mind it's conceptually the same thing across cases: evaluate fairly whether you're seeing clearly (using CIs for that when practical).

bbolker.bsky.social•23 days ago

(1/?) This test is not a large number of paired comparisons, but a single comparison of two samples of multidimensional vectors (i.e. differences in a vector of environmental variables between 'cases' and 'controls'). (In this case only 3-dimensional AFAICT but in practice often much higher-dim)

bbolker.bsky.social•23 days ago

(2/?) In line with your appendix, one way of replacing NHST here would be to come up with a meaningful interpretation of a distance measure between centroids of samples, and a CI on the distance (PERMANOVA typically uses robust/median-based distances to construct the null distribution)

matloff.bsky.social•23 days ago

Yes, similar to the Appendix , though not motivated by model error.

Denote the vector means by μ and λ. A natural and easily interpretable pop. measure would be Σ (μ_i - λ_i)^2. (Replace by absolute diff. etc. if desired.) The Delta Method or the bootstrap could then be used to get a CI. 🧵 1/

matloff.bsky.social•23 days ago

A more sophisticated approach would be to break the data into random chunks, and consider the species in our data to be a sample from the population of all species. One might then try a random effects model. A nice graphical description would be to graph a histogram of the estimated |μ_i - λ_i|. 2/

matloff.bsky.social•23 days ago

I like that last approach in particular, as it would be quite visually appealing, and immediately understandable. In other words, non-NHST tools need not be abstruse. 3/3

bbolker.bsky.social•23 days ago

Unfortunately this approach ignores community structure; it's the graphical version of a diff abundance analysis (for which we already know how to use random effects models/shrinkage estimators). Factor-analytic methods exist (Grantham, et al 2020 https://doi.org/10.1080/01621459.2019.1626242) but are *much* harder

matloff.bsky.social•23 days ago

We're back to the technical ecological-theoretic details, on which I can't comment. But if you are talking about communities in the graph-theoretic sense, it would seem that my proposal could easily be adapted.

Note that if you are doing shrinkage, you cannot do NHST in the first place.

bbolker.bsky.social•23 days ago

(3/?) that said, it seems reasonable to say "is what's going on here distinguishable from what we expect under an [¡ADMITTEDLY UNREALISTIC!] H0? If so, we can look at & try to interpret 2- or 3-dim graphical summaries/univariate comparisons; if not, we could well be chasing noise".

bbolker.bsky.social•23 days ago

(4/?) alternatives to using NHST for high-dim comparisons in this way include [1] element-by-element comparisons (cf differential abundance/expression analysis in microbiome/gene expression studies); [2] reducing to scalar summary statistics (e.g. comparing Shannon diversity); ...

bbolker.bsky.social•23 days ago

(5/?) other dimension reduction techniques (PCA etc etc). The last is reasonable but is more assumption-dependent ... [happy to have @jebyrnes.bsky.social, @gsimpson.bsky.social, @wkhuber.bsky.social, @sherlockpholmes.bsky.social, or other comm-eco/bioinf-adjacent people boost/weigh in here ...]

matloff.bsky.social•23 days ago

PCA makes no assumptions (beyond the technical one of finite variance).

Comments

Posting Rules

Reply