2) It's a bit strange that the paper repeats 3 times that the effect for the primary outcome, physical function was "clinically significantly". The point estimate is lower than what they themselves considered clinically significant before they saw the results... - ThreadSky

mecfsskeptic.bsky.social • 67 days ago

2) It's a bit strange that the paper repeats 3 times that the effect for the primary outcome, physical function was "clinically significantly".

The point estimate is lower than what they themselves considered clinically significant before they saw the results...

Comments

mecfsskeptic.bsky.social•67 days ago

3) The same applies to secondary outcomes: a difference of 2.4 points on the Chalder Fatigue Scale (0-33) and a score of 1.2 on the HADS-depression scale (0-21) are often considered to be not clinically significant.

mecfsskeptic.bsky.social•67 days ago

4) To be fair, estimates of what is the minimal clinically significant difference (MCID) are often quite arbitrary. It's possible that lower estimates may still be valuable and useful for patients.

mecfsskeptic.bsky.social•67 days ago

5) The point is that in a trial with no control intervention, no blinding, lage differences in care and attention, we would expect to see better results in the intervention group due to placebo effects, reporting bias, expectancies etc.

Even if the treatment did not work.

zacharygrin.bsky.social•67 days ago

It was a pragmatic study. They weren’t looking at how it worked. They wanted to know if CBT intervention was more effective than usual care. The study results say yes. Rehabilitative treatment always consists of contextual factors & can be considered part of the intervention (therapeutic alliance).

sun-moon-and-stars.bsky.social•65 days ago

Assumptions about how it worked are built into the treatment, it is more Lightning process /NLP than CBT - no setting baseline with testing small increases, just the assumption patients can ignore their symptoms which even patients who have recovered know is an unsafe approach.

sun-moon-and-stars.bsky.social•65 days ago

Also the focus on pleasurable activities is unsafe and involves the false assumption that a pleasurable activity will somehow be less demanding of energy than a routine activity i.e. it is based on enjoyment rather than an assessment of the level of physical and cognitive activity involved.

mecfsskeptic.bsky.social•67 days ago

Let me perhaps put it another way: a treatment that does not work at all also has a big chance of appearing effective using this study design.

Take for example surgeries. A lot of surgeries appeared effective until more reliable studies were conducted:
https://mecfsskeptic.com/history-of-surgery/

zacharygrin.bsky.social•66 days ago

I understand, but the study is still comparing two choices a clinician would recommend to a patient. Based on this study, a clinician should refer to CBT-based rehabilitation. Even if the benefit is mostly due to contextual factors, it did better than usual care with very low risk.

andlon.bsky.social•67 days ago

A devil's advocate might point out that the study did not show that it was more effective, it suggested that the treatment was *perceived* slightly more effective, which could easily be accounted for by bias in self-reported outcomes

andlon.bsky.social•67 days ago

Yet at least one of the authors is pushing results like these as definitive proof that these conditions are not associated with a biological disease process.

The study itself appears thorough and solid, but I'm concerned about the apparent lack of scientific impartiality

cfsresearch.bsky.social•65 days ago

If that was the case then we would expect the effects to wear off at the 12 month follow-up, but that didn't happen...improvements were maintained.

mecfsskeptic.bsky.social•67 days ago

6) If despite these biases, the difference was still lower than the MCID, it gives an indication that the treatment is likely not effective.

Studies like these cannot demonstrate that the treatment works, but they can give an upper limit to what the treatment effect might be.

cfsresearch.bsky.social•65 days ago

I would say that it very likely is effective, due to fairly large statistically significant effect from a relatively short trial. Hopefully they will do a longer trial which will show a larger effect. It also fits in with patients experiences, which show that doing these things tends to help.

cfsresearch.bsky.social•65 days ago

See for example: https://www.reddit.com/r/mecfs/comments/1hjstdk/went_to_concert_last_night_and_felt_great_and/

mecfsskeptic.bsky.social•65 days ago

I don't think the p-value and length of the trial matter in terms of the risk of biases that I mentioned.

mecfsskeptic.bsky.social•67 days ago

7) Much like in the REGAIN trial, that upper estimate is lower than what I expected and lower than the authors pre-specified MCID.

And as in the REGAIN trial, the authors of this study still argue that the treatment was clinically significant, ignoring their own threshold.

mecfsskeptic.bsky.social•67 days ago

8) Interestingly, the study reports few adverse effects and no worse (slightly better) results for those with high PEM scores. This is similar to ME/CFS studies where trials do not corroborate the harms patients report in surveys.

Not sure what explains this discrepancy.

popkirov.bsky.social•67 days ago

The treatment effect was about +17 points in the intervention group, suggesting a clinically meaningful change (on average, not necessarily for every patient). The difference between groups should be calculated as effect size, not absolute points. Cohen's d looks to be around 0.5, which is solid.

mecfsskeptic.bsky.social•67 days ago

Change over time is not a 'treatment effect' because it includes much more than the effect of treatment such as regression to the mean, prognosis, placebo effect, response bias etc.

A comparison to a control group is necessary to try to estimate a treatment effect.

mecfsskeptic.bsky.social•67 days ago

The absolute point difference is the effect size. Cohen'd simply changes the scale on which it is reported by dividing by the sd.

it's therefore influenced by sampling, outliers and other factors that influence variance.

mecfsskeptic.bsky.social•67 days ago

Most guidelines therefore recommend interpreting the effect size on the original scale if possible. See for example:
https://bpspsychub.onlinelibrary.wiley.com/doi/abs/10.1348/000712608x377117

popkirov.bsky.social•67 days ago

Isn't there a difference between "treatment effect" as in how does the intervention affect participants, and "treatment effect" as in how much additional benefit does the intervention bring compared to placebo?

mecfsskeptic.bsky.social•67 days ago

If I have a cold, take homeopathy and my cold is cured afterwards, my improvement is not a treatment effect. It likely had nothing to do with taking homeopathy.

So without controlling for other, more plausible explanations, I do not think it's fair to speak of a 'treatment effect.'

popkirov.bsky.social•67 days ago

I think it's fair to say that observed benefits were 'clinically important', and group difference were 'statistically significant' and effect size was moderate.

mecfsskeptic.bsky.social•67 days ago

Agree it's debatable what is clinically significant.

But given that the authors had pre-specified a MCID of 10 points in the protocol, I do not think it's fair to simply ignore this and claim the effect of 9 points was clinically significant without any discussion or justification.

popkirov.bsky.social•66 days ago

They write "self-reported physical function improved statistically and clinically significantly in the intervention group after 2 to 8 outpatient encounters" and I think that's correct. In the other two instances they add "compared to controls" which I agree can be challenged.

snowypanthera.bsky.social•65 days ago

It's the same old bait and switch.

Comments

Posting Rules

Reply