There are now 5 reports like this—#AI performing better than physicians + AI—and we don’t have the explanation for why yet (hybrid was supposed to be best) Gift link nytimes.com/2024/11/17/hea… - ThreadSky

erictopol.bsky.social • 104 days ago

There are now 5 reports like this—#AI performing better than physicians + AI—and we don’t have the explanation for why yet (hybrid was supposed to be best)
Gift link https://nytimes.com/2024/11/17/hea…

Comments

johnnyehm.bsky.social•104 days ago

Interesting stuff. I think this could be a huge game-changer for Emergency Room medicine, especially for triage and cutting down on 12+ hour waits.

Plus, there is no end of stories of people being sent home from ER without a diagnosis only to come back later with life-threatening issues.

markd64.bsky.social•104 days ago

There's got to be something in the psychology of AI use, at least within the subset of users who believe they are experts on the topic. Something's getting in the way of benefiting by the tool.

mcenkidu.bsky.social•104 days ago

There have been hundreds of AI better than radiologist, physicians, GPs etc etc for years.

epatientdave.bsky.social•104 days ago

Good heavens, @erictopol.bsky.social , you are our agentic AI in human form! You spotlight SO many important papers. Do you have any idea how much of a force multiplier you are, like a good AI?? :-)

oranj.org•104 days ago

AI is only about pattern recognition. Seems to really work well in diagnosis.

navin.halfloop.com•104 days ago

Seems we are starting to hit this phase of AI.

Here is a talk I did about AI in medicine 7 years ago. And the concept of over ruling an AI decision support system:

https://youtu.be/LCTW3IoX5jQ?t=810&feature=shared

It would be interest to have a large scale study to understand the level of false positives here too.

navin.halfloop.com•104 days ago

Ufff typos 😂

flempire.bsky.social•104 days ago

AI has no prejudices, no preconceived notions, no biases.

It’s straight and to the point. It’s prerogative is direct

natanael.bsky.social•104 days ago

It *does* have bias and prejudice, it's just different from humans. They learn from data created by humans, human written medical histories and medical study data, etc. Some of that will carry through when it is trained on that data.

natanael.bsky.social•104 days ago

The main difference is that biases that don't fit well into the model doesn't get reinforced and falls out, so a lot of irrational and personal biases doesn't get learned. However systemic biases and common flawed theories tend to stick

calvinjburke.bsky.social•104 days ago

^^^ this right here

ukemike.bsky.social•104 days ago

This has been proven wrong over and over. AI has prejudices built in by the assumptions baked into the training data.

billballentine.bsky.social•104 days ago

True, is it even possible to have bias free training data? After all, a human had to collect or make that data at some point

jbc3.bsky.social•104 days ago

Exactly. The ‘L’ in LLM is “large”. In order to make statistically significant predictions (i.e., what word is most likely to come next, for generative AI), the training data has to be absolutely massive. So basically it has to come from scraping the internet, where all of our biases are on display.

jrob77.bsky.social•104 days ago

Possibly that would apply to text generation, but not specific for image classification or segmentation tasks.

billballentine.bsky.social•104 days ago

That’s sort of what I was thinking. What if an AI was trained purely on ultrasound or MRI images? Would the bias from the techs that took the images be significant enough to harm the AI? Or what about whose images make it in the dataset? Are there ways to mitigate that type of bias?

natanael.bsky.social•104 days ago

Even those are trained by samples plus descriptive labels of the samples. If some samples are incorrectly labeled due to prejudice then the model might learn to replicate that prejudice, because during learning it's *literally being told to believe the prejudice* in those labels

epatientdave.bsky.social•104 days ago

Please provide links to some of those "over & over" fails. Important info.

I'm not sure why you'd assert that Adam's result is not true. I'm expecting the studies you'll point to are much larger and/or have better methods.

flempire.bsky.social•104 days ago

I don’t disagree. I was thinking more in the sense when a.i. is used for good and trained for specific things it is much more efficient

beerbabe.bsky.social•104 days ago

Hmmm did we just discover a new method for identifying and assessing provider bias in healthcare?

simplesarahbear.bsky.social•104 days ago

Right?

perceptions420.bsky.social•104 days ago

I remember reading a similar result from a paper where the implication was that in hybrid systems the human doctors would often exclude valid predictions from the bots due to some amount of ego or misjudgment. We do know that judgement of the individual users of an AI-oriented system is key.

patsybaloney.bsky.social•104 days ago

I appreciate this. At the same time I am deeply resentful of (and thus, completely resistent to) the intrusion by corporate health care into my provider's office--without my consent--to record my conversations w my doctor/nurse/PA/... apparently for the purpose of training a 3rd party's AI.

jbc3.bsky.social•104 days ago

You can tell them that you do not consent to that.

patsybaloney.bsky.social•104 days ago

Thank you.

leoguari.bsky.social•104 days ago

Similar to much older research showing nomograms and statistical risk assessment methods outperforming physicians.

4livingtheory.bsky.social•104 days ago

I think that this is a small study, but there are issues to think on. Drs & HCWs are under pressure, staffing & overworked. I recall yrs ago in another country a Dr sat w/me using a medical database to help in figuring out my issue & assess treatment options best for me. I think it may be helpful/1

4livingtheory.bsky.social•104 days ago

To figure out how to help Drs & HCWs have the assistance of clinical info, maybe use of a limited AI review as reference- and clinical researchers that help keep Drs up to date w/their practice areas in short blurbs w/links. I’ve long thought each clinic & hospital could use clinical researchers/2

4livingtheory.bsky.social•104 days ago

That can not only review recent applicable research & turn it into a bi weekly briefing AND help look at complex cases & feed data into a program that can bring more insight that the researchers can assess & look into and provide as reference. I think we need to see it as potential to help assess/3

4livingtheory.bsky.social•104 days ago

But not rely on it as the decider or truth or anything. Humans use of tools can help, but a lot of info comes from human assessment as well. /4

kevlare.bsky.social•104 days ago

Here is the full link, it got somehow displayed incorrectly.
https://www.nytimes.com/2024/11/17/health/chatgpt-ai-doctors-diagnosis.html

coloradowildflower.bsky.social•104 days ago

artsyhonker.bsky.social•104 days ago

I can't read the article from here but given that AI is basically just a very large spreadsheet that can do statistics better than humans can, and given that many medical providers discount common symptoms (like fatigue), this doesn't seem hugely surprising.

knowledgeabledog.bsky.social•104 days ago

Well, AI chatbots don’t have to work two weeks of nights before getting shoved into clinic to see a complex overbooked patient in a 20 minute slot while the EMR is down ok?

infrecursion.bsky.social•104 days ago

We need a very large-scale study of this, especially with latest models. Small studies won't be very convincing and this is very important.

zombaekillz.bsky.social•104 days ago

I entered in my labs and symptoms and used it to help guide my drs as to what could be wrong when I was being treated bc they were obviously missing things. And chat gpt was right. More often than not.

shelaghcampbell.bsky.social•104 days ago

Almost like there is a specific agenda being pushed? Whatever could that be?

yetanotheruseless.com•104 days ago

> hybrid was supposed to be best

Have you *met* doctors? Try telling them the answer to something, and inevitably they will tell you why it's not the answer.

I can very easily imagine ChatGPT suggesting the correct answer and institutionalized oppositional-defiant disorder kicking in *hard*.

gludge.bsky.social•104 days ago

I think the difficulty with a lot of these "studies" is that in your gut you know AI is not better than humans. It might be in certain strained conditions but humans always win in diagnosing real people. GPs can smell, listen, and firm opinions on how a person sits etc which AI still can't manage.

erinsopran.bsky.social•104 days ago

Strained conditions is right! How many of these doctors had a day off recently, how many of their clinics are severely understaffed?

jesrcerulean.bsky.social•104 days ago

Physicians can also blow off patient input if it conflicts with their prejudices- race, class, gender, age, BMI all decrease their valuation of what their patient actually says.

gludge.bsky.social•104 days ago

Totally agree. They are not infallible, they are not machines, they are human.

rockyuphill.bsky.social•104 days ago

The lack of ego in the AI likely helps a lot. If nothing else the growth of poorly diagnosed long covid conditions indicates that many doctors are not that interested in putting in the work for a more challenging diagnosis.

erictopol.bsky.social•104 days ago

corrected link https://www.nytimes.com/2024/11/17/health/chatgpt-ai-doctors-diagnosis.html

art1963.bsky.social•104 days ago

Well. Being waiting my urologist for 2 hours in a full office. Happy to replace him and his staff with chat gpt. System is a joke. Reviews in Google of this gigantic practice are a disaster. No consequences foe practices acquired by corporations to milk prs with procedures and no real human care

brunomaggi.bsky.social•104 days ago

It seems we need to train the user properly, to be useful. As expected.

rpast.bsky.social•104 days ago

yeah seems like a poorly designed study

shantanu-singh.cc•104 days ago

IIRC Pranav Rajpurkar and colleagues probed this phenomenon (in radiology specifically) https://www.nature.com/articles/s41591-024-02850-w

esp this bit "They didn’t listen to A.I. when A.I. told them things they didn’t agree with”

bluefarmgirl.bsky.social•104 days ago

It would help if doctors would really listen to patients and believe what we’re saying…

brianwilcox.bsky.social•104 days ago

Supports the idea that medical training imparts too much medical arrogance.

M. Deities

sheldonrampton.bsky.social•104 days ago

I think the explanation is pretty straightforward if you know how AI works. It's just a highly sophisticated collection of algorithms for making predictions based on data. Not surprisingly, computers are also better than people at calculating square roots and cosines.

natanael.bsky.social•104 days ago

Selection bias? Unless studies are just bad.

LLM is great at association but terrible at complex logic. A human expert should be able to reliably beat it if assisted with good info like relevant statistics for treatment options.

nutty.land•104 days ago

“Should” is bearing a heavy load here. Many doctors’ clinical knowledge is distressingly out of date, and an AI doesn’t get tired; it won’t blow off your symptoms because they’re inconveniently weird or just because it’s the end of the day and they want to go home.

4livingtheory.bsky.social•104 days ago

It’s why I think there should be clinical researchers on staff at clinics/hospitals helping assess straight data but also looking into practice area research & putting together briefings for clinicians.

lspogo.bsky.social•104 days ago

Maybe AI doesn't refuse to listen to symptoms, order tests, or diagnose illness in any woman who is overweight. That would be an easy way to outperform all penised physicians.

shikra-redkite.bsky.social•104 days ago

All endo cis male physicians

eighthdayarts.bsky.social•104 days ago

Framing it as a competition is the wrong way to go. One of my greatest hopes is that AI solves medical problems that humans cannot.

Diagnosing conditions and treating the patient are two different things

blkntvterraffvi.bsky.social•104 days ago

Genuinely looking forward to AI taking over diagnosis, it's already helped me more than 10 useless and callous doctors

ukemike.bsky.social•104 days ago

The key words are "small study". Small studies often have results that are eventually disproven.

philmarsden.bsky.social•104 days ago

Were the reports also written by AI?

Comments

Posting Rules

Reply