Some science today: I had an opportunity to evaluate OpenAI's "Deep Research" software, so I asked it to summarize the toxicity of thalidomide, with respect to species differences and compound isomers. The results were superficially impressive, but concealed deeper problems: - ThreadSky

dereklowe.bsky.social • 17 days ago

Some science today: I had an opportunity to evaluate OpenAI's "Deep Research" software, so I asked it to summarize the toxicity of thalidomide, with respect to species differences and compound isomers. The results were superficially impressive, but concealed deeper problems:

Comments

andreadimitracopoulos.com•17 days ago

Thanks for sharing your interesting analysis! I’m curious - did you try, or intend, to explicitly ask Deep Research to prioritise recent literature over older sources? Keep up the great work!

dereklowe.bsky.social•17 days ago

No, the prompts were as described. I wanted to give it a neutral but specific task to see what might happen.

ravenonthill.bsky.social•17 days ago

It's my experience with the public facing version of the software; it doesn't understand the words it puts together and so it makes mistakes.

These systems use verbal facility to fake knowledge. This is a sort of fraud, and the developers of the systems ought to be held to account.

dereklowe.bsky.social•17 days ago

That goes for the non-public-facing versions, too, of course. None of these models "understand the words".

ravenonthill.bsky.social•17 days ago

What's discouraging is it takes real knowledge to know what's wrong with what these text generators do. People who are raised on this output, who are educated by it, are not going to learn that it is wrong except by harsh experience.

gabegomes.bsky.social•17 days ago

Derek: huge fan of yours here and i am always in favor of calling out issues with all new technologies arising. for this one, i think you missed the mark a bit. here's my own Deep research chat on the topic that you showed in your post: https://chatgpt.com/share/67a4f9f8-bbc8-800e-a9dc-6f8953b84f0f [1/7]

gabegomes.bsky.social•17 days ago

you can see that this time it focused on more recent literature. you will notice that my prompt is vastly different from yours. i took your prompt and added the following two sentences: [2/7]

gabegomes.bsky.social•17 days ago

"Your report MUST include references that back up your claims. Make sure to ground your research on the most recent outcomes going backwards as you will find that a lot of newer papers and results elucidating these effects have been published." [3/7]

dereklowe.bsky.social•17 days ago

I am sure that this would fix the problem. But for a review of the scientific literature, I would have thought that this would have been "baked in" as a default behavior. . .

gabegomes.bsky.social•16 days ago

not really and it kinda makes sense that it isn’t: these are developed to be general purpose tools. if we want it to behave more like what a scientist would do in producing a review of the literature, you can push it on that direction. i think this is a plus, the control is in the hands of the user.

gabegomes.bsky.social•17 days ago

then i used o1-pro to augment the prompt specifically for Deep research. you see it here: https://chatgpt.com/share/67a4fab0-e1e4-800e-a286-8f15a5326cc7 [4/7]

gabegomes.bsky.social•17 days ago

my point is the following: the tools are going to make mistakes, but these are tools, not oracles. knowing how to use them and their limitations is as important as know their capabilities. we are all learning and adapting, but things are moving very fast. [5/7]

gabegomes.bsky.social•17 days ago

PS: before anyone jumps on me here, i did NOT do an in-depth review of the report that Deep research generated. i see some issues with general language and share my concerns with @dereklowe.bsky.social on how it talks about "stability." [6/7]

Comments

Posting Rules

Reply