After playing more feels like the normal RAG generated answers while imperfect is not the major source of crazy results. It seems when you use the main search it will sometimes decide to use one of the other "guided tasks" which dont work well. e.g the "seminal paper" it finds is often horrible(1) - ThreadSky

aarontay.bsky.social • 3 days ago

After playing more feels like the normal RAG generated answers while imperfect is not the major source of crazy results. It seems when you use the main search it will sometimes decide to use one of the other "guided tasks" which dont work well. e.g the "seminal paper" it finds is often horrible(1)

Reposted from Aaron Tay

I then ran exact same query. The results are now a totally different TYPE of answer. Leaving aside it totally went off the rails in the answer, the type of response was listing SEMINAL papers and "View additional documents relevant to this response" goes to a fixed list of items (3)

Comments

metasynthesis.net•3 days ago

At least they know "seminal papers" are a thing, unlike most of the Deep Research models.

aarontay.bsky.social•3 days ago

but i agree, the problem with these RAG models & even deep research is like they are just seeing the trees (more like leafs) not the forest. You get a sense they are just looking at individual chunks and trying to mix them. Thats why they often cite papers in odd ways, not wrong per se but weird.

metasynthesis.net•3 days ago

It's totally cargo cult research, and the sad thing is, that is probably going to look convincing to a lot of people.

aarontay.bsky.social•3 days ago

i dont think it is an unsolvable problem. Up to now, people were trying mostly on Q&A type tasks, but as they move towards longer report generation, the problems can be reduced by a combination of careful handcrafted workflows + agent reasoning

metasynthesis.net•3 days ago

It's definitely solvable. However, it's much easier to present something that looks like a solution to people who don't know any better.

aarontay.bsky.social•3 days ago

Hence my view people typically undergraduates who don't know what a good literature review look like shouldn't even use these tools.. particularly untutored...

aarontay.bsky.social•3 days ago

i have no idea how they even identify it, its kinda crazy some times. probably some bibliometric magic.

aarontay.bsky.social•3 days ago

feels to me the seminal papers it finds is just totally weird or at best sort of relevant but way too far away. e.g. asking about large language models and getting LDA and Pagerank has suggested seminal papers!

aarontay.bsky.social•3 days ago

For example here i even specify the context of Binary independence model to be information retrieval when really you shouldn't need to, yet it still gives me as seminal papers "code switching" which is sociolinguistics??

aarontay.bsky.social•3 days ago

the main issue is even if i ignore this guided task and use the main search, it will still randomly decide to forgo the usual RAG answer (which tends to be okay) and instead try to find "seminal papers"...and there's no way to even get a clue why the results were surfaced.

aarontay.bsky.social•3 days ago

I also find Web of Science RA confusing as heck, besides the RAG search it seems there are like a ton of other tools/workflows available. There's "seminal papers", topic maps, Co-citation map (not same thing), topic over markness models, top authors all with diff visualizations. It's really a mess

aarontay.bsky.social•3 days ago

Add the fact that WOS RA seems to be able to recognize all sort of fields in natural language, not just pub year, but affiliation, country, journal issue vol, doi etc, citation counts etc. Look at the Types of questions WOSRA can answer https://webofscience.zendesk.com/hc/en-us/articles/31437630410129-Web-of-Science-Research-Assistant#h_01JGKZK0Q4JYJ3T9DHD5KNPCMW:~:text=Types%20of%20questions%20WOSRA%20can%20answer%C2%A0

aarontay.bsky.social•3 days ago

while it sounds nice to be able to ask for papers on topic X by authors from affiliation Y with citations > Z, it also means you can accidently trigger a workflow you didn't expect. I saw that in a demo where one query gets you a normal RAG answer another slightly diff gets the top author workflow

Comments

Posting Rules

Reply