Is there a decent chunking algorithm library on NPM? I know Langchain and LlamaIndex have some, but figured there were probably some unbundled from frameworks. Chunking: chunking text documents to be fed into a RAG system. - ThreadSky

About ThreadSky

mattpocock.com • 27 days ago

Is there a decent chunking algorithm library on NPM?

I know Langchain and LlamaIndex have some, but figured there were probably some unbundled from frameworks.

Chunking: chunking text documents to be fed into a RAG system.

Comments

nickthesick.com•27 days ago

I think the problem with a generic chunker is that you can always do better by knowing your format.

Like what it means to chunk an arbitrary markdown readme looks very different than a dissertation in Latex.

So there is a wide gap here between openly available tools for this.

jaycooperbell.dev•27 days ago

Either this (which is langchain but unbundled)

https://www.npmjs.com/package/@langchain/textsplitters

Or likely some kind of NLP library (but will have more than the splitters)

jpschroeder.com•27 days ago

I have a pretty strong opinion on this, that at this point, its mostly best to write your own stuff in AI. landscape is too fresh and moves too fast to get that kind of lock in. Plus you learn the space much better.

mattr.blog•27 days ago

Went down this rabbit hole not too long ago and came up empty outside of the ones you mentioned. Super curious to see if anyone knows of others out there.

Posting Rules

Be respectful to others
No spam or self-promotion
Stay on topic
Follow Bluesky's terms of service

Comments

Posting Rules

Reply