I made this free and open source dev tool: gitingest.com Replace "hub" with "ingest" in any github URL to get a text digest that you can feed into any LLM - ThreadSky

yasbaltrine.bsky.social • 78 days ago

I made this free and open source dev tool:
https://gitingest.com

Replace "hub" with "ingest" in any github URL to get a text digest that you can feed into any LLM

Comments

lordymiyagi.bsky.social•55 days ago

This is awesome, I can lend a hand if you won't me.

yasbaltrine.bsky.social•55 days ago

You are very welcome!
Check out the repo if there's any issue you like or open your own if you have an idea ! https://github.com/cyclotruc/gitingest

niccolozanotti.com•60 days ago

Awesome tool, thank you!!

paulsample.bsky.social•78 days ago

Nice tool and the UI is great. It would be helpful if you could filter for specific file types. For example, ingesting https://github.com/google-gemini/generative-ai-python pulls out tons of stuff you wouldn't actually need for an LM.

yasbaltrine.bsky.social•78 days ago

Thank you! I plan to add some settings and filters very soon

A temporary fix would be to get a specific subdirectory from the url like so:
https://gitingest.com/google-gemini/generative-ai-python/docs/api/google

paulmwatson.com•49 days ago

Does it respect licenses?

yasbaltrine.bsky.social•48 days ago

I'm not sure I get what you mean here:

There's no logic in my code to to anything special about licences, but I fail to see which kind of licence would say "this code shouldn't be read by humans or machines"

paulmwatson.com•48 days ago

AGPL or a growing set of specific no-AI licenses https://github.com/non-ai-licenses/non-ai-licenses

paulmwatson.com•48 days ago

You've create a utility that could be used improperly. Might be legal but it can be unethical.

yasbaltrine.bsky.social•48 days ago

Oh thank you!
I didn't know about those types of licence

While on a personal note I fail to see their purpose, I agree that gitingest should respect those.

I'm definitely adding this to the roadmap. Do you know if there's any standard that could help me identify those licences?

yasbaltrine.bsky.social•48 days ago

It could be funny if the best way to check if a Licence has anti-ai clauses was to use AI to read them

akadapa.bsky.social•62 days ago

well done. Thank you!

masonjames.bsky.social•53 days ago

I absolutely love this project - not a coder so using it all the time to provide context :)

One thing I've hit a couple times is that if the repo has a data outputting image/png the gitingest includes the whole hash. This quickly skyrockets the number of tokens, but I'm not sure how to exclude it.

yasbaltrine.bsky.social•53 days ago

Interesting thanks for bringing that up; could you show me a screenshot of that? Or even better an example link?

masonjames.bsky.social•52 days ago

Yup yup. In this repo's output, I removed 6 occurrences, which took it from 2.7M tokens to about 400k: https://github.com/roboflow/supervision

yasbaltrine.bsky.social•52 days ago

I don't see what you mean: When I try it shows 280k tokens with default ingestion settings

masonjames.bsky.social•51 days ago

Ah! Then here it is - I thought this slider meant it would simply include random files in the repo that were unusually large. When I up the slider, that idea is confirmed by seeing more estimated tokens... I thought there was just more data available. What does the slider do? 🤷‍♂️

masonjames.bsky.social•51 days ago

Not "random", but now the output is trying to include images as SVGs or something because I raised the slider?

If so maybe the interface could warn or offer an option to exclude assets? I was hoping to capture oddly long contribution docs, examples, etc, but not media.

maxgooned.bsky.social•74 days ago

this is fantastic, i've been doing this manually!

teodos.bsky.social•37 days ago

this is super cool ping @peignoir.bsky.social

chrisartmann.bsky.social•21 days ago

Gitingest is really cool!

jacky.wtf•47 days ago

I don't enjoy nor consent having the code I worked on being added to tools like this. Do you have a means of opting out from this service?

Better yet, an opt-in model?

yasbaltrine.bsky.social•46 days ago

The only way to make sure your code won't be read by an AI is to not open-source it

I heard there is ANTI-AI licences that exists but taking those into account would require.. using an AI to read it

Also, your code is not "added" to a tool like gitingest, someone has to use gitingest on your code

jacky.wtf•46 days ago

There's another option: designing opt-in/opt-out logic. It's something many tools do and provide a modicum of consent.

SPDX licensing has an API and generally most copy left licensing could be anti-AI training

yasbaltrine.bsky.social•46 days ago

I'm interested in this but I don't fully understand how you imagine this opt-in opt-out logic, can you send me a DM to elaborate?

jacky.wtf•46 days ago

Opt-out can be done by allowing someone to sign in with GitHub and add them to a table that prevents the tool being used with them.

Opt-in can be inverted such that folks can only use this with repositories that they have collaborator access to (for example).

yasbaltrine.bsky.social•46 days ago

I think there's an inherent issue with those two approaches:
In the end my tool is free and open source

Anyone can just fork it and remove the restrictions in code directly, then ship the modified product that will eventually become more popular as people hit the limits of my own "censored"version

a.lebaux.co•76 days ago

how can I make this work with a private app?

yasbaltrine.bsky.social•74 days ago

I'm working on making a CLI out of it very soon

a.lebaux.co•72 days ago

check this one https://github.com/khromov/ai-digest

a.lebaux.co•73 days ago

The cherry on top would be a way to include and exclude files and folders.

Keeping the context focused is the key

a.lebaux.co•76 days ago

Insanly cool idea, style and execution, I give this app 5/5 stars.

The space between AI and coding has a lot of space to grow. A tool I was thinking about was a VSCode extention where I can right click a file and copy the entire file but with the file name on top, for promting

yasbaltrine.bsky.social•76 days ago

Thank you!
I also believe there's a lot of things to do, starting from the most simple ones

a.lebaux.co•76 days ago

This way I can paste into a chat more easily, currently I'm doing it by hand.

I would love for someone to steal that idea.

Hope you keep building mini tools in the like. Feel free to ping me for a re-tweet or however is called here