Wanted to talk about Neocities' approach to AI. First, some new positions going forward: - We will never make you use AI if you don't want to - We will help you stop AI crawlers if you want to - We won't use false positive heavy AI blockers - We won't be "anti-AI" either and ban sites for using it - ThreadSky

neocities.org • 104 days ago

Wanted to talk about Neocities' approach to AI. First, some new positions going forward:

- We will never make you use AI if you don't want to
- We will help you stop AI crawlers if you want to
- We won't use false positive heavy AI blockers
- We won't be "anti-AI" either and ban sites for using it

Comments

freshasketch.bsky.social•103 days ago

come on guys i just switched to you because of this ai bullshit

satnav.neocities.org•102 days ago

at least they're not directly adding ai to the site... yet. i remember there being talks about adding an ai chat assistant to the code editor for premium subscribers, but i don't know if that ever manifested

yatpay.bsky.social•103 days ago

I don't like AI either but this seems like a pretty measured take. They're giving tools to opt out of AI scraping and making it clear that spam content is not allowed. I think the nuance is situations they point out, like people using AI tools to learn HTML. Pretty much "you do you.. within reason."

freshasketch.bsky.social•103 days ago

why can't more places have the guts to denounce it outright?
there's so many coding resources why help kill the planet to get help that is probably wrong anyway?

neocities.org•103 days ago

RE CO2: IIRC a trained high level model only generates a small amount of CO2 emissions. GPT4 training has been estimated at a one-time cost of 200 flights from NYC to SF.. this is two days of normal airplane travel. If you drive a gas or even EV car it produces far more CO2 than if you use an LLM.

neocities.org•103 days ago

RE having the guts to denounce AI, generated AI content sucks and you shouldn't use it because it will make your site have bad content and nobody will want to go to it. It's a pretty gutless statement to make IMO, people don't really like looking at weird photos of dudes with 15 fingers.

freshasketch.bsky.social•103 days ago

I'm glad you feel that way but I wish you'd say that more upfront ya know?

I'm not asking you to ban AI sites but the initial post didn't feel like the same sentiment as this one

yatpay.bsky.social•103 days ago

Yeah, it'd be nice. I'm completely making this up but I imagine they might want to avoid something unenforceable. Or finding themselves having to judge when someone says "this person's using AI" and they say "no I'm not".

That said, they could use this policy AND say "btw, we don't like AI".

onekthesquirrel.bsky.social•103 days ago

thank u 4 all ur hard work 🩷

neocities.org•104 days ago

Starting this month, all new Neocities sites now come with a robots.txt. robots.txt is the file that controls how bots access your site. The newly provided robots.txt contains a commented out list of known AI crawlers that you can uncomment if you would like to opt out of AI crawling.

leatherand.ink•103 days ago

These should be off by default.

foxtwashere.neocities.org•103 days ago

is there somewhere where we can download robots.txt from?

neocities.org•103 days ago

Right here https://raw.githubusercontent.com/neocities/neocities/refs/heads/master/views/templates/robots.txt

foxtwashere.neocities.org•103 days ago

merci!!!

707kat.bsky.social•62 days ago

It doesn't stop AI bot crawlers. It only stops the ones who abide by it and most don't.

https://lwn.net/Articles/1008897/

neocities.org•104 days ago

Why are we choosing AI crawler management with robots.txt? Because this approach gives sites total control over the process, and also respects web standards. This is the agreed upon way by all parties to maintain a web site's control over the process. It's not perfect, but everything else is worse.

neocities.org•104 days ago

For example, some platforms approach this with IP/Agent blocking but this is a disaster. For example CF's AI powered AI blocker blocks legitimate users from ~viewing~ web sites (https://news.ycombinator.com/item?id=42577076). Sans DDoS mitigation, Neocities never blocks IP addresses from browsing web sites.

neocities.org•104 days ago

This approach also works in scenarios where IP blocking won't. For example. Applebot is used for all Apple crawling, but Applebot-Extended in the robots.txt file opts out of AI (https://support.apple.com/en-us/119829). If you just block Applebot you're blocking search engine indexing and link previews too.

neocities.org•104 days ago

If you have an existing site and want to block AI, you can create your own robots.txt and add the blockers. We're going to be updating our list using this one for now (note the Applebot entry - will be sending a PR later today): https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt

neocities.org•104 days ago

Lastly: Neocities is also not going to stop anyone from using AI or be tinfoil hat crazy about it either. If you're using AI to help learn HTML or to fix bugs in your code, that's totally fine. If you want AI to index your site and provide it for people using LLMs as search engines, that's fine too.

rasslinwithdemons.bsky.social•103 days ago

This reads as an opt-out model for AI crawling, rather than an opt-in model, so sites are opted in to crawling, by default, a decision explicitly not left to the site, which requires opting out. This is the opposite of your stated goal of giving sites control rather than imposing conditions on them.

cassiterite.bsky.social•103 days ago

that's unfortunately a decision that's not up to the website host, but instead lies on the fault of aggressive AI bots. it's opt-out for *every* website, not just for neocities users. this is the fault of the people running AI bots, not the fault of neocities (or any other web host).

neocities.org•103 days ago

Having to choose between blocking all bots (including googles search bot) and not doing that, I think the safe default is to let users decide what they want to not allow to visit their sites. I've never seen a site template that blocks all bots by default without informing the webmaster.

rasslinwithdemons.bsky.social•103 days ago

I guess. Just not a fan of being opted in to things rather than asked. Not my wheelhouse, though, so if that's the way it makes sense to do it, that's the way to do it.

goeshard.bsky.social•72 days ago

how you doing these days, kyle?
we should get together again some time. :)

esmemarion.bsky.social•103 days ago

My apologies for my last couple of posts - completely misunderstood this. I still think it should be blocked by default however

Comments

Posting Rules

Reply