Hey Squarespace users! Did you know Squarespace opted you into being crawled by AI bots without telling you? I didn't... #AI #scrapers #bots - ThreadSky

cindylouwho2.com • 472 days ago

Hey Squarespace users! Did you know Squarespace opted you into being crawled by AI bots without telling you?

I didn't...

#AI #scrapers #bots

Comments

johnneedee.bsky.social•472 days ago

Thanks. Gonna fix that on my website now.

codarobo.online•472 days ago

This isn’t really how crawlers work. Crawlers generally crawl the entire public web by default already, but you can add a text file onto your server that prevents it -when a crawler decides to care about that-. This page just provides a way to add that opt out (or not). It’s not an opt in.

codarobo.online•472 days ago

I think a lot of people misunderstand this as, “by default, we will ourselves submit your site TO crawlers on your behalf unless you opt out” which is really what “opt in” implies to a lot of people

codarobo.online•472 days ago

You could make the argument that they should automatically create the opt-out config file for everyone, by default, and i think that for AI (not search) that’s probably a good idea and would make people feel better about it.

codarobo.online•472 days ago

Put another way - if you coded your own website by hand and hosted it on a computer in your own house today, you would be “opted in” in the way you’re describing, just by putting it on the public internet, until you made that config file asking crawlers to go away. That’s not squarespace’s fault.

aimeecozza.com•472 days ago

This this this this. Because the whole web was crawled without consent the default "yes" is not having a meta tag against or no disallow in your robots.txt. This is like getting mad bc by default all websites are indexed for search engines & you need to specifically have a "noindex" in to prevent it

codarobo.online•472 days ago

This is precisely what I’m getting at, yeah. It’s still good to bring about awareness but i wouldn’t want anyone to leave squarespace under the idea that it’ll make them more protected somehow (not that anyone has suggested that here but I’ve seen that kind of response elsewhere re similar opt outs)

brekekekiwi.bsky.social•472 days ago

But they could have just as easily said that they automatically opted everyone out of this and gave users a switch to opt-in. Yes, public assumption might be the default, but if they built a switch that doesn't mean it has start at "on".

alzamon.bsky.social•472 days ago

Well I do run my own website and have a robots.txt file chock full of AI crawler bot disallows. Unless Squarespace is using an AI crawler bot of its own I have no notice about.

codarobo.online•472 days ago

That’s what I mean though, setting up a robots file is an intentional and affirmative step for most admins IME unless web server software has started shipping with those particular settings by default.

(That would be a fairly recent development if so, and probably a good idea also)

Comments

Posting Rules

Reply