I know I've said it before, but bots and crawlers -- especially those belonging to AI companies -- will be the death of many a good website. Currently 90% of the traffic to the Huddersfield Exposed server is bots, with only 15% of that being from legit sources (e.g. search engine crawlers). - ThreadSky

huddersfield.exposed • 2 days ago

I know I've said it before, but bots and crawlers -- especially those belonging to AI companies -- will be the death of many a good website. Currently 90% of the traffic to the Huddersfield Exposed server is bots, with only 15% of that being from legit sources (e.g. search engine crawlers).

Reposted from British History Online

British History Online is still experiencing downtime due to bots and crawlers.

We are working to mitigate this and restore access as soon as possible.

Comments

huddersfield.exposed•2 days ago

I've had to delay to launch of the upgraded site until I've got effective blocks and filters in place. At present, the bulk of the non-legit bot traffic is trying to fake its identity and pass itself off as normal web users.

huddersfield.exposed•2 days ago

What's more worrying to me is that most of the requests are coming from domestic ISP IP addresses, e.g. 12.6% of the requests from UK Virgin Media IP addresses (AS5089) are currently bots. Got TikTok on your phone? It's likely busy harvesting web content for a Chinese AI company.

longmind.bsky.social•2 days ago

Why is that worse than AI made by white people?

huddersfield.exposed•2 days ago

Didn't mean to imply that it was, but the evidence I've collated points towards Chinese companies being more underhand and unethical in their harvesting methods. Other AI companies are just as bad but their traffic is easier to identify. They're all a scourge tho.

huddersfield.exposed•2 days ago

Huddersfield Exposed was one of the many sites DDoS'd by Anthropic and Amazon AWS last year when their bots ran out of control.
https://www.businessinsider.com/openai-anthropic-ai-bots-havoc-raise-cloud-costs-websites-2024-9
https://www.theregister.com/2024/07/30/taming_ai_content_crawlers/

tballam.bsky.social•2 days ago

Do you have a robots.txt file?

huddersfield.exposed•2 days ago

Yep, but some AI companies (Anthropic, ByteDance, Alibaba, etc) ignore it so it's no longer a viable method of controlling access.

tballam.bsky.social•2 days ago

Have you put web Ts and Cs that they are liable to pay via contract for crawling your website if they ignore the T's and C's then invoice them for it (parking charge notice style) issue small claims against one and they will soon buck their ideas up.

huddersfield.exposed•2 days ago

I've been tempted to do that :-D At the moment, the bogus bot requests are using random web browser user agent strings, so not too difficult to sniff them out (mostly very old versions of Chrome). Giving them a captcha (e.g. Cloudflare Turnstile) instead of the requested content works well.

tballam.bsky.social•2 days ago

The other thing to do is poison the website (Text off screen) so it ruins their ai?

huddersfield.exposed•2 days ago

My lawyer has asked me to neither deny nor confirm that I've been using this as my revenge on several of the major AI companies ;-P https://github.com/yunginnanet/HellPot

Comments

Posting Rules

Reply