bro this is literally what robots.txt was invented to do.
Post image

Comments