AI companies are violating a basic social contract of the web and and ignoring robots.txt
AI companies are violating a basic social contract of the web and and ignoring robots.txt
www.theverge.com
The rise and fall of robots.txt
198 comments
-
What social contract? When sites regularly have a
robots.txtthat says "only Google may crawl", and are effectively helping enforce a monolopy, that's not a social contract I'd ever agree to.-
I had a one-eared rabbit. He was a monolopy.
-
Sounds like a Pal name lol
-
Only if its model is a Lopunny missing an ear
-
I was thinking of a short lil bunny wearing a top hat and monocle with one ear sticking out of the center of the top hat but that works too
-
-
-
-
When sites regularly have a robots.txt that says “only Google may crawl”
Is that actually true?
If so, why would they do that?
-
198 comments