Technology @lemmy.zip BrikoX @lemmy.zip 1y ago The text file that runs the internet www.theverge.com The rise and fall of robots.txt 3 crossposts InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)TE Technology @midwest.social alex [they, il] @jlai.lu 1y ago The rise and fall of robots.txt www.theverge.com /24067997/robots-txt-ai-text-file-web-crawlers-spiders Technology @beehaw.org alyaza [they/she] @beehaw.org 1y ago The rise and fall of robots.txt: As unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart. www.theverge.com /24067997/robots-txt-ai-text-file-web-crawlers-spiders Technology @lemmy.world Andy Reid @lemmy.world 1y ago AI companies are violating a basic social contract of the web and and ignoring robots.txt www.theverge.com /24067997/robots-txt-ai-text-file-web-crawlers-spiders
