Reddit blocking all major search engines, except Google
Reddit blocking all major search engines, except Google
%%excerpt%% Reddit has commenced its assault on search engines, blocking those that don’t have a commercial relationship with the company, like Google.
You're viewing a single thread.
this is just going to cause indexers to ignore robots.txt
"We always obey the robots.txt"
- A bunch of corporations that have no accountability and plenty of incentive to just ignore it and have all been caught training AI on off-limits data.
Rate limiting could “fix” that unfortunately.
They're likely blocking user agents too, which I think also doesn't have legal enforcement (as in DuckDuckGo can just use "Google" unless they said otherwise.
LinkedIn tried blocking scraping that way but as long as the scraping isn't burdensome it's basically legal but you can still be bound by TOS and civil claims
https://natlawreview.com/article/hiq-and-linkedin-reach-proposed-settlement-landmark-scraping-case