Comment by jsheard
11 days ago
It's also not much use to anyone who doesn't use Google ever since Reddit started blocking all crawlers besides Googlebot. Old cached results might still show up in Bing/DDG/Kagi but they can't index any of the newer stuff.
What makes you say this? I just tried a few Reddit searches about events from the last few days and they work as expected (tested in Kagi)
https://www.404media.co/google-is-the-only-search-engine-tha...
https://www.reddit.com/robots.txt
Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google.
Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit.
Kagi didn't make it explicit, but based on what they wrote I think they API call Google and other search engines to show as part of their results.
https://help.kagi.com/kagi/search-details/search-sources.htm...