Comment by jsheard
11 days ago
https://www.404media.co/google-is-the-only-search-engine-tha...
https://www.reddit.com/robots.txt
Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google.
Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit.
Kagi didn't make it explicit, but based on what they wrote I think they API call Google and other search engines to show as part of their results.
https://help.kagi.com/kagi/search-details/search-sources.htm...