← Back to context

Comment by alberto_balsam

11 days ago

What makes you say this? I just tried a few Reddit searches about events from the last few days and they work as expected (tested in Kagi)

2 comments

alberto_balsam

Reply
jsheard  11 days ago

https://www.404media.co/google-is-the-only-search-engine-tha...

https://www.reddit.com/robots.txt

Reddit serves a different robots.txt to Googlebot, you can see a snippet of it in Googles summary. If Kagi is getting recent Reddit results then they must be either ignoring robots.txt, using Google as a backend, or also paying Reddit for access like Google.

Bing and DuckDuckGo are certainly still locked out, I just tried searching for "reddit hurricane milton" on both and none of the results are actually from Reddit.