NarviSearch/robots.txt

forked from Korbs/4get
User-agent: *
Disallow:
Host: search.sudovanilla.org
Sitemap: https://search.sudovanilla.org/sitemap
still missing things on google scraper 2023-07-22 14:41:14 -04:00			`User-agent: *`
			`Disallow:`
Remove all trace of the name "4Get" and "lolcat.ca" and remove swear words 2024-06-07 20:47:08 -04:00			`Host: search.sudovanilla.org`
			`Sitemap: https://search.sudovanilla.org/sitemap`