Fynd Developer Blog
Date Posted: April 2026
When people build a website, they usually want some pages to show up in search engines and some pages to stay private. That’s where something called a robots.txt file comes in. It’s a simple file that lives on a website and tells crawlers what they are allowed to look at and what they should avoid.
If you want to allow or block Fynd specifically, you can do that by using the user agent name “fyndbot” in your robots.txt file. Here is a simple example:
User-agent: fyndbot
Disallow: /private/
This tells Fynd not to crawl anything inside the /private/ folder.
If you want to block Fynd completely, you can use:
User-agent: fyndbot
Disallow: /
And if you want to allow everything:
User-agent: fyndbot
Allow: /
Fynd also follows the general rules used by other search engines. If there is no robots.txt file, it assumes everything is allowed. If the file cannot be reached, it will avoid crawling until it can safely determine what to do.
If you run a site and want to manage how Fynd interacts with it, adding a robots.txt file with the fyndbot user agent is the best place to start.