Robots.txt Generator

What Is a robots.txt File?

A robots.txt file is a plain-text file placed at the root of a website. It can provide instructions to compatible web crawlers about which paths they are allowed or disallowed from requesting.

How to Create a robots.txt File

1. Select a user-agent.
2. Add allow rules when needed.
3. Add disallow rules for paths you want compatible crawlers to avoid requesting.
4. Optionally enter your sitemap URL.
5. Click Generate Robots.txt.
6. Review the generated output carefully.

What Does User-agent Mean?

The User-agent directive identifies the crawler to which the following rules apply. An asterisk (*) is commonly used when a rule is intended for all crawlers that respect robots.txt instructions.

What Does Disallow Mean?

A Disallow rule indicates a URL path that compatible crawlers should not request under the applicable user-agent rules.

What Does Allow Mean?

An Allow rule can be used to indicate a path that compatible crawlers may request. How overlapping rules are interpreted can vary by crawler, so site owners should test important configurations.

What Is a Sitemap URL?

A Sitemap directive can point compatible crawlers to the location of an XML sitemap. Enter the full absolute URL when using this option.

Is robots.txt a Security Tool?

No. robots.txt is not designed to protect private information. A compliant crawler may choose not to follow the rules, and the contents of the robots.txt file itself are publicly accessible.

Where Should robots.txt Be Placed?

For a standard website, the robots.txt file is normally placed at the root of the host, for example: https://example.com/robots.txt.