Robots.txt
Generator

Robots.txt is a text file at the root of your domain that gives instructions to web crawlers. You can allow or disallow access to specific URLs or directories for all bots or specific crawlers.

Disallowing a URL in robots.txt prevents crawlers from fetching it, but if the URL is linked from other pages Google may still index it with limited information. Use the noindex meta tag to reliably prevent indexing.

Add a Disallow: /directory/ rule for the bot. The generator provides a form to add multiple rules without writing the file manually.

Yes. Add Disallow: /wp-admin/ (or your CMS admin path) for all user-agents to prevent search engines from attempting to access or index admin pages.

Place it at the root of your domain, accessible at https://yourdomain.com/robots.txt. It must be at this exact path to be recognised by crawlers.