Robots.txt for blogs: how to avoid blocking Google and AI search engines

The robots.txt file is a plain text document in the server root that instructs search robots about which pages and files of a blog can or cannot be crawled. Incorrectly configuring this file can unintentionally block Google and AI search engines, directly compromising the site's visibility and SEO.

For managers, marketing leaders, and professionals seeking to transform their blog into a predictable acquisition channel, understanding robots.txt rules is a technical infrastructure requirement. A syntax error or a forgotten directive after site approval can make months of content production invisible.

How robots.txt works and the difference between crawling and indexing

Robots.txt guides the crawling of your blog, but does not directly prevent a URL from being indexed by Google if it is discovered through other links.

There is a fundamental distinction in technical SEO that causes a lot of confusion:

• Crawling: It is the process in which robots (like Googlebot) visit the server, read, and download files, scripts, and the page's HTML code. Robots.txt acts exclusively at this stage.
• Indexing: It is the inclusion and organization of that page in the search engine's database to be displayed in search results.

If you block a page in robots.txt (using the Disallow directive), Googlebot will not crawl its content. However, if external links point to that URL, Google can still index it and display it in results without showing the text snippet. To completely prevent indexing, the correct approach is to allow crawling and apply the HTML tag <meta name="robots" content="noindex"> on the page.

If your company needs to reactivate an abandoned blog, the first technical check should be to confirm that robots.txt is not preventing search engine access to the articles directory.

Essential robots.txt syntax: Allow, Disallow, and Sitemap

Allow and Disallow directives control robot access to specific directories, while the Sitemap line points the sitemap directly to search engines.

The file is always located at the domain root (for example, seudominio.com.br/robots.txt) and works with a simple syntax based on User-agent declarations and permission commands:

• User-agent: Defines which search robot will receive the rules below. The wildcard character * applies to all bots not individually specified.
• Disallow: Indicates which directories, URLs, or files should not be crawled.
• Allow: Opens access exceptions to subdirectories or files inside a folder blocked by Disallow.
• Sitemap: Declares the absolute URL of the XML file listing all priority pages of the blog.

A standard and safe robots.txt example for WordPress blogs or modern CMSs is:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://seudominio.com.br/sitemap.xml

The most common error in migrations or new projects is keeping the line Disallow: / active in the production environment. This rule prevents crawling of absolutely the entire site, eliminating the blog's ability to generate organic traffic.

Blocking AI search engines: OAI-SearchBot, GPTBot, and the impact on generative SEO

Differentiating AI training bots from generative search bots in robots.txt is crucial to avoid losing citations and direct traffic coming from ChatGPT Search and other AIs.

With the advancement of answer engines and generative experiences, the crawler ecosystem has split into two main categories of AI bots:

1. Training Bots: Collect public data to train language models (LLMs). Examples include GPTBot (OpenAI) and CCBot (Common Crawl).
2. Search and Citation Bots (Search Bots): Process pages in real time to answer user questions with direct reference links. Examples include OAI-SearchBot (used in ChatGPT Search) and PerplexityBot.

Blocking GPTBot prevents your content from training future AI models, which many companies choose to do for intellectual property protection. However, blocking OAI-SearchBot prevents your blog from being cited as a source and receiving traffic from users on ChatGPT Search.

When designing a blog content automation strategy, make sure your server's robots.txt allows access for bots focused on generative search and SEO (GEO - Generative Engine Optimization).

Comparison of directives and search bots for blogs

The table below details the function of the main bots and the effect of blocking them in the blog's robots.txt.

Case study: Technical blog reactivation and robots.txt fix

Properly enabling crawling combined with a continuous content operation allows recovering organic traffic and site indexing in a few weeks.

A B2B corporate services company spent eight months without recording new organic impressions after redesigning its website. The internal team did not understand the lack of traffic, as new articles continued to be written and published manually.

During a technical SEO audit, it was identified that the robots.txt file contained the command Disallow: /blog/, inherited from the staging development phase. Googlebot was blocked from reading any published article.

After removing the restrictive line and integrating the site with the BlogPost Automático platform to scale the production of qualified topics, the blog recorded the following qualitative transformations:

• Googlebot resumed crawling the directory and re-indexed existing pages in less than 14 days.
• The continuous publication volume established new qualified entry points in the sales funnel.
• The blog started appearing among cited sources in generative searches about corporate consulting.

Understanding how a blog on autopilot works assumes that the basic technical infrastructure — including robots.txt and sitemap.xml — is properly clean and accessible.

Frequently asked questions

Frequently asked questions

What happens if I put Disallow: / in my robots.txt?

The Disallow: / instruction prevents all search robots from crawling any page on your site. This blocks Google and AI search engines from reading your content, gradually eliminating the blog's organic visibility.

Does robots.txt replace the meta noindex tag?

No, robots.txt controls crawling only, while the noindex meta tag specifically instructs non-indexing of the page. If you want to ensure a private page does not appear on Google, you should use the noindex tag and allow the robot to crawl it to read that instruction.

Should I block GPTBot and other AI bots on my company's blog?

Blocking GPTBot only prevents your content from being used to train AI models, without affecting traditional SEO. However, if your goal is to gain visibility and authority in ChatGPT Search, you should not block generative search bots like OAI-SearchBot.

Where is the robots.txt file located on a blog?

The robots.txt file is located in the root folder of your domain's web server. It should be publicly accessible by typing the exact URL seudominio.com.br/robots.txt in the browser.