If you are the owner of a WordPress website and want to control the search engine access for your site, then creating robots.txt is the first thing you should do. This is an ASCII text file that tells search engines which pages on your site should be indexed and which should not be indexed.
Knowing how to edit robots.txt can have a significant impact on your website’s SEO, which will help you increase your website visibility while keeping your sensitive files secure and protected.
In this blog, we will explain what a robots.txt file is and how to create and customize it according to your needs.
If you’re looking for quality WordPress hosting for your website, Cloudways Hosting is an excellent option. With Cloudways, editing your robots.txt file is straightforward, thanks to easy access via the platform’s file manager or SFTP.
With all that said and done, let’s get started with our guide…
- Brief Overview of Robots.txt File
- Where Is the WordPress Robots.txt File Located
- Understanding Rules in Robots.txt File
- What Does an Ideal Robots.txt File Look Like?
- How to Create a WordPress robots.txt File (3 Methods)
- Add Rules to WordPress Robots.txt
- Specify User Agents in Robots.txt File
- How to Test Your Robots.txt File
- Avoid Common Mistakes in Robots.txt
- Summary
Brief Overview of Robots.txt File
A robots.txt file is a simple text file. In it, you can instruct web crawlers (bots) which parts they may crawl and which parts should stay hidden. This makes sure that search engines don’t index pages or files on your website that shouldn’t be public in the first place.
By default, WordPress creates a virtual robots.txt for itself: www.yourdomain.com/robots.txt. Creating a real customized one removes this default file to prevent indexing the files you don’t want to see in search results.
– Source: Elliance, Inc.
Ready to take control of your WordPress site?
With Cloudways, you can easily manage your hosting needs—whether you’re securing your website or looking for a reliable host for your site. Join the Cloudways family today and experience hassle-free WordPress hosting that puts you in charge.
Where Is the WordPress Robots.txt File Located
The robots.txt file is usually found in your website’s root directory. WordPress does generate a virtual version of this file, but it won’t actually save one on the server unless you make one specifically yourself. You can view your current robots.txt file by going to https://yourdomain.com/robots.txt in your browser.
- To view the robots.txt file for your website, simply open a web browser and navigate to the URL your-website.com/robots.txt. If the file is present, its contents will be displayed in the browser.
- If nothing appears, this means that a robots.txt file has not been created for your website.
Understanding Rules in Robots.txt File
The directives in a robots.txt file define how web crawlers should behave. There are essentially two parts to the directives:
- User-agent: This defines which crawler the rule refers to (e.g., Googlebot).
- Disallow: This indicates which pages or directories should not be crawled.
For instance:
User-agent: * Disallow: /wp-admin/
This code tells all bots not to crawl anything in the directory /wp-admin/ but to crawl other areas on the site.
It is important to understand the rules in a robots.txt file and use them correctly, as incorrect usage can result in unintended consequences, such as blocking search engines from accessing important pages on your website.
Below is a table that outlines some commonly used rules for robots.txt files and briefly explains their purpose.
| Rule | Explanation |
| User-agent: * | This rule specifies that the following rules apply to all crawlers. |
| Disallow: / | This rule disallows all crawlers from accessing any page on the site. |
| Disallow: /private/ | This rule disallows all crawlers from accessing any page under the /private/ directory. |
| Allow: /public/ | This rule allows all crawlers to access any page under the /public/ directory. |
| Sitemap: https://www.example.com/sitemap.xml | This rule specifies the location of the sitemap file for the site. |
What Does an Ideal Robots.txt File Look Like?
Ideal robots.txt file balances accessibility for search engines with protection for sensitive areas. Here is a general file code:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.yourdomain.com/sitemap_index.xml
This code structure allows a good deal of your site to be crawled by search engines while ensuring the admin area is locked out and gives a link to your sitemap for better indexing.
How to Create a WordPress robots.txt File (3 Methods)
Creating a robots.txt file can be done in a couple of ways, but for this blog, I’ll cover 3 simple methods:
- By Editing the robots.txt file using plugins.
- Create and upload a robots.txt file using FTP.
- Using Cloudways.
Method#1. Editing Robots.txt File Using Plugin
Plugins like WPCode and Virtual Robots.txt make it easy to manage your robots.txt file without touching code directly. Simply install the plugin, navigate to its settings, and add your custom rules. This method is user-friendly and ideal for beginners.
For WPCode:
- Install the plugin WPCode from the Add Plugins tab.
- Once the plugin is installed, you can click on the Code Snippets option from the dashboard and then go to the “File Editor” option to proceed.
- You can see the robots.txt file option after going to the “File Editor” option. Modify and save the changes for testing later.
For Virtual Robots.txt:
- Install the plugin Virtual Robots.txt from the Add Plugins tab.
- Once the plugin is installed, you can go to settings to update or modify the robot.txt file.
- You can check the default rule provided by the plugin, or else you can add the rules as per your requirements.
Method#2. Create and Upload Your WordPress robots.txt File Via FTP
If you like to have more hands-on control, you can make a physical robots.txt file with a text editor:
- Open a text editor (such as Notepad).
- Write down your desired rules.
- Save the file as robots.txt.
- Upload it to your site’s root directory via FTP.
This gives you full control over what’s in your robots.txt file.
Method#3. Create Robots.txt File in WordPress Using Cloudways
Here are the easy steps to create a robots.txt file for your WordPress website if you’re using Cloudways.
- Log in to your WordPress hosting dashboard. For example, if you use Cloudways, log in to your account.
- From the top menu bar, select the Servers tab.
- Navigate to Server Management and select Master Credentials to obtain access to your SSH/SFTP.
- You can use any FTP server application to access your WordPress database files. I use FileZilla and connect to my server using the Master Credentials I get from Cloudways.
- Once you have connected to your server using FileZilla, navigate to the /applications folder of your WordPress database files. Inside this folder, you will see different subfolders.
- After navigating back to the Cloudways Platform, select the Applications option from the top left bar. From there, choose the application for which you want to add the robots.txt file.
- From the left pane of the Cloudways Platform, navigate to Application Management, select Application Settings, and finally, General. Here, you will find the folder name for your selected application.
- After navigating back to FileZilla, go to the /applications/[FOLDER NAME]/public_html directory, where [FOLDER NAME] is the folder name for the selected application that you found in the Cloudways Platform. In this directory, create a new text file and name it robots.txt.
- Once you have created the file, right-click on it and select View/Edit to open it in a text editor. You can use any text editor of your choice, such as Notepad. This will allow you to edit the contents of the robots.txt file for your WordPress website.
Add Rules to WordPress Robots.txt
Now that you have designed your robots.txt file, you can add many rules depending on what you need to achieve.
1. Block Access to Your Entire Site
Adding the following code block in your Robots.txt file, you can block the access to your website to be crawled.
User-agent: * Disallow: /
2. Block a Single Bot From Accessing Your Site
By adding the following code block in your Robots.txt file, you can block access to the specific bot on your website.
User-agent: Dotbot Disallow: /
3. Block Access to a Specific Folder or File
Adding the following code block in your Robots.txt file, you can block access to a specific file or folder of your website to be crawled.
User-agent: * Disallow: /private-folder/
4. Allow All Bots to Have Full Access to Your Site
By adding the following code block to your Robots.txt file, you can allow access to all the bots or crawlers on your website.
User-agent: * Disallow:
5. Allow Access to a Specific File in a Disallowed Folder
Adding the following code block in your Robots.txt file, you can allow access to a specific File in a disallowed folder of your website to be crawled.
User-agent: * Disallow: /private-folder/ Allow: /private-folder/public-file.html
6. Prevent Bots From Crawling WordPress Search Results
By adding the following code block to your Robots.txt file, you can prevent bots from crawling WordPress Search Results on your website.
User-agent: * Disallow: /?s=
7. Create Different Rules for Different Bots in Robots.txt
Adding the following code block in your Robots.txt file, you can create different rules for different bots on your website to be crawled.
User-agent: Googlebot Disallow: /no-google/ User-agent: Bingbot Disallow: /no-bing/
8. Disallow GPTBot to Access Your Content
Adding the following code block in your Robots.txt file, you can disallow the GPTBot to access the content of your website.
User-agent: GPTBot Disallow: /
These examples show how flexible the rules can be based on what you want to achieve using your site’s SEO strategy.
Specify User Agents in Robots.txt File
You can target specific bots by specifying the user agents in your rules. These wildcards also help to refine the ways bots interact with different sections of your site.
1. Using Wildcards in robots.txt File
Using wildcards helps you apply broad or specific rules based on patterns.
2. Using Asterisk (*) Wildcard
Asterisk (*) represents any user agent:
User-agent: * Disallow: /private/
3. Using Dollar ($) Wildcard
The dollar sign ($) refers to the end of a URL:
User-agent: * Disallow: /*?*
How to Test Your Robots.txt File
Immediately after creating or editing a robots.txt file, it is very important to validate it using tools such as Google Search Console or dedicated online validators such as technicalseo.com.
I’m using a free tool to validate my robots.txt file.
- You will need to submit your URL against the required placeholder.
- Select your concern agent or bot from the dropdown list provided.
- Click on submit to initiate the verification process.
That’s it. With this procedure you will be able to verify your robots.txt file with ease.
Avoid Common Mistakes in Robots.txt
When you make a Robots.txt file for your website, be sure to avoid these common mistakes:
- Don’t block pages that should be allowed: If you block pages that should be allowed, search engines won’t be able to find them.
- Don’t allow pages that should be blocked: Block pages that search engines should not see. Otherwise, private information could be exposed.
- Test your Robots.txt file: After you create your robots.txt file, test it to ensure it works correctly. Make sure all the pages you want to block are actually blocked.
- Update your Robots.txt file: As your website changes, update your robots.txt file too. If you don’t, search engines might not see your new pages or might see pages you don’t want them to see.
- Understand what Robots.txt does: Make sure you know what the robots.txt file does and how it works with search engines. You might make mistakes when setting it up if you don’t understand it.
Summary
In conclusion, managing your robots.txt file is an essential part of effective SEO management for your WordPress site. By following the guidelines above, you will create a tailored robots.txt file that enhances visibility and protects sensitive areas on your site.
However, it’s important to be cautious when using it because a misconfiguration can lead to complete deindexation of your site (e.g., using Disallow: /).
Generally, the best practice is allowing search engines to crawl as much of your site as possible while protecting sensitive information and avoiding duplicate content. For example, you can use the Disallow directive to block specific pages or folders or the Allow directive to override a Disallow rule for a particular page.
Not all bots follow the rules outlined in the Robots.txt file, so controlling what gets indexed is not foolproof. Nonetheless, it’s a useful tool to include in your SEO strategy.
If you’re looking for a hosting provider that matches your efforts to optimize your site, consider Cloudways Hosting. With its scalable plans, excellent customer support, and 99.99% uptime guarantee, you can be assured that your website will be at its best while you focus on developing your online presence.
Q) What is a robots.txt file?
A robots.txt file is a text file that instructs web crawlers about how to index pages of your website.
Q) How do I add a robots.txt file in WordPress?
You can create the file using tools like WPCode or upload a text file to your site’s root directory via FTP.
Q) Can I block specific bots using robots.txt?
Yes, you can add rules in robots.txt to exclude certain bots from being allowed to index and crawl parts of your website.
Salwa Mujtaba
Salwa Mujtaba is a Technical Content Writer at Cloudways. With a strong background in Computer Science and prior experience as a team lead in Cloudways Operations, she brings a deep understanding of the Cloudways Platform to her writing. Salwa creates content that simplifies complex concepts, making them accessible and engaging for readers. When she's not writing, you can find her enjoying good music, reading a book, or spending quality time with her family.
