This website uses cookies

Our website, platform and/or any sub domains use cookies to understand how you use our services, and to improve both your experience and our marketing relevance.

WordPress 5.0 is out. Test all the great new features before upgrading your Live website. Read How!

How to Set up Robots.txt for WordPress Websites

Updated on  22nd November

5 Min Read
WordPress Robots.txt
Reading Time: 5 minutes

Just creating a website is not enough. Getting listed in search engines is the essential goal of all website owners so that a website becomes visible in SERP for certain keywords. This listing of a website and visibility of freshest content is mainly due to search engine robots that crawl and index websites. Webmasters could control the way in which these robots parse websites by inserting instructions in a special file called robots.txt.

Today, you will learn how to easily setup a WordPress robots.txt file for the best SEO. Several pages of a WordPress website need not be indexed by the search engines.

Understanding Robots.txt File

Every website has a “robots.txt” file that tells search engines what pages to index. This file is usually found in the root directory of a website. If not, you can easily create one.


Source: Elliance, Inc.

Set up WordPress Robots.txt File

Usually, for WordPress websites, you do not need to add robots.txt file. Search engines index the entire WordPress sites by default. However, for better SEO, you can add robots.txt file to your root directory to specifically disallow search engines to access specific areas of your WordPress site.

How to Create Robots.txt File

Login to your hosting Dashboard. In our example, I will be using Cloudways – a high-performance Managed Cloud WordPress Hosting platform.

Go to “Servers” tab from top left, and get into your server. You will find your FTP details there.


Open an FTP server application to access your WordPress files. I will be using “FileZilla.” Launch it and connect to a server by using above “MASTER CREDENTIALS”.

Connect to Server

Once connected, go to the Applications folder. You will see some folders here. Do not get confused. Go back to the Cloudways platform and from the top left pane, go to Applications.

Select the application that you want to add the robots.txt file for:

Cloud Server

From the left pane, go to Application Settings. You will find the folder name of your application.

Folder Name Cloudways

Come back to FileZilla and then navigate to “/applications/[FOLDER NAME]/public_html.” Create a new text file here. Name it “robots.txt

Cloudways FTP

Open robots.txt WordPress file with any text editor (Notepad is a handy option).

Advanced robots.txt for WordPress

Search engines like Google and Bing support the use of wildcards in the robots.txt file. These wildcards can be used to allow/disallow specific file types throughout the WordPress website.

An asterisk (*) can be used to handle sequence.

Here, * means that all images starting with “image” and with “jpg” extension will not be indexed by search engines.

Example: image1.jpg, image2.jpg, imagexyz.jpg will not be indexed by search engines.

The power of * is not limited to images only. You can even disallow all files with a particular extension.

The above statements will ask all search engines to disallow all files with extensions “pdf” & “png” found in the downloads folder.

You can even disallow WordPress core directories by using *.

The above line asks search engines not to crawl directories starting with “wp-”.

Example: wp-includes, wp-content, etc will not be indexed by search engines.

Another wildcard symbol used in WordPress robots.txt file is the dollar symbol ($).

The above statement will ask search engines not to index referral.php and also referral.php?id=123 and so on.

But what if you want to block referral.php only?  You only have to include $ symbol just after the referral.php.

The symbol $ ensures that only referral.php is blocked but not referral.php?id=123.

You can use $ for directories too.

This will instruct search engines to disallow wp-content folder plus all directories that are located inside wp-content. If you want to disallow only wp-content rather than all sub-folders, you should use the $ symbol. For example:

The $ symbol ensures that only wp-content is disallowed. All the directories in this folder are still accessible.

Below is the robots.txt file for Cloudways blog.

The first line indicates the User-agent. This refers to the search engine that is allowed to access and index the website. A complete list of all search engine bots is available here.

Where * means all search engines. You can specify each search engine separately.

This will not allow search engines to crawl the “admin” directory. It is often not necessary for search engines to index these directories.

If your WordPress site is a blogging site, it is the best practice to restrict search engine bots to not crawl your search queries.

If your site has a sitemap. Adding its URL helps search engine bots in finding the sitemap file. This results in faster indexing of pages.

What to include in my WordPress robots.txt file?

You decide which parts of WordPress site you wish to be included in SERP. Everyone has their own views on setting WordPress robots.txt file. Some recommend not to add robots.txt file in WordPress. While in my opinion one should add and disallow /wp-admin/ folder. Robots.txt file is public. You can find robots.txt file of any website by visiting

We’re done with robots.txt file in WordPress. If you have any query about setting robots.txt file, feel free to ask in the comment section below.

Frequently Asked Questions

Q1. What is robots txt WordPress?
The robots.txt is a text file placed at the root of your website. This file is intended to prohibit search engine robots from indexing certain areas of your website. The robots.txt file is one of the first files scanned by spiders (robots).

Q2. Why robots txt file is used?
The robots.txt file gives instructions to the search engine robots that analyze your website, it’s an exclusion protocol for robots . Thanks to this file, you can prohibit the exploration and indexing of your site to some robots (also called “crawlers” or “spiders”).

Share your opinion in the comment section. COMMENT NOW

Mustaasam Saleem

Mustaasam is the WordPress Community Manager at Cloudways - A Managed WordPress Hosting Platform, where he actively works and loves sharing his knowledge with the WordPress Community. When he is not working, you can find him playing squash with his friends, or defending in Football, and listening to music. You can email him at

Be the first to check out WordPress 5.0

Use our Staging feature before upgrading your website to WordPress 5.0.

Get Our Newsletter
Be the first to get the latest updates and tutorials.