How To Configure Robots.txt File In Magento 2

by Fayyaz Khattak  February 19, 2018

One of the most essential parts of a site’s SEO optimization is configuring robots.txt in a web application. When you allow search engines to index your website, it is also necessary to instruct web crawlers to avoid indexing the mainly disallowed pages via the robots.txt file.

Magento 2 Robots txt

By default, Magento 2 configuration includes settings to generate and manage instructions for web crawlers and bots that index your store. These instructions are saved in the Magento 2 robots.txt file that reside at the root of the Magento 2 installation directory. These directives are recognized by most search engines to avoid indexing particular pages of your Magento 2 ecommerce store that are used internally by the system. You can use the default Magento 2 settings or define your custom instructions for search engine bots.

In this post, I will show you how to configure the robots.txt file in Magento 2. I am assuming that you already have Magento 2 installed on your web hosting server. If not, sign up for Cloudways account right now and launch Magento 2 store with the convenience of one-click application installation feature.

Configure Robots.txt File In Magento 2

1. First of all, login to your Magento 2 Admin Panel.

Cloudways Magento Hosting For Developers

2. Click Content and under Design, choose Configuration.

Magento Configuration

3. From the left side, Edit the Global Design Configuration.

Magento Global Design

4. Expand the Search Engine Robots section.

Magento Search Engine Robots

5. Now you need to do the following:

a) Set Default Robots option to one of the following:

INDEX, FOLLOW: Instructs search engine crawlers to index the store and check back later for changes.

NOINDEX, FOLLOW: Instructs search engine crawlers to avoid indexing the store but check back later for changes.

INDEX, NOFOLLOW: Instructs search engine crawlers to index the store once but do not check back for changes.

NOINDEX, NOFOLLOW: Instructs search engine crawlers to avoid indexing the store and do not check back for changes.

b) In Edit Custom Instruction Of Robots.txt File option, you can enter the custom instructions if needed. For example, while developing your Magento 2 ecommerce store, you can disallow access to all folders by entering custom instructions.

c) Reset To Default button will remove your custom instructions and reset the Magento 2 robots.txt file to system’s default.

6. Once you’re done with the configuration, hit Save Configuration button to apply the changes.

Custom Instructions For Magento 2 Robots.txt

Like any other web application or ecommerce platform, you can add custom instructions to the robots.txt file in Magento 2. Here are some examples of “Disallow” to consider for Magento 2 Robots.txt file.

Allow Full Access

Disallow Access to All Folders

Default Instructions

Restrict User Account & Checkout Pages

Disallow Catalog Search Pages

Disallow URL Filter Searches

Restrict CMS Directories

Disallow Duplicate Content

Conclusion

The robots exclusion standard, also known as the robots.txt file, is essential for your Magento 2 ecommerce store when communicating with web crawlers. This standard defines how to inform the search engine bots about the pages of your Magento 2 store that should be excluded or opened for crawling. That’s why the robots.txt file is significant for the correct Magento 2 store indexation and its overall search visibility.

As Magento 2 provides a mechanism for creating a robots.txt file, there is no need to create one manually. All you need to do is add some configuration in Magento 2 itself, and a robots.txt will be generated periodically.

I hope that you found this quick guide on how to configure Robots.txt file in Magento 2 useful. Now, successfully add custom instructions in the robots.txt file to enhance your rankings in the SERPs.

If you have any question, feel free to share them in the comments below.

Start Creating Web Apps on Managed Cloud Servers Now!

Easy Web App Deployment for Agencies, Developers and E-Commerce Industry

About Fayyaz Khattak

Fayyaz is a Magento Community Manager at Cloudways – A Managed Magento Hosting Platform. His objective is to learn & share about PHP & Magento Development in Community. Fayyaz is a food lover and enjoys driving. You can email him at m.fayyaz@cloudways.com

Stay Connected:

You Might Also Like...