Jump to content
Rhett

Robots.txt file via Pages Application

Recommended Posts

You can create and manage your robots.txt file via the pages application, this makes everything editable in the admincp, which is quick and easy. 

1. Create a new pages page.

robots-1.jpg

Select the manual HTML method

robots-2.jpg

Name it as follows. (be sure it's exactly as below- not uppercase)

robots-3.jpg

Select the content tab and add a reduced crawl rate below.

robots-4.jpg

Set permissions 

robots-5.jpg

 

Then using the SQL toolbox you can view the sessions table to see real active sessions and block any bots as needed

robots-6.jpg

 

robots-7.jpg

robots-8.jpg

 

to block a bot that honors the robots file, you can place the following, in the example of Yandex, which does, and is often an issue.

robots-9.jpg

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×