Jump to content

Robots.txt file via Pages Application


Rhett

Recommended Posts

You can create and manage your robots.txt file via the pages application, this makes everything editable in the admincp, which is quick and easy. 

1. Create a new pages page.

robots-1.jpg

Select the manual HTML method

robots-2.jpg

Name it as follows. (be sure it's exactly as below- not uppercase)

robots-3.jpg

Select the content tab and add a reduced crawl rate below.

robots-4.jpg

Set permissions 

robots-5.jpg

 

Then using the SQL toolbox you can view the sessions table to see real active sessions and block any bots as needed

robots-6.jpg

 

robots-7.jpg

robots-8.jpg

 

to block a bot that honors the robots file, you can place the following, in the example of Yandex, which does, and is often an issue.

robots-9.jpg

 

Rhett Buck

Exact Servers

 

 

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...