Jump to content
Rhett

Robots.txt file via Pages Application

Recommended Posts

You can create and manage your robots.txt file via the pages application, this makes everything editable in the admincp, which is quick and easy. 

1. Create a new pages page.

robots-1.jpg

Select the manual HTML method

robots-2.jpg

Name it as follows. (be sure it's exactly as below- not uppercase)

robots-3.jpg

Select the content tab and add a reduced crawl rate below.

robots-4.jpg

Set permissions 

robots-5.jpg

 

Then using the SQL toolbox you can view the sessions table to see real active sessions and block any bots as needed

robots-6.jpg

 

robots-7.jpg

robots-8.jpg

 

to block a bot that honors the robots file, you can place the following, in the example of Yandex, which does, and is often an issue.

robots-9.jpg

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...