Online Toolbox
共401tools were used 97253times
switching mode
tool collection
put it on the desktop
inclusion application
tool search
robots.txt file generator

robots.txt file generator

  • global settings

  • restricted directory

  • Sitemap

  • domestic search engines

  • foreign search engines

start generating copy clear

robots.txt file generator Introduction

what is robots.txt file:

1. robots.txt (unified lowercase) is a text file stored in the root directory of the website. it usually tells the web search engine spider what content on this website can be included and what cannot be included.

2. the file name of robots.txt should be unified in lowercase. robots.txt should be placed in the root directory of the website

3. if you want to define the behavior of search engine spiders accessing subdirectories separately, you can merge your custom settings into robots.txt in the root directory.

4. the robots.txt protocol is not a norm, but is only a convention, so it cannot guarantee the privacy of the website.

5. note that robots.txt uses string comparison to determine whether to obtain the url, so there is a different url at the end of the directory that does not have a slash "/" means

robots.txt file content

1. whether the search engine spider is accessible or crawling

2. search engine spider accessibility for directories or files

3. website map sitemap path definition

4. time interval limit for search engine spiders crawling

about robots.txt file generator

1. set the data to be configured through the web interface, click generate to generate the robots.txt file content

2. create a blank text file named "robots.txt", and then copy and paste the above content into "robots.txt"

3. put "robots.txt" in the root directory of your website and access robots.txt to make sure it allows the search spider to access it.

Internet Business Promoter

introduce

Tool box is an online service platform that integrates multiple functions. It provides utility tools such as decompressing files online, detecting META tags on web pages, online alarm clock, and sitting prompts to help users easily solve various problems in daily life and development.

Attention

Attention

Contact Us

You can contact us in the following ways

Url：TXT SitemapSitemapdisclaimer

Copyright https://tool-box.top/ Rights Reserved.