Task 3: Web Crawler Politeness There are certain standards a web crawler is expe

Question

Task 3: Web Crawler Politeness There are certain standards a web crawler is expected to follow, such as: Respecting the restrictions outlined by a site's robots.txt. Not making multiple, concurrent page requests to a single web server Waiting a certain time between multiple requests to the same web server (for example ten times the time it took to complete the last request). The only one responsible for controlling that a web crawler follows such standards is the crawlers operator. There is no central authority enforcing them. As such, they are called politeness standards. 1. 2. Explain why such politeness standards are necessary on the web Explain why it makes sense for the operators of web crawlers to follow such standards

Explanation / Answer

Web crawlers are very useful programs helpful for search engines at the web sites.
They gather valuable information from papge to page and prvide indexing information
for serach engines.They are also helpful in validating the HTML links on the page.

As for the crawlers and other information gathering information softwares, there should be some standards to follow as it id better to have formal things in place as it should be beneficial to every party and mutulally understandable.

Web crawler should follows the rules defined in the website’s robots.txt file. This file is usually available at the root of a website (www.example.com/robots.txt) and it describes what a crawler should or shouldn’t crawl according to the Robots Exclusion Standard. Some websites even use the crawlers’ user agent to specify separate rules for different web crawlers. Web crawlers should not hit websites too
hard as too much of requests may make the web server unrespomsive.Web crawlers should
provide contact information of the owner of the web crawler to the websites from which they are gathering information. It is important to follow these standards to
work smoothly with other sites.

Related Questions

Navigate

• Browse (All)
• Browse T
• Subjects

---

---

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.