AffiliateLinkCheckerBot

How the Affiliate Link Checker crawler behaves when it visits your site, and what to do if you would rather it did not.

User agent

Every request the crawler and the link checker make identifies itself with this user agent string:

AffiliateLinkCheckerBot/1.0 (+https://affiliatelinkchecker.com/bot)

A whole site is crawled only after its owner has verified it, and the verification is re-checked before every crawl, so the crawler does not turn up on a site that has not asked for it.

robots.txt

The crawler obeys robots.txt. It reads the rules for the user agent token AffiliateLinkCheckerBot, drops every URL your rules disallow, and follows the sitemap the file points at. A Crawl-delay directive is honored as well, up to 10 seconds, so a site can slow the crawl down without stalling it indefinitely.

A single-URL check you run yourself fetches only the page you give it and does not read robots.txt, because it is a request you make about a page you typed in. Whole-site crawling is the part that reads your rules.

Pace and page cap

• One request at a time per host. A crawl never runs parallel requests against the same site.
• Page requests to one host are at least 500 ms apart, and longer when your Crawl-delay asks for it.
• When it resolves an affiliate link, it makes at most one request to the same host in any one second.
• A crawl stops at the page cap the account's plan allows: a crawl of up to 2,000 pages on Pro and up to 20,000 pages on Business. The crawl stops there and records how many pages it found and left uncrawled.

Limits on every request

• No request waits longer than 10 seconds. A request with no answer by then is abandoned and reported as a timeout.
• It follows at most 5 redirects. A longer chain is stopped and reported as too many redirects.
• It fetches plain http and https only, on ports 80 and 443 only. A URL on any other port is refused before a connection is opened.
• It refuses private, loopback, link-local and reserved addresses, and it checks the address again on every redirect hop.

What it never does

• It sends only GET and HEAD requests. It never submits a form and never signs in or enters an authenticated area.
• It sends no cookies and no credentials, so it sees exactly what a signed-out visitor sees.
• It never bypasses bot protection. A challenge page is an answer: the crawl stops and reports that the site blocked it.

How to block it

Add this to your robots.txt:

User-agent: AffiliateLinkCheckerBot
Disallow: /

You can also block the user agent string at your server or firewall. If your site answers a crawl with a bot protection challenge, that is taken as a no: the crawl stops and reports that the site blocked us.

Support

A question about a crawl we ran on your site, a block we should honor, or a request to delete data: write to [email protected]. The privacy page says exactly what is stored and for how long.