Crawling and indexing

To index your pages for related links, GraphiteBot crawls the pages in the sitemap you provide.

When you first submit your configuration, your crawl is queued. Your Graphite point of contact provides the endpoints, which return related links only after the initial crawl and indexing are complete.

After the initial crawl, we re-crawl your sitemap on a schedule: weekly by default, and at most daily. Each successful crawl refreshes the related links for every page, and the updates appear in the API response. Crawls run at 60-240 pages per minute, starting at 00:00 UTC.

User agent

GraphiteBot/1.0 (+https://www.graphitehq.com)

Crawler IPs

• GraphiteBot crawler (3.131.206.24): the IP the bot crawls from. Allowlist it if you restrict access by IP.
• Site exploration and diagnostics (3.21.78.26): used while we explore a site during setup, in case IP-based authorization is needed.

Graphite Growth™