API Host

Internal Links API endpoints

Related links endpoint

This endpoint delivers a list of related links from a source webpage to other target webpages within the same website. The source and target pages can be of the same type (e.g., blog-to-blog links) or different types (e.g., blog-to-product links).

Prerequisites

On the client's (or API user's) request, Graphite creates a related links endpoint. The client specifies the sets of source and target pages for which related links will be built. Graphite then crawls those pages to collect the data, computes the related links, and activates the endpoint. Links are chosen to maximize relatedness (measured by the semantic similarity of webpage text) while ensuring each webpage receives 'k' incoming links on average.

Request

Endpoint Path: /<client_id>/<module_id>/related-links
HTTP Method: GET
HTTP Authentication: None
Required Headers: None
CORS: Yes, accessible from all origins without the need for authentication or headers
Input Parameters: Query strings

Path parameters

client_id: (Required, Provided by Graphite) A Graphite client (or API user) ID.
module_id: (Required, Provided by Graphite) A single-slug identifier for the endpoint module, for example blog for same-type pages or blog-to-product for different types.

Query string parameters

`url`

(Required) A canonical or unique webpage URL. The API derives a unique ID for each webpage from its URL and uses it to look up the webpage.

For best results, query the API with the webpage's canonical URL. If your canonical URLs include query string parameters, always use those canonical URLs so the parameters are taken into account when matching the webpage. Otherwise, query string parameters are ignored and the webpage is identified by its URL without them. The order of query string parameters never affects which webpage is matched.

Valid values:

A full URL with a scheme, e.g., https://example.com/example/path.html. This is the only accepted value that is technically a complete URL.
A domain and path without a scheme, e.g., example.com/example/path.html.
A path only, which must start with a slash (/) indicating the root folder, e.g., /example/path.html.
A percent-encoded form of any of the above, e.g., https%3A%2F%2Fexample.com%2Fexample%2Fpath.html.
Query string parameters, when included, must be percent-encoded, e.g., https%3A%2F%2Fexample.com%2Fexample%2Fpath.html%3Fq%3Dv.

Example usage:

https://ilapi.graphite.io/example-client/example-module/related-links?url=https%3A%2F%2Fexample-client.com%2Fexample-section%2Feu-wants-to-see-if-lawmakers-will-block-brexit-before-striking-new-deal-uk-s-johnson

Response

HTTP Status: 200
Content-Type: application/json

Response Body Properties

message: (String, Non-null) Text that gives details about the response. Informational only.
related_links: (List of JSON objects, Non-null) A list of link objects. A link object is a JSON object containing link data, which supports the creation of HTML link elements from a source webpage to a target webpage the link object describes.

The list may include random links (link objects selected uniformly at random, without replacement) when the random link completion feature is enabled, which is the default. Random links are still considered related: at a minimum they belong to the same set as the target pages, so they are the same kind of webpage. Their primary purpose is to fulfill the API's SEO constraint: that every webpage receives the same number of incoming links ('k') on average. They fill the gaps whenever the API cannot assign the expected number of related links. Use the type field on each link object to tell related links from random ones.

This list can be empty in some cases. For example, endpoints that serve multiple languages need to know a webpage's language (captured at crawl time) before they can select same-language random links; if the requested webpage has not been crawled, its language is unknown, so no related or random links can be returned.

Example call:

curl "https://ilapi.graphite.io/example-client/example-module/related-links?url=https%3A%2F%2Fexample-client.com%2Fexample-section%2Feu-wants-to-see-if-lawmakers-will-block-brexit-before-striking-new-deal-uk-s-johnson"

Example response:

{
  "message": "Related links found",
  "related_links": [
    {
      "type": "related",
      "author": "Reuters Editorial",
      "image_url": "https://s3.reutersmedia.net/resources/r/?m=02&d=20190903&t=2&i=1425818863&w=1200&r=LYNXNPEF821I6",
      "published_time": "2019-09-03T16:27:26Z",
      "description": "Prime Minister Boris Johnson...",
      "title": "UK public must decide next steps if parliament votes against Johnson: PM's spokesman",
      "url": "https://example-client.com/example-section/uk-public-must-decide-next-steps-if-parliament-votes-against-johnson-pm-s-spokesman",
      "url_path": "/example-section/uk-public-must-decide-next-steps-if-parliament-votes-against-johnson-pm-s-spokesman"
    },
    ...
    {
      "type": "related",
      "author": "Reuters Editorial",
      "image_url": "https://s2.reutersmedia.net/resources/r/?m=02&d=20190903&t=2&i=1425826613&w=1200&r=LYNXNPEF821KH",
      "published_time": "2019-09-03T16:53:48Z",
      "description": "Earnings and revenue expectations for European...",
      "title": "European third quarter profit outlook improves slightly but still in recession: Refinitv",
      "url": "https://example-client.com/example-section/european-third-quarter-profit-outlook-improves-slightly-but-still-in-recession-refinitv",
      "url_path": "/example-section/european-third-quarter-profit-outlook-improves-slightly-but-still-in-recession-refinitv"
    }
  ]
}

Other response statuses

HTTP 204

The API endpoint exists, but data is not yet available for it. The response has no body and carries a Cache-Control: no-store header instructing clients not to cache it.

Error responses

Error responses are usually delivered in application/json format and include details about the error that occurred.

HTTP 400

A required query parameter is missing.
A query parameter failed validation.

HTTP 404

The API endpoint does not exist, or is unavailable for this client (e.g., disabled).

HTTP 500

Internal server error.

Crawling

To index pages for related links selection, Graphite's bot crawls pages from a source of URLs.

Optimal URL sources are:

XML sitemap index (preferred).
- A sitemap URL pattern could also be specified to avoid crawling all sub-sitemaps in massive websites.
XML sitemap.
HTML sitemap.
robots.txt file with sitemaps.

We run crawling at most once per day starting at 00:00 UTC with a request rate of 60-240 pages per minute.

📘
Crawl settings
For more information about our crawler, including speed, IP, and user agent, please visit our Crawl settings documentation.

Graphite's bot identity

These are the currently used Graphite bot user agents:

Base bot identifier: GraphiteBot.
Identifier with version number and comment: GraphiteBot/1.0 (+https://www.graphitehq.com).

We recommend allowing GraphiteBot to crawl your site by allowlisting its user agent. If you are worried about "User-Agent" spoofing, Graphite can provide static IP addresses for the bot's connections; they can be used to grant access authorization.

Indexing

Graphite indexes the webpages that make up an endpoint's source and target sets. These sets are most commonly identified by URL patterns (e.g., example.com/blog/{slug}), though more complex or multiple patterns are also supported.

Page Status Codes

HTTP 200

The GraphiteBot crawler indexes all pages from URL sources that return an HTTP 200 status and belong to those sets.

HTTP 301 and 302

The GraphiteBot crawler follows HTTP 301 and 302 statuses by default.
The link URL the crawler indexes is the canonical URL, or the last seen URL if there are redirections.
The redirected URL must match the same URL pattern as those sets, or it is excluded.
The URLs in between redirects can have different patterns, but the first and last URLs to crawl need to follow the identified pattern from the set.

HTTP 4xx

By default, the GraphiteBot will not include any pages found with an HTTP 4xx status.

Preventing page indexing

API users have several options to prevent page indexing:

Provide URL sources that omit the pages the user wishes to exclude from indexing.
- This can be done by providing a dedicated sitemap for the API that only includes pages that should be in the index.
Use the HTML robots meta tag to mark a webpage as noindex. Any webpage you block from search-engine indexing this way is also skipped by GraphiteBot. For example, use <meta name="robots" content="noindex">.
- Other tags like <meta name="googlebot" content="noindex"> are also supported.
- The X-Robots-Tag HTTP header is another supported method, as detailed in https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag.
To block GraphiteBot specifically, use either the meta tag <meta name="graphitebot" content="noindex"> or the X-Robots-Tag: graphitebot: noindex header.

Notes on data extraction

GraphiteBot uses a hierarchy of data sources to extract information from HTML documents:

Reads the webpage's first <h1> element for its title.
Reads schema.org JSON structured data, particularly the CreativeWork type and its subtypes (e.g., Article, NewsArticle, WebPage), to gather the title, description, authors, categories, images, published time, and text content.
Falls back to the Open Graph protocol to gather the title, description, authors, images, and published time.
Falls back to standard HTML metadata to gather the title tag, meta author, meta description, and the language declared on the <html lang> attribute.

Precedence is applied per field: each field takes its value from the highest-precedence source that provides it, falling back to the next source when a higher one has nothing.

Text extraction

Accurate text content is important to the API's algorithms. The API first uses text from schema.org structured data (for example, an Article's articleBody, or the step text of a HowTo). If that is unavailable, it uses the content of elements within the <body> marked with the itemprop="text" attribute. As a last resort, it falls back to the full text of the <body> element (excluding hidden content).

Recommendations

To build links, you usually only need the target URLs and your own data. But if you rely on the data the API extracts, the following help both your SEO and the API's extraction:

Use a unique canonical link: Give each page one unique canonical URL. This avoids duplicate-content issues, improves SEO, and is what the API returns as the link url.
Use a single <h1>: Mark up the page title with one <h1> element. It helps search engines and is the API's primary source for the title.
Provide structured data: Add schema.org or Open Graph markup (or both). It gives search engines and the API richer, more reliable metadata.
Add an ID to the main content: In the less common case where Graphite needs a custom extractor for your set, a stable id on the main content element lets it target the content accurately.
Add IDs to relevant elements: Likewise, stable ids on elements such as authors, breadcrumbs, categories, published time, and images make custom extraction more reliable.

Performance & usage

The API runs on AWS, with 99% uptime and an average response time of 150 ms.

Recommended integration

The ideal pattern is to call the API server-side, on demand, when a user requests a page, and to serve from a cache:

A user requests a page on your site.
Your server checks its cache for that page's related links.
On a cache miss, your server calls the API and caches the response, keyed by the request URL (including its query string parameters).
Your server renders the related links into the page.

Because the links are rendered server-side, your API usage stays server-to-server: the API endpoints are never exposed to the internet or to third-party bots.

❗️
Caching is strongly recommended
Always serve from cache when you can, and only call the API on a miss. Links are refreshed at most once a day, so a 24-hour TTL (Time To Live) is appropriate.

Rate limits

API endpoints are not rate-limited. The pattern above (one call per user request, served from cache) needs no rate limiting. If you instead retrieve links in batches, we recommend staying under 20 requests/second per endpoint; at that rate you can update links for 10,000 pages in under 10 minutes. Graphite can advise on batch-job frequency.

Graphite Growth™

Standard ILAPI Documentation