Cursor-based Pagination

Cursor-based pagination uses tokens to navigate through datasets, unlike traditional offset-based pagination that uses page numbers. Instead of requesting "page 1, page 2, page 3", you use tokens that represent specific positions in the data.

Cursor-based pagination is available on a select few routes, and is slowly being added to others.

What are tokens?

Tokens are encoded strings that contain information about a specific position in the dataset. They are designed to be used as-is without any interpretation or parsing by your application. Think of them as "bookmarks" that the server creates to remember exactly where you are in the data.

Don't try to decode them - The internal structure and encoding of tokens is not meant to be understood by clients.
Treat them as black boxes - Just pass them back to the server exactly as you received them.
They're position markers - Each token represents a specific point in the ordered dataset.
They're server-generated - Only the server knows how to create and interpret these tokens.

How pagination works

Initial request: Make a request without pagination parameters to get the first batch of records.
Token-based navigation: Use the returned tokens to move forward (after) or backward (before) through the dataset.
Change detection: The after token can be used later to detect new or modified records.

Request parameters

limit: The maximum amount of records to retrieve. This is a limit: you might receive fewer records.
before: Retrieve records that come just before the position specified by this token.
after: Retrieve records that come just after the position specified by this token.

When you omit both before and after parameters, you get the most recent records (up to the specified limit).

For each route supporting cursor-based pagination, the API specification will mention how to set these request parameters.

Data ordering

Records returned by these listing routes are always ordered by the date they were last modified or created. A record cannot change without the last modified being updated.

The most recent modified records are always last in the dataset. This means that using before returns older records, and using after returns newer records.

Initial Data Collection

Start by making your first request without any pagination parameters:

GET /<listing-route>

You'll receive a response like this:

{
  "...": [
    {"id": 1, "name": "A", "modified": "2024-01-15T08:30:00Z"},
    {"id": 2, "name": "B", "modified": "2024-01-15T09:15:00Z"}
  ],
  "cursor": {
    "before": "MjAyNC0wMS0xNVQwODozMDowMFo=",
    "after": "MjAyNC0wMS0xNVQwOToxNTowMFo="
  }
}

What to do:

Remember the after token for later.
Store the data you retrieved.
Make a new request with the before token.
Go back to step 2 until you retrieve an empty list (no more records). You now have retrieved all data available in the dataset.

Monitor for new data

After your initial scan, monitor for changes using the after token.

Make a new request with the after token.
Store the data you retrieved.
Go back to step 1 until you retrieve an empty list (no more records). You are now up-to-date with the latest changes.
Come back any time to check for new / modified records. Make sure to use the latest after token you retrieved.

Token lifetime

Tokens remain valid for a long time. Feel free to come back 24 hours later or a week later, and use the after token to check on what information you have missed.

Fear of missing out

With this system you don't have to fear missing out on data. As long as you remember the after token you used last time, you always get the records that were updated since your last request.

Remember: records are sorted by last modified, and after returns anything updated after your last request.

Handling of Duplicates

Records may appear multiple times across different requests. This is normal and indicates the record was modified between those requests. When using after it could happen because the record was updated. When using before it can happen because you received a (partially) cached page.

To handle duplicates correctly:

If you were using the before token, the record is older than what you have already stored. Keep your existing stored record.
If you were using the after token, the record is newer than what you have already stored. Replace your stored record with the retrieved record.

Example

Python

import requests
import time

def fetch_records(url, headers, limit=100, before=None, after=None):
    params = {'limit': limit}
    if before:
        params['before'] = before
    if after:
        params['after'] = after

    response = requests.get(url, params=params, headers=headers)
    response.raise_for_status()
    return response.json()

def collect_current_data(url, headers):
    all_records = {}

    after_token = None
    before_token = None
    while True:
        print(f"Collecting data from before {before_token}...")

        data = fetch_records(url, headers, before=before_token)
        records = data.get('records', [])
        cursor = data.get('cursor', {})

        if not records:
            break

        for record in records:
            # As this is from a "before", the records in the data are always older than the ones already stored.
            if record['id'] in all_records:
                continue
            all_records[record['id']] = record

        before_token = cursor['before']
        # Remember the after-token of the first request.
        if after_token is None:
            after_token = cursor['after']

        print(f"  Found {len(records)} records")

    print(f"  End of record; found {len(all_records)} records")
    return all_records, after_token

def monitor_new_data(url, headers, after_token):
    print(f"Checking for new data from after {after_token}...")

    new_records = {}

    current_after = after_token
    while True:
        data = fetch_records(url, headers, after=current_after)
        records = data.get('records', [])
        cursor = data.get('cursor', {})

        if not records:
            break

        for record in records:
            # As this is from an "after", the records in the data are always newer than the ones already stored.
            new_records[record['id']] = record

        current_after = cursor['after']
        print(f"  Found {len(records)} new records")

    print(f"  End of record; found {len(new_records)} new records")
    return new_records

if __name__ == "__main__":
    url = "https://esi.evetech.net/listing" # ... (replace with actual route)
    headers = {
        "User-Agent": "ESI-Example/1.0",
        "X-Compatibility-Date": "2025-09-30",
        "Authorization": "Bearer <your-token>",
    }

    all_data, last_after_token = collect_current_data(url, headers)
    # ... (do something with all_data)
    while True:
        new_data = monitor_new_data(url, headers, last_after_token)
        # ... (do something with new_data)

        time.sleep(30) # ... (throttle before checking for new data again)

EVE Developer Documentation

Crowdsourced documentation for third-party developers

Other