Dev Insights Blog

Hi all,

Today we're announcing changes to how ESI deals with pagination. This new system will be used for the first time with the introduction of Corporation Projects later this summer. We'll also be opening a discussion thread on Discord for Q&A and feedback from third-party developers.

Moving forward, new routes that list information will no longer use offset-based pagination, but rather cursor-based pagination. This will come in the form of before and after parameters which take an opaque token to crawl the dataset.

But why?

Current routes that list information use so-called “offset-based pagination”. This means that when you request data that doesn't fit in a single request, you receive a header telling you how many “pages” there are, and you can request the other pages one by one. This is done via the page parameter (taking in a number) and X-Pages response header (showing you the total number of pages available).

This is troublesome for two reasons:

1. Often your question is: “what has changed?” but you are served everything. You have to figure out what has changed. And depending on the sorting of the list, this can get complicated.

2. Adding to that, these pages are cached. But while you are crawling the pages, the cache can expire, and you can be presented with a new dataset. It takes some care to detect this happening and to deal with that properly.

Especially the second point is often overlooked, which can mean you miss data or see the same data twice. To compensate, people came up with clever tricks, like polling the first page until the cache expires, and only then quickly request all the other pages. Although we love the creativity here, we knew we could do better.

Introducing: Cursor-based pagination

Where offset-based pagination uses page numbers to indicate where to continue the listing, cursor-based pagination uses an opaque token. On every request you make, we send you back before and after tokens. These tokens look like random strings to you, but to us they represent what data you have and haven’t seen. On your next request, you can either send the before or after token as a parameter, and we will continue giving you the next batch of information from before or after (respectively) the last batch. If you get an empty result back, it means you either reached the beginning or the end of the dataset.

Side note: although this is often called cursor-based pagination, a better description is token-based pagination; the token you receive isn’t actually pointing to something. It is just a marker to tell us where you left off.

Additionally, we will sort these datasets by “last modified” time. If anything changes in a record, its “last modified” time is updated. This unlocks another powerful feature: if you receive an empty list with an after (so you reached the end of the dataset), you can query with this after token again a bit later. If any record in the dataset has changed, it will show up on the next request. You will get a new after token, and the cycle continues. This means that if you scanned the whole dataset once, you can just keep on checking the empty after token until it isn’t empty anymore to stay up-to-date with the latest information. No FOMO!

Important: you might see the same record multiple times while fetching different pages. An example:

• You made a request that returned records A, B, and C.

• You used the after token as a parameter, and you now receive D, B, E.

This isn’t a typo: you saw B twice. This happens because between your two requests, B changed. And so the system is informing you that it has changed. And if you requested more detailed information about B in between those two requests, you might want to refresh that information.

Also, the system does not guarantee that if you request 10 entries, you will receive 10 entries. You might receive fewer. This doesn’t mean this is the end of the dataset. You’ve only reached the end of the dataset if you receive an empty list. This is a side-effect of caching and doesn’t impact the consistency of the results.

Best practices

A few things to keep in mind while working with cursor-based pagination:

• The tokens you receive are opaque. In other words: to you, they are just a string. Process them as such, and don’t try to validate them, interpret them, etc.

  • Although the tokens have an internal representation, that representation is not relevant to you. We might change, without notice, the format, encoding, and/or meaning of the token at any time. They are our tokens. And you can use them, as-is. We cannot stress this enough.

• Scan the whole dataset once by using the before token. After that, keep monitoring the after token for change. This way, you always have full visibility of the whole dataset. No need to rescan the whole set again.

• Feel free to keep the cursor tokens around for a while. You can make a request with them hours, days, or even weeks later. The listing will continue where you left off, and you will receive all updates since. This allows you to find an update frequency that works for you.

• You might see the same records multiple times in different requests. As these lists are sorted by “last-modified”, that means with before the returned record contains older data, and with after it contains newer data.

• Non-empty results can be considered immutable, and they will be cached as such. Use after to catch up with all new information.

• Keep reading the after token until you reach an empty response. Even if you request 100 records and only get 50, that doesn’t mean it is the end of the dataset.

When can I test this?

This new way of pagination will roll out with the new routes. The system we designed is generic and can easily be added to other new routes we have planned. Up first: Corporation Projects later this summer.

As this new system requires a different way of retrieving data from our datastore, we cannot easily adjust the existing routes. As such, they will remain offset-based for now. But if/when we give those routes an overhaul, we will surely be replacing them with this new system.

Wrapping up

Although this new system might seem complicated on the surface, working with it is a pleasure. You don’t have to be afraid of missing anything, and local caching has never been easier. There will be [a thread](https://discord.com/channels/940573867192221696/1398273731394146395) on the EVE Online Discord to discuss these changes. We will be interested in hearing your feedback on this.

Cheers,

Your friendly developers of the EVE Swagger Sandbox Ingress (working title)