Dev Insights Blog

Hi all,

Today we're announcing a complete rework of how we build and distribute the Static Data Export (SDE). The new SDE is now available at developers.eveonline.com/static-data and will eventually automatically build with every Tranquility deployment. There will also be a discussion thread on Discord for Q&A and feedback from third-party developers.

But why?

Historically, the SDE has been built alongside the server and client. This approach worked well initially, but it created some challenges over time.

The build process broke a few weeks ago. Fixing it was non-trivial and highlighted some other fundamental issues with how we were handling the SDE. The biggest problem was that we had to be extremely careful about what data was included for the server versus what was meant for the client/SDE. We didn't want to accidentally leak information that shouldn't be public, but this made adding or changing data in the SDE difficult.

Introducing: Client-Driven SDE

We've reworked how the SDE is created. Instead of building it alongside the server and client, it is now generated entirely from data that's already available in the client.

The new system uses a single "mapping" file that tells us how to convert client data into SDE data. This mapping approach gives us much more control over what gets included and how it's structured. We can also transform some data to create easier data structures for third-party developers to work with. The result is a cleaner, more maintainable system that's much easier to update and modify.

The new SDE is now available at developers.eveonline.com/static-data. It automatically builds with every Tranquility deployment, but is currently behind a manual gate before it goes live. This gives us time to evaluate our new system, ensuring the data is correct. Delays between Tranquility updates and SDE releases should be minimal.

Backwards (in)Compatibility

This rework is not backwards compatible. Several structural changes will require updates to existing tools and applications:

• The old bsd folder is now removed. This data has been integrated into other files where it makes more sense.

• The universe folder no longer exists. This data is now part of the map files and follows the same structure as other SDE files.

• Some fields that weren't exported to the client are no longer available in the SDE.

• Parts of types have been split into masteries and typeBonus for better organization.

All schema changes are documented in the schema-changelog.yaml file.

New Features and Improvements

While reworking the SDE, we've also added several new features and improvements:

• New files with data that was previously missing from the SDE:

  • agentTypes

  • dbuffCollections

  • dogmaUnits

  • dynamicItemAttributes

• More consistent types, translations, and naming throughout (no more nameID field for strings; it is now just name)

• Added the JSON Lines format alongside YAML

The JSON Lines format was added primarily because our build pipeline had trouble working with mapMoons in YAML format. JSON Lines can be streamed, making it faster and less memory intensive for large datasets. This should be particularly useful for developers working with the map data, and could speed up your ingestion pipeline by a lot.

Wrapping Up

This rework represents a significant improvement to how we build and distribute the SDE. While there are some backwards compatibility changes to work through, the benefits far outweigh the costs. The new system is more maintainable, more reliable, and easier to work with.

There's still more work to be done, but this is a solid foundation for the future. We're looking forward to hearing your feedback on the new SDE and how it works with your applications.

Cheers,

Your friendly developers of EVE Static Information (working title)