Storing really very quite big online-game universes

[Findthepin1]

So there's this game called EVE Online. It has remarkably asymmetrical spaceships and some of their Earthlike planets are over half the radius of Jupiter. Anyway, the game's universe has over 5000 star systems. They seem realistically enough spread out. I don't understand how they are able to store all that information, for every meter of every planet in 5000 solar systems, plus everything on them and everything in space. It would probably take up tens of terabytes at the very least, or petabytes at the most. Likely, they don't have that kind of capability. What am I missing?

[RainDreamer]

Procedural generation? Just need a seed and run it through algorithms that would put things where they want them to be. So instead of storing all the data at once, they just need to keep the seed, and everything else will be loaded on demand. We have powerful enough machines to do such calculations in a flash, even if we don't have the capacity to store that much data all at once. Likely they will also unload sectors that are not being used too, to reduce loads on servers.

It is like, instead of building a whole theme park for you to walk around, they have an army of super human moving at flash speed to build everything you are about to interact with, and dismantle everything you left behind. The actual space needed is very little.

But this is just my guess, I have no idea what they do there. Edited November 17, 2015 by RainDreamer

[GeneralVeers]

The planets could be generated the same way Minecraft worlds are. Every Minecraft world is merely the result of a seed number shoved through a fairly complex mathematical formula that spits out a three-dimensional graph as a result (Perlin noise being one of the major parts of the formula). And Minecraft only loads those parts of a world that are within sight range of a player. The entire world (which is around eight times the square footage of real-world Earth) never needs to be stored anywhere. Only the seed number, the locations of the players and their inventories, and any changes that have been made to the world.

As to the storage: a hard drive is around a hundred bucks a terabyte these days. Slower media (such as the optical drive that reads your DVD-ROMs) are cheaper per gigabyte. A company that invests ten thousand dollars or so can have plenty of storage.

[Jovus]

You're missing that they do have that kind of storage.

Each system is on a different server slice. The game has literal millions of players paying $20/mo; data storage is cheap.

[Stargate525]

[quote name='WedgeAntilles']Only the seed number, the locations of the players and their inventories, and any changes that have been made to the world.[/QUOTE]
Unfortunately, Minecraft keeps firm data on any subsection of the world that has ever been loaded. It generates once, then stores the lot. That's why there are worlds that are several tens of gigabytes in size.

[Yourself]

[quote name='Findthepin1']So there's this game called EVE Online. It has remarkably asymmetrical spaceships and some of their Earthlike planets are over half the radius of Jupiter. Anyway, the game's universe has over 5000 star systems. They seem realistically enough spread out. I don't understand how they are able to store all that information, for every meter of every planet in 5000 solar systems, plus everything on them and everything in space. It would probably take up tens of terabytes at the very least, or petabytes at the most. Likely, they don't have that kind of capability. What am I missing?[/QUOTE]

What you're missing is that they simply don't store that much data because there's no need to. Interaction with planets in EVE is pretty limited. You can't actually travel to the surface. In fact, you can't even really get close enough to see much surface detail, so there's not much data required. The distance between planets is largely irrelevant in terms of the data requirements (beyond additional memory required to store larger numbers accurately, but this memory requirement would be negligible next to literally anything else; it's on the order of bytes).

Assuming the planet surface textures aren't generated procedurally and that each planet is truly unique, the memory requirements for all planets together would probably be on the order of hundreds of GB, which honestly isn't that much data for servers to be working with.

I would assume there's some amount of procedural generation simply because of how expensive it would be to have artists create several thousand unique planets.

[YNM]

I guess with all the warps and whatnot it's not as resource-heavy as, say, n-body KSP. After all, you don't need to know where everyone else is - and centralized system should handle the whole game easily, if they use the decimals to the last place amd set that to ridiculous size.

[stibbons]

[quote name='Stargate525']Unfortunately, Minecraft keeps firm data on any subsection of the world that has ever been loaded. It generates once, then stores the lot. That's why there are worlds that are several tens of gigabytes in size.[/QUOTE]

Minecraft is a different issue. Initial terrain generation is done using a procedural algorithm, but after that the whole thing has to be stored, because it's so easily modifiable. My first thought would be to instead of storing the whole world just keep track of a delta - blocks that have been added to or removed from the procedural terrain. But no idea whether that would be more efficient or not.

Remove the terrain deformation requirement and you can be a lot more efficient. My favourite example of procedural world generation is still the original Elite. 8 galaxies with 256 planets each, running on a BBC Micro with 32KB of RAM.

[GeneralVeers]

[quote name='Stargate525']Unfortunately, Minecraft keeps firm data on any subsection of the world that has ever been loaded. It generates once, then stores the lot. That's why there are worlds that are several tens of gigabytes in size.[/QUOTE]
Yup. Been there, done that. Watched Kurtjmac do it. He's up to 17 gigabytes, if I recall correctly.

The thing is, even the parts of a Minecraft world that have never been loaded still exist in a logical sense. The same seed will always generate exactly the same planet, every single time. In fact, here's a way Minecraft could save some disk space, if they desired to add it: any terrain chunk that has been generated [B]but not changed by the player[/B] can be deleted from the world files when the player leaves load range. Then re-generated when the player returns. And everything will be exactly the same.

Eve Online or any other such computer game could do the same thing, storing only those things the player changes and using the Magic Formula to create the terrain every time anybody flies, walks, or lands nearby.

[SargeRho]

While EVE's 7600 solar systems would seem to be huge, they're actually very tiny, data wise. A long-time player's inventory probably takes up more space than a solar system does, since EVE's systems aren't particularly complex. An 'empty' solar system is probably no more than an MB in size. Since planets don't move in EVE, all you need to store a solar system is a list of objects in it, and unique properties. Things like player ships, containers, wrecks, quickly fill that up though.

There's a much better example of an insane game world: Elite Dangerous, with its 400 billion star systems. Edited November 17, 2015 by SargeRho

[andrewas]

[quote name='stibbons']Minecraft is a different issue. Initial terrain generation is done using a procedural algorithm, but after that the whole thing has to be stored, because it's so easily modifiable. My first thought would be to instead of storing the whole world just keep track of a delta - blocks that have been added to or removed from the procedural terrain. But no idea whether that would be more efficient or not.[/QUOTE]

It's would be, if storage were the major problem. In most worlds, most chunks are passed through, or have some trees cut down, a few blocks taken from the surface or a mining tunnel passes through. A relative handful of chunks get huge excavations or large buildings. But storage is cheap, bandwidth isn't that expensive, and worldgen is surprisingly expensive.

[SomeGuy12]

[quote name='Findthepin1']So there's this game called EVE Online. It has remarkably asymmetrical spaceships and some of their Earthlike planets are over half the radius of Jupiter. Anyway, the game's universe has over 5000 star systems. They seem realistically enough spread out. I don't understand how they are able to store all that information, for every meter of every planet in 5000 solar systems, plus everything on them and everything in space. It would probably take up tens of terabytes at the very least, or petabytes at the most. Likely, they don't have that kind of capability. What am I missing?[/QUOTE]

First of all, the planets don't have 1 meter detail, do they? You cannot get that close to the surface.

And second, yeah, you can define a whole planet with 1 meter detail with a handful of random numbers that would fit in this sentence. You generate the actual terrain as needed, or you cache just the portions that players actually have visited. This is how minecraft does it. (notably, the minecraft world does have a practical size limit, but it has to do with the binary size of the numbers used and not the fundamental technique)

The real problem with this kind of random generation is that it isn't very interesting. Randomly rolling terrain gets boring to explore after a while, and the ultimate limit is the actual unique resources, such as the textures for the terrain blocks and biomes and such.

[magnemoe]

[quote name='RainDreamer']Procedural generation? Just need a seed and run it through algorithms that would put things where they want them to be. So instead of storing all the data at once, they just need to keep the seed, and everything else will be loaded on demand. We have powerful enough machines to do such calculations in a flash, even if we don't have the capacity to store that much data all at once. Likely they will also unload sectors that are not being used too, to reduce loads on servers.

It is like, instead of building a whole theme park for you to walk around, they have an army of super human moving at flash speed to build everything you are about to interact with, and dismantle everything you left behind. The actual space needed is very little.

But this is just my guess, I have no idea what they do there.[/QUOTE]
Procedural generation don't require much resources. Its basically an random number with an fixed seed.
You can also override it with fixed stuff. Mun surface details are procedural.
You will anyway have to override with the savegame who will change stuff

[Stargate525]

[quote name='andrewas']It's would be, if storage were the major problem. In most worlds, most chunks are passed through, or have some trees cut down, a few blocks taken from the surface or a mining tunnel passes through. A relative handful of chunks get huge excavations or large buildings. But storage is cheap, bandwidth isn't that expensive, and worldgen is surprisingly expensive.[/QUOTE]
It's not the storage, it's the read time. As the world gets bigger, the time it takes the game to look up whether it has already generated said chunk increases.

[SomeGuy12]

[quote name='Stargate525']It's not the storage, it's the read time. As the world gets bigger, the time it takes the game to look up whether it has already generated said chunk increases.[/QUOTE]

Minecraft is a hacked together mess put together by a single....extremely wealthy...guy. It's written in both a very slow programming language and the code itself is very slow. Notably, chunk generation is quite slow - just running around the Minecraft world, the whole game hitches and chugs as it slowly creates new chunks. Another issue is the chunks are made inefficiently - bedrock to ground level gets made, stored, and sent over the network connection for each chunk, even though the majority of the time, the player will never dig into a particular chunk and thus never see the blocks underneath the surface layer. (obvious solution is cubic chunks, a solution several clones of Minecraft have used)

The Windows 10/tablet version of Minecraft was written in C native code by a team of experienced programmers. If you have tried it, you'll see chunks snap into being basically instantly with no noticeable hitching.

[CleverWalrus]

[quote name='Findthepin1']So there's this game called EVE Online. It has remarkably asymmetrical spaceships and some of their Earthlike planets are over half the radius of Jupiter. Anyway, the game's universe has over 5000 star systems. They seem realistically enough spread out. I don't understand how they are able to store all that information, for every meter of every planet in 5000 solar systems, plus everything on them and everything in space. It would probably take up tens of terabytes at the very least, or petabytes at the most. Likely, they don't have that kind of capability. What am I missing?[/QUOTE]

Hello,

I have played EvE online for something like 10 years, and i have a pretty good knowledge on how games are made (used to be a game developer before selling my soul to enterprise business).

Storing information about the EvE Online universe is a LOT simpler and less data intensive then for example a huge game map like Fallout 4, the reason is very simple and relies on these 2 concepts:
- You just scale the "frame of reference" of your coordinates, for example you don't need to measure the distance between 2 planets in 2 different solar systems, you just need to know where are the planets inside their "frame of reference" in this case the solar system itself. You don't need to know their position "precisely" at all, you just need a rough estimation on where you will render them on the user client. This concept adapts to the whole EvE Universe, you just take in account the current "frame of reference" and you just ignore anything else because you don't need those distant things to interact with eachoter.
- The planets are very nice but they are not "unique" at all :) years ago they were just a good amount of "textures" for planets, now i think they have generated the textures to make them look different but there is nothing like "every meter of every planet in 5000 solar systems" you just need to input a bit of "variation" and you are fine :)

Anyway the "static data" (including a lot of positional information) of the EvE Online universe are available at this url: [url]http://community.eveonline.com/news/dev-blogs/eve-universe-static-data-export/[/url] and it's just a few mb :D

Have fun :)