Massive

by FIREking
Massive is a real-time save/load chunk system that allows worlds to have tens of millions of tiles and objects saved and loaded efficiently.
ID:1333160
 
Massive is currently written as a demo but could easily be used as a library. However, there is a lot to explain here and using it would require to ditch BYOND's map editor entirely as you are now using maps that are bigger than Dream Maker allows! That said, its not quite ready to be a library because it would require a huge amount of the developer refactoring his code in order to be used with how this system works. Aside from that, feel free to play with the demo and check out the system!

Currently, BYOND is limited by having a 3 BYTE reference address per turf, which gives you a maximum allowed total number tiles of 16,777,216

#define CHUNK_SIZE 50
#define MAP_DEPTH 5

world/New()
..()

world.log << "Changing map size..."
chunk_system.SetMapSize(1750, 1750)
//This is a world of 1750x1750x5 which is 15,312,500 tiles!


Demo Controls:
w,a,s,d moves
space places a coin item
click on grass to create wall
click on wall to create grass
click "FillWithTrees" to put random trees in every chunk (triggers almost all chunks to become dirty instantly! saving will take a while!)
Player speed is 10 which is quite fast. This allows you to see that chunks can keep up with quick movement speeds, but in real-world situations where players move much slower, you'd really have NOTHING to worry about in terms of loading times and performance.

How it works:
The world is broken up into invisible chunks. Each chunk is a set of tiles. When the player moves around in the world, we tell the chunk system where the player is and the chunk system figures out which chunks to load. When the world is shut down, the chunk system saves all the chunks that were dirty (changed)!

Why it works:
We put most of the heavy lifting in the saving, which is something that only needs to happen after you're done making all of your modifications to the world (at the end of the day). Loading is pretty much instant, because we're only loading the chunks we care about seeing or need to know anything about. Since we don't need to know about all chunks at once, this is fast. We also only save chunks that are changed, if a chunk isn't changed there's no need to re-save it which is wasteful of the cpu. Chunks are also only loaded once. From there, the chunks remain in server memory until saving.

IMPORTANT:


1. Starting up the world will take a minute. This is because it simply takes BYOND/DM a really long time to change world.maxx, world.maxy, world.maxz and initialize all of the tiles.

2. The first save is usually a long one, this is because the more dirty chunks you have, the more there is to save and in a new world, you're going to have a lot of dirty chunks. In a live game situation, you'd have much less chunks dirty each day, and save-time wouldn't be a huge issue. There's just no way to get around it, you're trying to cram a lot of bytes into the hard drive and you're only gonna be able to do it so fast. The idea here is to only save chunks that are changed and to only do it once at the end of the game session. The system also gets faster as time goes on and chunks become more "stable" and less updated. If the server has longer up times, eventually you will reach a point where chunks are 100% loaded and there will be no extra strain on the world.cpu at all!

3. Any time you change a turf, drop an item, or pick up an item you should be marking the chunk as dirty. Failure to do so could cause some really nasty issues, such as item duplication. Imagine the player logging in 2 minutes before server reboot, he picks up something on the ground but it doesn't set the chunk as dirty, server reboots and player logs back in... the item is loaded in the chunk because the chunk wasn't marked as dirty and therefore it wasn't saved. There are some built-in safety measures to try and manage this for you, but ultimately your game is going to be different than any other game so its up to you to tell the chunk system when a chunk is classified as dirty.

4. If you save chunks marked as dirty at run time, the server will freeze depending on the number of chunks to save. This is because the system is designed to save chunks when world is deleted (so it hogs all of world.cpu to do so). If you must save chunks at run time, you're going to have to put in safety measures that prevent players from changing chunks while chunks are saving. The chunk system itself will lock the chunks, but that doesn't factor in your game logic. If you let people put items on the ground while a chunk is locked (without first checking if that chunk was locked), well the chunk system might not know about that item. It is best and recommended to just save chunks when the world shuts down. I threw in a "Save" verb so you could see how it works if you wanted to, but its definitely not intended to be called during a game session.

5. Concurrent sessions (two worlds saving and loading from the same chunk folder) is really bad and you should totally avoid doing that! I can't guarantee what type of nonsense could happen!

6. Quick Sessions: It is possible to close the world then start the world again immediately after while the first world appears to be gone but is actually still saving in the background. This is because when you run the server and client in the same thread (ala hosting from dream seeker) seeker will disappear after world/Del is called but the world is still running in the task manager. Be careful not to run the world too many times too quick or you'll risk concurrent file saving and loading which is dangerous to the data!

7. Turfs: Currently only location and type are saved for turfs. If you wanted more info saved about a turf, it would be fairly easy to implement by just adding more data points to the turf's save string.

8. Objs: Everything about an OBJ is saved

9. Mobs: Saving mobs is beyond the scope of this demo. Saving mobs should be handled in your own game-specific way.

Edit: I added a time-delay for in-game saving which makes it take longer but allows you to move around as its saving. I don't recommend editing chunks while chunks are saving. The chunk system can't predict the result.