Saving/Loading performances with turfs

ID:2606355
 
Aug 21 2020, 11:16 am
Basically, I've made a simple system that just saves and load the entire world, turfs, variables, etc.. Its using sqllite to save the data since I found it generally more flexible and much faster than the built-in save system.

I'm also using the "my first world" base project for testing, with some extra turf fields and types, also made the map 100x100.

Saving takes around ~10 seconds which is fine. But then loading the map takes about ~10 minutes, and makes the whole thing hang. By selectively disabling loading specific things from the save, I noticed that loading turfs was the big bottleneck here.

Here's the code loading the turfs for reference:
//Turf are special and gotta be created differently
/proc/DeserializeTurf(var/database/db, var/database/query/q, var/instanceid, var/list/rowdata = null, var/recursive_counter)
    var/start = world.timeofday
    //Skip if cached
    if(saved_instances[instanceid])
        return saved_instances[instanceid]
    if(isnull(rowdata))
        q.Add("SELECT * FROM [TABLE_INSTANCES] WHERE `[TBL_FIELD_ID]` = '[instanceid]';")
        q.Execute(db)
        CheckQueryErrors(q)
        if(q.NextRow())
            rowdata = q.GetRowData()
    if(!length(rowdata))
        world.log << "Missing data for saved turf '[instanceid]'"
        return null

    //Get our area
    var/turf/T
    var/area/myloc = rowdata[TBL_FIELD_LOC]
    if(!isnull(myloc))
        myloc = DeserializeInstance(db, q, myloc, null, recursive_counter)
    //Get our position
    var/turfty = text2path(rowdata[TBL_FIELD_TYPE])
    var/x = text2num(rowdata["x"])
    var/y = text2num(rowdata["y"])
    var/z = text2num(rowdata["z"])
    //del(locate(x,y,z))
    var/turf/oldturf = locate(x,y,z)
    if(oldturf.loc)
        var/area/A = oldturf.loc
        A.contents.Remove(oldturf)
    //turfs_to_delete += oldturf
    T = new turfty(locate(x,y,z))
    //Add us to our area
    if(istype(myloc))
        myloc.contents.Add(T)
    T.deserialize(db,instanceid)
    saved_instances[instanceid] = T
    //world << "Turf \ref[T]:area([T.loc]):([T.x], [T.y], [T.z])"
    //world << "##T \ref[T] loaded in [(world.timeofday - start) / 10] -- RECURSION [recursive_counter]"
    SSTATS.total_time_turfs += (world.timeofday - start)
    return saved_instances[instanceid]


I was wondering if there was anything to be done to make loading turfs from a save faster? Seems like having to overwrite existing turfs and create a new turf is really slow done this way, but I can't find much info on a faster way to do it?


Aug 22 2020, 7:53 pm
I have a question related to this. Every time I have ever made a build system I always made the build system based on OBJ's instead of Turfs. Do turfs save is the question.
Aug 23 2020, 7:47 am
Solidyote wrote:
I was wondering if there was anything to be done to make loading turfs from a save faster? Seems like having to overwrite existing turfs and create a new turf is really slow done this way, but I can't find much info on a faster way to do it?

Yes. First of all, I see that you have commented out the deletion part. That's good because creating a new turf will automatically overwrite the old one (only one turf can exist at a coordinate). However, you don't need to create new /turf objects at all. Instead, try changing the properties of the existing object.

With that, you can pare down the amount of data you need to save and load. Ideally you could save only what type of turf you need, and then just read all the necessary properties from a list. If you really need to save more data (such as how much damage the turf has taken) then it will take more time, so only saving the important stuff is advisable.

Also, this whole setup looks very inefficient outside of the actual saving and loading. All of this work is being done just to get one turf. I don't know a ton about database methods but I imagine that querying again for each data point is going to slow you down. Couldn't you grab a larger chunk of turfs at once? Furthermore, you are parsing coordinate information... can you avoid that by saving the turfs in order and then simply reading the file back in the same order?

Aug 23 2020, 7:53 am
In response to Raffile
Raffile wrote:
I have a question related to this. Every time I have ever made a build system I always made the build system based on OBJ's instead of Turfs. Do turfs save is the question.

Nothing 'saves' without your instruction. Turfs can be saved in the same way as objs, only difference is that their coordinates are included by default.

Aug 23 2020, 1:27 pm
In response to Magicsofa
Magicsofa wrote:
Raffile wrote:
I have a question related to this. Every time I have ever made a build system I always made the build system based on OBJ's instead of Turfs. Do turfs save is the question.

Nothing 'saves' without your instruction. Turfs can be saved in the same way as objs, only difference is that their coordinates are included by default.

This doesn't involve this post directly but I see. With what you just stated I am going to try to make a system to save and load turfs then and release it if I ever get it complete. I am used to just using obj's but I will attempt something like this myself as well.

Anyways Magicsofa seems to be onto something Solidyote. Hopefully, they have helped you.
Aug 23 2020, 11:03 pm (Edited on Aug 23 2020, 11:12 pm)
Saving turfs as individual records in a database is possibly the least efficient method of saving and loading turfs I can imagine. The fact that you got it working as well as you have is a testament to your insanity. Props, but also, you might consider something more tailored to your specific needs.


And to add to what Magicsofa said, not only can two turf instances not exist in the same space, but a turf instance IS its space. A turf can only be created when the size of the world is expanded, and a turf can only be destroyed when the size of the world is decreased.

When you delete a turf, it isn't destroyed. It is mutated into the default world turf. --This can lead to all kinds of weirdness, such as currently running procs from that turf now being active with a src object that the proc is not defined on.

When you create a new turf, it doesn't destroy the old one. Again, the turf is mutated into the new type.
Aug 26 2020, 11:20 am
Magicsofa wrote:
Yes. First of all, I see that you have commented out the deletion part. That's good because creating a new turf will automatically overwrite the old one (only one turf can exist at a coordinate). However, you don't need to create new /turf objects at all. Instead, try changing the properties of the existing object.

With that, you can pare down the amount of data you need to save and load. Ideally you could save only what type of turf you need, and then just read all the necessary properties from a list. If you really need to save more data (such as how much damage the turf has taken) then it will take more time, so only saving the important stuff is advisable.

I was trying several things to try to figure out the things not in the docs about turfs. I know that deleting them is kinda pointless, but I was curious as to what happened to the old turfs when I deleted them. As in, do they stay in memory, or stack under the new turf, and are they even like queued up like everything else by the GC, or deleted immediately?

The reason I'm having to create new turfs on load is because this is intended to be used for a SS13 server. In the SS13 codebases there are lots of different turf types with loads of data used to keep track of each tile's states. So, yeah, I can't really skip over saving turfs sadly.

Magicsofa wrote:
Also, this whole setup looks very inefficient outside of the actual saving and loading. All of this work is being done just to get one turf. I don't know a ton about database methods but I imagine that querying again for each data point is going to slow you down. Couldn't you grab a larger chunk of turfs at once? Furthermore, you are parsing coordinate information... can you avoid that by saving the turfs in order and then simply reading the file back in the same order?

And well, that depends on BYOND's implementation I guess? Databases are optimized for this kind of things. A lot of this is cached behind the scene, so its not as inefficient as it seems. Though, I should probably query more turfs at once that's true. I'm mainly trying to get the basics down for now. And I noticed how insanely disproportionately slow loading turfs is to loading everything else, or even to saving turfs themselves. So I'm just trying to deal with that first.
Also, I'm pretty worried about loading too much at once and running out of memory. Since this is meant to be used on a SS13 server, having the whole thing in memory several times would be pretty bad ^^;

Also, its probably inefficient, but its several orders of magnitude faster than the built-in saving system. So I'd still consider that an improvement.. ^^;

And I can't really just read it back in order, because from what I can see, several turfs may exist in a single spot? And that's something very likely to happen. I'm not yet 100% sure how to handle it, but at least I'm keeping the coordinate data for reference when debugging later on.



Ter13 wrote:
Saving turfs as individual records in a database is possibly the least efficient method of saving and loading turfs I can imagine. The fact that you got it working as well as you have is a testament to your insanity. Props, but also, you might consider something more tailored to your specific needs.

Well, I mean its not like I have many options? Since this is literally just a proof of concept to be adapted and used on a SS13 server, there's a ton of different types of turfs, each containing lots of data about the state of that particular tile in the world. Every single instances has to have its data stored somewhere.


Ter13 wrote:
And to add to what Magicsofa said, not only can two turf instances not exist in the same space, but a turf instance IS its space. A turf can only be created when the size of the world is expanded, and a turf can only be destroyed when the size of the world is decreased.

When you delete a turf, it isn't destroyed. It is mutated into the default world turf. --This can lead to all kinds of weirdness, such as currently running procs from that turf now being active with a src object that the proc is not defined on.

When you create a new turf, it doesn't destroy the old one. Again, the turf is mutated into the new type.
And that's also something else, there weren't much details at all in the reference on turfs about how stacked turfs works, or how the game decides what turf "wins" when you try to get the turf an atom is on.

And that sounds like turfs are even more messed up to work with than I thought ^^;

When you say "mutated", what would that be behind the scene? Is the mutated turf a new instance completely? Or are turfs one of those things that behave unlike anything similar, like say the "contents" list variable?

Also, what about stacked turfs? I basically completely ignored them so far because there's hardly any documentation on them, but it seems like they probably behave in a very hacky way?
Aug 28 2020, 4:38 am
In response to Solidyote
And I can't really just read it back in order, because from what I can see, several turfs may exist in a single spot?

No, there's only one /turf object per tile. Always. You are reading so much into turfs that you shouldn't be. This is what it says in the reference:

Turfs cannot be moved. They can only be created or destroyed by changing world.maxx, world.maxy, or world.maxz. When you create a new turf with new(), it always replaces the old one.

It is possible to stack turfs on top of each other in the map editor, but when you do that I'm pretty sure the top object retains its properties and just adds the icons of the turfs below it as underlays.