Maps in memory. in BYOND Discussion

Page: 1 2
ID:132365
 
Mar 10 2011, 11:22 pm
I was looking at this.
http://www.byond.com/developer/forum/?id=777060

I then created a 400x400x400 map.
It took about a 1.5g of memory, and saving it took a while.
I then restarted dm, and opened the map file.
The memory went up to where it was before, then crashed.
Compiling it also causes a crash.

Instead of having the entire map in memory, Why not limit the loading of the map into memory some way?
Maybe, by only loading the Z above and below the current one?

Side Note:
Would anybody support having the option of making the map file independent of the dmb? It seems to be somewhat responsible for long compile times.
#1 Mar 10 2011, 11:46 pm
I would agree with that. I wished that we only had to compile the map only when it was edited.

Like unnecessarily taking time to go through dmm would take a lot of time and would be pointless.

Would it be ok, if we could keep an intermediate file (like a plug into .dmb format) stored on the hard drive,so that each time it compiles(without the map edited) you could just take that right from the hard drive and insert it into the .dmb.

Edit:
However I do not ever know if that could work.
Since I have no idea on how the dmb is made.


#2 Mar 11 2011, 2:35 am
In response to Quixotic (#1)
The issue I would potentially see regarding that idea (I haven't looked at the documentation) is the anonymous object constructors that are made when you alter map object values (change that particular trap to do 10 damage etc) in the map editor. That creates separate functions, which are tied to the object model in code, which you may have changed.
#3 Mar 11 2011, 4:55 am
400x400x400 is at least 64 million objects. That's 64 megabytes if they're 1 byte per tile which is impossible, it'll be at least 100 bytes per tile, making 6,4 GB in size.
#4 Mar 11 2011, 6:32 am
In response to Ripiz (#3)
Turfs take about 21 bytes each on a blank map. So if we times 64mb x 21b we get 1.344gb as an estimate.

As proof, create a 100x100x100 map in a new environment, it takes less then 40mb in task manager.
#5 Mar 11 2011, 9:13 am
In response to Chowder (#4)
Not entirely true. If a turf has any defined variables, procs, verbs and so on, then it will most likely take more memory.

I tested this myself a long time ago, a 1000x1000 map with random turfs takes up anywhere from 80-100mb of ram. If you add a few vars to each turf, then the ram usage does jump slightly.

Although in most cases you wouldn't need it, having the ability to load and unload maps would certainly be nice.
Even nicer would be the ability to load and unload certain sections of a map (if the map is 1000x1000 and the player can only see a small section of it, there is no point in loading the rest of it until the player gets close enough to be able to see/interact with it).
#6 Mar 11 2011, 10:03 am
Chowder wrote:
I then created a 400x400x400 map.

Memory isn't your problem then. That map is 64 million turfs, which is about four times DM's actual upper limit.

I'm not sure of the actual memory size of maps because there's some space-saving that goes in when the turfs are "stock", that is if they don't have any vars assigned beyond their base type. I know the 21-byte estimate is incorrect because of byte alignment issues, but I'd have to look in the source to be certain.

At any rate, turfs only let you go up to 3-byte identifiers, so the upper limit is a little over 16.7 million.

Lummox JR
#7 Mar 11 2011, 5:35 pm
In response to Lummox JR (#6)
If its above a limit shouldn't DM stop people from making a map that big? I know the 21 byte estimate is off. My methods of calculating that number were ruff at best, but were accurate enough to discredit Ripiz's estimate. The problem I have with the current way the map is dealt with is that if you have a very large map, it shouldn't be loaded completely in dream maker, and in some cases while it's running.
#8 Mar 11 2011, 5:39 pm
In response to The Magic Man (#5)
Well I know it's not applicable to every situation. Thats why I said a new environment. I was speaking of the memory consumption in dream maker, but having it also do this while it was running would save a lot of memory as well, though the functionality would have to be opt-in.
#9 Mar 12 2011, 2:51 am
In response to Chowder (#4)
21 bytes? Weird... I'd imagine it's much more with all the variables:
  • 2 bytes for each coordinate = 6 bytes
  • density, opacity, visibility, direction, luminosity = 5 bytes
  • layer = 2 bytes(?)
  • contents, underlays, overlays = 12 bytes
#10 Mar 12 2011, 5:34 am
In response to Ripiz (#9)
They know the turf's type. If you haven't changed any of those values from their compile-time values, it probably doesn't bother storing any information.
#11 Mar 12 2011, 8:09 am
In response to Forum_account (#10)
Forum_account wrote:
If you haven't changed any of those values from their compile-time values, it probably doesn't bother storing any information.

This is true. If you have ever messed with savefiles, this is a noticeable behavior. Any variables defined at compile-time that are not changed during runtime, and a savefile is created, any variables that are equal to the compile-time value are not stored.
#12 Mar 12 2011, 9:03 am
In response to Ripiz (#9)
2 bytes for each coordinate = 6 bytes

As Lummox Jr stated above " turfs only let you go up to 3-byte identifier". Meaning the location could be a byte for each coordinate, or a unique incrementing number with a 3 byte limit for each turf with modulo being used to determine the coordinates.


density, opacity, visibility, direction, luminosity = 5 bytes
layer = 2 bytes(?)
contents, underlays, overlays = 12 bytes

As Lummox Jr also stated above "there's some space-saving that goes in when the turfs are "stock"".

Anyways, can we please go back onto topic? I really have no desire to figure out the exact number of bytes each turf takes up...
#13 Mar 12 2011, 9:55 am (Edited on Mar 12 2011, 10:01 am)
In response to Chowder (#12)
> As Lummox Jr stated above " turfs only let you go up to 3-byte identifier". Meaning the location could be a byte for each coordinate, or a unique incrementing number with a 3 byte limit for each turf with modulo being used to determine the coordinates.

2^10 = 1024, enough to store one coordinate, that's 10 bits.
3 bytes makes 24 bits. It's not enough for all three coordinates, unless Z level has different limit.
#14 Mar 12 2011, 10:21 am
In response to Ripiz (#13)
The coordinates are stored as 16-bit numbers, but that's irrelevant. The other vars that were mentioned (Luminosity, Density, Opacity) are easily crammed into one byte and some of the other stuff can easily be condensed. The reference number is irrelevant to the location.
#15 Mar 12 2011, 12:09 pm
In response to Ripiz (#13)
24 bits
000000000000000000000000
2^24 = 16,777,216 unique combination's

Lets me try and explain this...
Assume every tile has a incrementing reference number starting from 1 and that this map is 5x3x2 and it has each tile show it's incrementing ref number

1 2 3 4 5
6 7 8 9 10
11 12 13 14 15

16 17 18 19 20
21 22 23 24 25
26 27 28 29 30

It's pretty easy to figure out the x, y, and z from just this incrementing reference number if you know the maximum numbers.

If you read this http://www.byond.com/developer/forum/?id=777111, It backs up my theory. Although, my map is upside down compared to how byond does it.

I really don't care how many bytes it needs. Please go back onto topic.
#16 Mar 12 2011, 12:14 pm
In response to Audeuro (#14)
I'm not speaking of the actual reference number that's accessible with the ref macro, but an internal one used to determine the positions of tiles.

Edit:
Scratch that. I can calculate the x,y, and z from the ref macro for turfs. Further testing is needed to see if it's reliable.
#17 Mar 12 2011, 12:18 pm
In response to Chowder (#16)
That's the reference number LummoxJR is speaking of. The turfs themselves would probably be stored in a 3D array or a 2D array and just accessed by their X,Y, and Z coordinates.
#18 Mar 12 2011, 12:19 pm
In response to Audeuro (#17)
No it's flat.
http://www.byond.com/developer/forum/?id=777111
http://www.byond.com/developer/forum/?id=777145

It's extremely easy to make a flat array act like a 3D array
#19 Mar 12 2011, 12:20 pm
In response to Chowder (#18)
Chowder wrote:
It's extremely easy to make a flat array act like a 3D array

He said the base turf info. He didn't give any insight into the actual map's layout. Either way, as you pointed out you'd still use the X, Y, and Z coordinates to reference the tile.
Page: 1 2