ID:2344085
 
Problem description:

Hi guys,

I'm coming back to BYOND to see how things have changed with the new versions and I'm finding that the newest version seems to be incapable of creating big worlds. For example, I'm trying to create a 1000x1000x100 map file and every time I try to create it, it comes with the error "out of memory." I remember being able to create map files this big in previous builds on BYOND so is there something different that needs to be done now? Is there a way to allocate more memory for BYOND to use?

My PC specs are:
i7-6600U w/ 8GB of RAM on my laptop
i7-8700k w/ 16GB of RAM on my desktop

Both come up with the same error. And while I did originally try to create a new map file in an old build of a BYOND game I created, I started a new world and just tried to make a blank map with the same 1000x1000x100 size and same error: out of memory.
Yut Put wrote:
The solution to your problem is to not create a 1000x1000x100 map file lmao

What ungodly kind of game are you making that requires a map this big

What can I say...I have very big dreams...

Like I said, I thought I could remember doing it in previous builds of BYOND. Is it not possible anymore or am I remembering wrong?

EDIT: Just downloaded a build from 2013 and tried it again. I suppose I am wrong....

Still, is there no way to increase the amount of RAM that BYOND can use? Is BYOND still running at 32-bit?
In response to Akira Sieghart
Akira Sieghart wrote:
Is BYOND still running at 32-bit?

Yes.
So is there no way to get around this?
In response to Akira Sieghart
Making Dream Maker 64bit/multi-threaded was deemed unfeasible a while ago.
To be honest, I just mean creating larger maps. Even at 32-Bit, Dreammaker should be able to use ~4GB of memory. Is creating a 1000x1000x100 map file really using more than 4GB of memory?
1000x1000x100 map file really using more than 4GB of memory?

Probably. At 1000x1000x100, that's 100 million turfs. That means that you have at most about 43 bytes per turf before it hits 4GB.

So yes, almost certainly it is. In reality, you are limited to about 2GB per process by default... So... That's even worse, because you have a budget of about 21 bytes per turf. I'd wager each turf instance eats at least 100 bytes of memory.
What you need to do is exactly what game developers have done for eons: dynamically load and unload resources only when needed. There is no reason you should need 10^8 turfs simultaneously, and honestly there are probably no engines out there that could support loading all that stuff at the same time. Even if they could, it would be atrocious for performance.
Yeah, dynamically loading and unloading maps only when needed is the best approach as Popisfizzy stated. Not even the best engines are able to load all that data all at once due to RAM limitations. That's how Minecraft can easily handle very huge worlds without any issue. It loads in chunks (which has been that way since an earlier version) based on the area you're in.
I'm fairly certain they are joking because if we are being extremely generous they'd need around 4300GB of RAM to run such a map.

Is it any surprise they're out of memory?

Plus there is not a single sane person who would want to make a map that big. No way you can make it even remotely good. Would just be quantity over quality.
Hey guys, thanks for the responses. I was serious about my inquiry but I think you guys are right that what I want to do is just too big for any game engine right now. While it's true that not all 100 1000x1000 maps would need to be loaded at the same time, if you have hundreds of players moving around, I would assume quite a bit of them would need to be loaded simultaneously. While I do have ideas of how to use just about every tile in that 100,000,000 tile hypothetical map, I suppose I'll just have to wait until game engines get more advanced and optimized.
In response to Akira Sieghart
Akira Sieghart wrote:
I suppose I'll just have to wait until game engines get more advanced and optimized.

Or you'll have to think about how to actually approach it, instead of waiting for someone to do all the work for you. This isn't a matter of game engines being limited, this is a matter of running up against physical limitations for computers. The amount of RAM you would need to do this is absurd, which suggests that your idea is either bad, or the idea you have to implement it is bad.

What you have to do is partition your 100,000,000 tiles into manageable sections that you load and unload as needed. Players are not equidistributed along the game world: they will naturally clusters in areas, cities and dungeons and other quest areas. The far-off areas will generally be uninhabited, so you will not need to load them all into memory at once.

Also, I am suspicious that you would be doing anything interesting with all of these 10^8 tiles. Assuming you spend no more than 1 sec/tile on average when creating your map, you will be spending 38 months on your map. This suggests that instead you will be doing a lot of random generations on the map, which is far less interesting than a hand-crafted map.

The Elder Scrolls: Arena was a few times the size of Europe (c. 6 million km^2), and The Elder Scrolls II: Daggerfall was about half the size of Germany (c. 160,000 square kilometers). To contrast, TES III: Morrowind is far smaller (you can cross the island real-time walking in under an hour), but the entire map is hand-crafted and hand-populated. This approach was followed by the next two games in the series, and these three are far more popular than the first two. Now, correlation does not imply causation and there are other mitigating factors that would also lead to the latter three games to be played more (a transition to full 3D, modern hardware and OSs having issues playing the first two, etc.) but a common criticism of Arena and Daggerfall was that there was little of interest going on.

The point is that instead of saying that contemporary game engines are the issue (though another engine that was better at handling client-side stuff would mitigate this), perhaps you should consider that the way you want to approach things entirely wrong and comes from misconceptions.
I've played around with trying to make huge worlds in BYOND and regardless of your solution nothing, will be ideal.

The basic idea is to split your world into a grid and only load the sections with players on (and surrounding areas), then unload everything else. BYOND can do this with some effort but it is a pain to do and because BYOND doesn't support threading anytime you load a map from a file, the game will freeze briefly (larger maps will result in more freezing).

http://files.byondhome.com/TheMagicMan/seamless_src.zip
This is how far I got with it before deciding it just wasn't good enough for my needs.
The code is old, not optimised and probably buggy as hell, but it shows you how you could split a world into smaller slices and load them in as needed. (This example splits a 1000x1000 world into 400 50x50 slices).

You'd be better off just using another engine if you want to make a huge world.
The map that I was looking into would be about 10 square kilometers. Figuring that a single tile is about a square meter, that would put the map at 1000x1000 for a single square kilometer.

Of course not every tile of the world would have something for players to do but they would be heavily spread out between towns/cities and quests. While it's a big undertaking, I have a team of 19 other people behind me and we're trying to look into game engines that would work for a project of this size. We're looking specifically into 2D game engines because while we have a lot of talented coders and artists, our 3D modeling experience is very limited.

I've had experience with playing and developing (albeit much smaller) BYOND games in the past so this was my suggestion. While I do agree that there are ways to get around the memory issue, specifically by having the map load in chunks, I know that BYOND doesn't work very well using that method.
BYOND's infrastructure creates a lot of problems that developers of most games don't have to solve.

For most online games, for example, the server doesn't need the entire world loaded at once. The client does most of the work to determine what it sees and how it thinks the player should interact with it. All the server does from that point is verify that the client isn't cheating in some way.

That's significantly less overhead than is required of BYOND, where the server keeps the entire world loaded all at once for as long as it's running.

If you're using BYOND, you have to be smart, arguably smarter than the people developing games on kits like Unity or UE4.

If you really need a huge world (which you don't, not even for an MMO, but that's a design topic, not a developer one) then you should think about clustering out your server. Running multiple instances of a server across several ports on one machine would help you utilize more of your computer's resources and limit just how much crunching each individual server has to do.

You might do this by having each "zone" in a world loaded onto a separate server instance and clever use of the Topic() function to send the player across properly.

Now, you'd likely need to implement some kind of backend databasing for each server to pull data from like player's characters, equipment, inventory, etc, so it's consistent across all servers, but that's simple SQL.

Your front-end "game" would likely just be the login server which sends the player to the proper play-server once they've been validated.

I'M GETTING WAY AHEAD OF MYSELF, BUT YOU START TO UNDERSTAND JUST HOW COMPLICATED MAKING UBER-LARGE WORLDS BECOMES, NOT JUST IN BYOND BUT WITH ALL ONLINE GAMES IN GENERAL

My advice is to only open as much map space as you have stuff to fill it with. Making a "large" map isn't about volume, it's about density. A big map with nothing in it is just a big, empty, boring vacuum to get lost in, and not the good kind of getting lost in like Skyrim, but the shitty "everything around me for a hundred blocks is green" kind of lost.
If you want to really build such large maps, then don't use any tile-based engine. Instead, use specific ones designed to handle large maps. Good examples of such are sector-based engines where they deal with sectors (which are composed of linedefs composed of vertices) instead of tiles. Not only you can achieve incredible sizes with such an engine, but also accomplish it with far less memory used compared to tile-based engines.

What's even better about sector-based engines is you're not bound to squares. Instead, you can practically create areas of many shapes. I even know sector-based engines (primarily used for 2.5D games) that sometimes come with very user friendly level/map editors like BUILD (used by Duke Nukem 3D, Shadow Warrior, and a few other games).

Those aren't the only kind of engines that could tolerate large distances with little memory usage. There are others out there.
Thanks for all of the advise, guys. While the world we're looking into will be big, we're not necessarily trying to make it big for the sake of making it big. That being said, I do understand that BYOND isn't the best engine for a world like this.

Back when I was into the whole developing scene of BYOND, it was probably around 2011 or 2012 when every other game here was a DBZ, Naruto, or Bleach clone. BYOND has undergone quite the changes since then and I'm happy to see that the development side of BYOND is still active.

My team and I are still looking into other engines which may suit what we need. I appreciate all the help that you've provided, though.