DreamDaemon crashes every 3/4 hours

Page: 1 2
ID:1821667
 
Mar 31 2015, 7:06 am
BYOND Version:507
Operating System:Linux
Web Browser:Chrome 41.0.2272.101
Applies to:Dream Daemon
Status: Open
Issue hasn't been assigned a status value.
Descriptive Problem Summary:

The game crashes. It doesn't freeze or appear to be running out of RAM as far as I've noticed. DreamDaemon just dies and throws out a message in the logs telling me to send in a bug report.

I'm currently hosting Phoenix: Sundered Earth of which the hub can be found here: http://www.byond.com/games/Archonex/PhoenixSunderedEarth

I'm running Debian 7 - 64bit

The game tends to peak around 110, I've seen it reach a 150 last week or so. I'm not sure if that is related information but I'm trying to not leave out any information.

BYOND version running: 507.1258

Numbered Steps to Reproduce Problem:

The cause of the problem is unknown to me.
The codebase has not been changed for months now and I'm having a hard time figuring out if something is really broken on my end or it is BYOND.
BYOND does however print out a backtrace and I have a bunch of them. The most recent being:


Does the problem occur:
Every time? Or how often? 2 / 3 hours
In other games? No
In other user accounts? No
On other computers? Yes, this doesn't seem hardware related

When does the problem NOT occur? When there are less than 20 players.

Did the problem NOT occur in any earlier versions? If so, what was the last version that worked? (Visit http://www.byond.com/download/build to download old versions for testing.)

This problem has been present since 503 if I recall correctly.

Backtrace:
Welcome BYOND! (5.0 Beta Version 507.1258)
The BYOND hub reports that port 1202 is reachable.
BUG: Crashing due to an illegal operation!

Backtrace for BYOND 507.1258 on Linux:
Generated at Tue Mar 31 11:54:14 2015

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bb44]
libbyond.so [0xf72a3000, 0x0], 0x1bbcfb
 [0xf779f000, 0xf779f410], [0xf779f000, 0xf779f410]
libbyond.so [0xf72a3000, 0x0], 0x1bbcfb
libbyond.so [0xf72a3000, 0x0], 0x1bc090
libbyond.so [0xf72a3000, 0x0], 0x1c12b9
libbyond.so [0xf72a3000, 0x0], 0x1c8a8b
libbyond.so [0xf72a3000, 0x0], 0x20352c
libbyond.so [0xf72a3000, 0x0], 0x222232
libbyond.so [0xf72a3000, 0x0], 0x22a67e
libbyond.so [0xf72a3000, 0x0], 0x232d43
libbyond.so [0xf72a3000, 0x0], 0x2336a0
libbyond.so [0xf72a3000, 0x0], 0x218606
libbyond.so [0xf72a3000, 0x0], 0x2116bf
libbyond.so [0xf72a3000, 0x0], 0x22a44c
libbyond.so [0xf72a3000, 0x0], 0x22bbe5
libbyond.so [0xf72a3000, 0x0], 0x1f3cc3
libbyond.so 0x2f62e0, 0x2f642a
libbyond.so 0x2c4c40, 0x2c4e42
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804aef4]
libc.so.6 0x16d60, 0x16e46 (__libc_start_main)
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a781]

Recent proc calls:
/mob/proc/Cancel_Meditation
/mob/proc/Cancel_Digging
/mob/proc/Cancel_Training
/mob/proc/Stop_TrainDig_Schedulers
/mob/proc/Knockback
/mob/proc/MeleeAttack
/proc/sign
/proc/getline
/proc/sign
/proc/sign
/proc/getline
/proc/sign
/proc/sign
/proc/getline
/proc/sign
/proc/sign

Mar 31 2015, 7:21 am
Version 507.1258 is way, way out of date now. If you can get results with a newer build that would be helpful. I know there were several bugs quashed early in 507's development and I don't want to go chasing something I might have already fixed.
Mar 31 2015, 7:57 am
I'll rehost with the latest (Beta) build and throw you a n ew backtrace if (when) it crashes.
Apr 1 2015, 9:57 am
It crashed again today. I have a core and trace file as well if so desired.

BUG: Crashing due to an illegal operation!

Backtrace for BYOND 507.1279 on Linux:
Generated at Wed Apr  1 14:30:12 2015

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bb74]
libbyond.so [0xf7298000, 0x0], 0x1ba77b
 [0xf77a5000, 0xf77a5410], [0xf77a5000, 0xf77a5410]
libbyond.so [0xf7298000, 0x0], 0x1ba77b
libbyond.so [0xf7298000, 0x0], 0x1bab10
libbyond.so [0xf7298000, 0x0], 0x1c6559
libbyond.so [0xf7298000, 0x0], 0x1d6d33
libbyond.so [0xf7298000, 0x0], 0x20673c
libbyond.so [0xf7298000, 0x0], 0x22531a
libbyond.so [0xf7298000, 0x0], 0x22d6de
libbyond.so [0xf7298000, 0x0], 0x235da3
libbyond.so [0xf7298000, 0x0], 0x236710
libbyond.so [0xf7298000, 0x0], 0x21b7a6
libbyond.so [0xf7298000, 0x0], 0x2148df
libbyond.so [0xf7298000, 0x0], 0x22d4ac
libbyond.so [0xf7298000, 0x0], 0x22ec45
libbyond.so [0xf7298000, 0x0], 0x1f6fa3
libbyond.so 0x303bc0, 0x303d0a
libbyond.so 0x2d2370, 0x2d2572
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804af24]
libc.so.6 0x16d60, 0x16e46 (__libc_start_main)
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a7b1]

Recent proc calls:
/mob/proc/Cancel_Meditation
/mob/proc/Cancel_Digging
/mob/proc/Cancel_Training
/mob/proc/Stop_TrainDig_Schedulers
/mob/proc/Knockback
/mob/proc/MeleeAttack
/mob/Move
/proc/sign
/proc/getline
/proc/blankBlocked
/proc/Mob_Range
/proc/blankBlocked
/proc/getline
/proc/sign
/proc/sign
/proc/getline
Apr 1 2015, 10:49 am
Thanks. I'll look at this right away.
Apr 1 2015, 11:59 am
My preliminary trace says this is happening when a new unique turf type is added, which would happen if the turf's area or appearance changed. Judging by your previous info, I suspect it was crashing in the same place there as well.

The place where the crash is happening pretty much only makes sense if the memory allocation failed, or if there was overflow. The unique turf array is allocated several items at a time, and it's possible if you were near that limit that it overflowed and tried to allocate 0 space. If this is the case (it's all I can guess), it'll be fixed in the next release. However, that would still put you very much up against that limit, which is not a good place to be.

Are you doing anything with turfs like assigning lots of different names (like "turf 1,2,3" or something), or using a zillion areas (I mean lots and lots), or anything like that? The number of unique turf types shouldn't be all that high, which is why we only support 64K of them. Factors that impact uniqueness are type, appearance, and area. Vars that don't impact appearance shouldn't affect the uniqueness test, nor should contents.
Apr 2 2015, 6:32 am
That sounds pretty wrong, yea.

I'm not doing anything that huge with the base game itself (when it starts) but players are capable of placing turfs to create their own buildings.

In order to keep the weather 'outside' of buildings, each player-made turf that is place has the an area 'Inside' placed on top of it.

Now that I'm looking at the actual saved files, the size of the save file that loads all these turf/area changes is around 60MB.

I guess the build system is really inefficient...?
Apr 2 2015, 7:24 am
The main thing is we're looking at unique info here. For the sake of nomenclature I'll call them cell types. Each cell type has its own turf type, area ID, and appearance.

If a turf is given a new appearance--which could be a change to icon, state, dir, layer, name, luminosity, etc.--that could create a new cell type. (Appearances, like cell types, are shared, so any distinct combination of values is a single appearance.) If a turf has a different type path, that could create a new cell type. And if the area changes, that can too.

If several turfs have type /turf/door, appearance 12, and area ID 3, they all share the same cell type. The issue I'm seeing is that, if I'm reading this crash right, you're coming up on the cell type limit--64K distinct combinations of type, appearance, and area--and our current code isn't handling that properly. Once the update is in, you'll still be in danger of hittng that limit if I'm right about what's happening.

So the big question at hand is: How many turf types are you using? How much variation, per type, is there in the appearance? If most of your turfs use 47-state autojoining and you have a lot of variation in how things are placed (thus using more of those states), there's another way for appearances and therefore cell types to creep up on you.

One thing I'm not positive about is if cell types get recycled. I'm not certain about that, so I'll look into it. If they're not, then anything that causes massive churn, creating lots of one-off appearances that are used and then discarded, would be an issue.

[edit]
There is a garbage check. It's done every 600 ticks, which is 1 minute in standard game time. It's not what I would consider perfect, as once any garbage is found it won't recheck until the array is all full up again, but it still ought to be hard to creep up to 64K cell types.
Apr 2 2015, 7:39 am
Alright, this might prove even more difficult to address than I imagined.

Players can build turfs, create items and clothes, change their icons, change icons of clothes and objects either created by them or others. These things can be renamed as well.
There a 67 turfs players can place themselves.
There are 105 icons as possible choices for clothes.
Edit: They can upload their own icons to modify the above.

And I just realised that everything, including hair and body icon can also be coloured at will. (I'm guessing this counts towards the limit too?)

Those with admin rights tend to customize entire areas for special occasions and the like as well.

I have no numbers (I'm not sure how to get them either), but if I understand you correctly then this does not sound like a bright future.
Apr 2 2015, 8:46 am
In response to Valekor
Objs and mobs won't impact this at all. Only unique turfs would be an issue. Honestly I have no idea how a game could easily creep up to the cell type limit without the map being absolutely huge and extremely varied.

One thing I'm curious about is if your game runs into this issue when no one logs in and the session merely stays inert. If so, that's something I could test myself.
Apr 2 2015, 11:10 am (Edited on Apr 2 2015, 11:19 am)
I've run that test in the past myself. I couldn't get it to crash on purpose. We don't have that many unique turfs as far as I'm aware but if you'd like to test I can always email what you need.

Edit: What about the ability to damage a single turf? Would that affect it?
Apr 2 2015, 11:17 am
I suspect I'll end up seeing the same as you: no crash without a lot of players logged in. I've put my fix in for the next build, anyway, so hopefully once that's up you can tell if the results change and we can go from there.
Apr 4 2015, 12:25 am (Edited on Apr 4 2015, 6:25 am)
I'm currently not behind a computer that has access to my servers. But I'll update the BYOND version soon as I can and post the results here.

Edit: Just to be clear, I'm going to host with the new build. Should I also be compiling with the new build? Does that make a difference?
Apr 4 2015, 7:15 am
Edit: I've compiled with the new version and am hosting with it.

Crashes with the new build happen more rapidly then with the previous version [ game wont run for more than 10 minutes, maybe 15]. The printed information is much more though, so I've thrown it into pastebin.

http://pastebin.com/ALf6BKmc
Apr 4 2015, 9:26 am
I'm thinking a copy of your source would probably help diagnose this. In particular I wonder what's happening in Enlarge.dm and in the calls preceding it. Maybe there's a set of steps I can follow where this will show up in the debugger.

From what I'm seeing, the issue at hand is heap corruption. That's probably also why it crashed in the last version; something's just running a tad bit differently in this one, causing the corruption to show up in a different spot.
Apr 4 2015, 12:51 pm
How do you I send it to you?
Apr 4 2015, 5:22 pm
I also noticed something strange with the new builds, aftar 30 min+ the Game starts to lose FPS and runs very slow only Fix I found it is to close the game and restart the game. There doesnt seem to be any CPU lose or bad procs on the server side this happends on the client the player plays.
Apr 7 2015, 6:26 am
Alright, there's clearly something else going on.
Game was up for almost 24 hours before it ran out of memory. I was logged in and so was one other person, putting the player count at just 2.

I was not doing anything in the game when it crashed and the second person had just saved HTML page that displays available character ranks to players.


But as far as I can tell there aren't any new objects/turfs being generated.
I ran it using 507.1279 as this triggers the crash more reliably than 507.1280.

If you want the source still, just the host files or perhaps access to the server then let me know.
In either case a public key / email would be nice.

Mon Apr  6 19:08:16 2015
World opened on network port 1202.
Welcome BYOND! (5.0 Beta Version 507.1279)

[snip]

The BYOND hub reports that port 1202 is reachable.
BUG: Unexpected hub certificate (65535)
BUG: Unexpected hub certificate (65535)
BUG: Removing corrupt rsc entry 'Black Wizard.dmi'
BUG: Out of memory.
BUG: Crashing due to an illegal operation!

Backtrace for BYOND 507.1279 on Linux:
Generated at Tue Apr  7 12:16:45 2015

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bb74]
libbyond.so [0xf722c000, 0x0], 0x1a122c
 [0xf7739000, 0xf7739410], [0xf7739000, 0xf7739410]
libbyond.so [0xf722c000, 0x0], 0x1a122c
libbyond.so [0xf722c000, 0x0], 0x1ab9d2
libbyond.so [0xf722c000, 0x0], 0x1abb67
libbyond.so [0xf722c000, 0x0], 0x227f3a
libbyond.so [0xf722c000, 0x0], 0x22d6de
libbyond.so [0xf722c000, 0x0], 0x21b493
libbyond.so [0xf722c000, 0x0], 0x22d6de
libbyond.so [0xf722c000, 0x0], 0x21b52c
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df
libbyond.so [0xf722c000, 0x0], 0x2148df

Recent proc calls:
/proc/get_area_turfs
/proc/SET_DOCILE_NPCS
/proc/SET_DOCILE_NPCS
/proc/get_area_turfs
/proc/SET_HOSTILE_NPCS
/proc/SET_HOSTILE_NPCS
/EventScheduler/proc/__shift_down_events
/EventScheduler/proc/__iteration
/EventScheduler/proc/__loop
/EventScheduler/proc/__shift_down_events
/EventScheduler/proc/__iteration
/EventScheduler/proc/__loop
/EventScheduler/proc/__shift_down_events
/EventScheduler/proc/__iteration
/EventScheduler/proc/__loop
/EventScheduler/proc/__shift_down_events

To help the BYOND developers debug this, please send the above trace as part
of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1
Apr 7 2015, 9:03 am
In a recent post I suggested a way to check on what might be using all that memory. My suggestion would be to look into that and see if there are any object leaks. A memory leak is a lot less likely at this point, but an object leak in your code is not out of the realm of possibility.
Apr 8 2015, 10:05 pm
I agree. It's most likely something silly or I don't quite understand being used in the code that's causing this.
Part of me was hoping you could help narrow it down :p

I was trying to figure out something that would give me a list of unique objects/turfs/mobs/whatever. Not sure if that'll provide me with enough information.

I'll be sure to make use of your suggestion and post my findings here.
Page: 1 2