DreamDaemon Crash - linux

ID:1337998
 
Jul 28 2013, 7:19 pm
BYOND Version:499.1201
Operating System:Ubuntu 13.04 64-bit
Web Browser:Chrome 29.0.1547.32
Applies to:Dream Daemon
Status: Open
Issue hasn't been assigned a status value.
Descriptive Problem Summary:
I just updated from 499.1197.
Crashes within an hour it seems. Isn't a freeze like my other report.

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a8fe]
libbyond.so [0xf720b000, 0x0], 0x24d22b
 [0xf7783000, 0xf7783410], [0xf7783000, 0xf7783410]
libbyond.so [0xf720b000, 0x0], 0x24d22b
libbyond.so [0xf720b000, 0x0], 0x24f4dd
libbyond.so [0xf720b000, 0x0], 0x253e68
libbyond.so [0xf720b000, 0x0], 0x22d99a
libbyond.so [0xf720b000, 0x0], 0x237c7b
libbyond.so [0xf720b000, 0x0], 0x238fe6
libbyond.so [0xf720b000, 0x0], 0x237ede
libbyond.so 0x2387e0, 0x23889b
libbyond.so [0xf720b000, 0x0], 0x2cc90d
libbyond.so [0xf720b000, 0x0], 0x2cf981
libbyond.so [0xf720b000, 0x0], 0x2b2072
libbyond.so 0x2c62b0, 0x2c63bb
libbyond.so [0xf720b000, 0x0], 0x2cf8c2
libbyond.so [0xf720b000, 0x0], 0x2b2072
libbyond.so 0x2c62b0, 0x2c63bb
libbyond.so 0x2c6430, 0x2c64e3
libbyond.so 0x2c8680, 0x2c874c
libbyond.so [0xf720b000, 0x0], 0x2382b6
libbyond.so 0x2387e0, 0x23889b
libbyond.so [0xf720b000, 0x0], 0x2c0840
libbyond.so 0x2c62b0, 0x2c63bb
libbyond.so 0x2c6430, 0x2c64e3
libbyond.so [0xf720b000, 0x0], 0x2c79a2
libbyond.so [0xf720b000, 0x0], 0x2b216c
libbyond.so 0x2c62b0, 0x2c63bb
libbyond.so 0x2c6430, 0x2c64e3
libbyond.so [0xf720b000, 0x0], 0x2c79a2
libbyond.so [0xf720b000, 0x0], 0x2b216c
libbyond.so 0x2c62b0, 0x2c63bb
libbyond.so 0x2c6430, 0x2c64e3
libbyond.so 0x2c8680, 0x2c874c
libbyond.so [0xf720b000, 0x0], 0x2cdcf0
libbyond.so [0xf720b000, 0x0], 0x2cf981
libbyond.so [0xf720b000, 0x0], 0x2b2072
libbyond.so 0x2c62b0, 0x2c63bb
libbyond.so [0xf720b000, 0x0], 0x2cf8c2
libbyond.so [0xf720b000, 0x0], 0x2b2072
libbyond.so 0x2c62b0, 0x2c63bb
libbyond.so 0x2c6430, 0x2c64e3
libbyond.so 0x2c8680, 0x2c874c
libbyond.so [0xf720b000, 0x0], 0x273b0e
libbyond.so [0xf720b000, 0x0], 0x289601
libbyond.so 0x28a370, 0x28a3ef
libbyond.so 0x2d6db0, 0x2d6e5c
libbyond.so 0x329f00, 0x32a1d6
libbyond.so [0xf720b000, 0x0], 0x329db8
libbyond.so 0x32ca50, 0x32d12f


Did the problem NOT occur in any earlier versions? If so, what was the last version that worked?
The crash occurred twice in 499.1201. I'm now running in 499.1200 to see if it continues.
Jul 29 2013, 9:58 am
for what its worth. Heres a valgrind log that resulted in the same final crash address.

http://sebsauvage.net/ paste/?3916275c00da36bd#4l09b9SzmFXCTYn12cncD8HCiak5ZVgRttoc yZM2+Mc
Jul 29 2013, 10:18 am
Thank you for the information. I'm going to see what I can find on this. I may ask you to conduct a few more tests.
Jul 29 2013, 12:18 pm
well, that valgrind log took nearly 19hours runtime to reach the crash (server was running somewhere around 250times slower and is completly unusable to play on).
Jul 29 2013, 12:21 pm
No worries, the tests I'm thinking about would just be on the live server, sending signals and giving output when put into a hung state. But I'll see if I can get anything from the provided logs. Thanks for the patience. We want to resolve crashes as much as you do.
Jul 29 2013, 1:12 pm
i already provided those data in the other post Tom. when it freeze, its always on a string operation. But just about never on the same line in the code.
Jul 31 2013, 12:40 pm
heres some extra data
#0 0xb7743430 in __kernel_vsyscall ()
#1 0xb6c4f681 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0xb6c52ab2 in *__GI_abort () at abort.c:92
#3 0xb6c9336b in malloc_printerr (action=2, str=0x6
, ptr=0x16417bd0) at malloc.c:6271
#4 0xb6c933ec in top_check () at hooks.c:222
#5 0xb6c96a54 in malloc_check (sz=6, caller=0x0) at hooks.c:264
#6 0xb6c991c7 in realloc_check (oldmem=0x0, bytes=6, caller=0xb7406bdc) at hooks.c:331
#7 0xb6c98f43 in *__GI___libc_realloc (oldmem=0x0, bytes=6) at malloc.c:3757
#8 0xb7406bdc in VSprintfRealloc(char**, int, int*, char const*, char*) () from /usr/local/lib/libbyond.so
#9 0xb741769f in StringEditor::AddVPrintf(char const*, char*) () from /usr/local/lib/libbyond.so
#10 0xb72fc1d5 in ServerPrintf(char const*, ...) () from /usr/local/lib/libbyond.so
#11 0xb7367667 in PrintProcDiagnostics(int) () from /usr/local/lib/libbyond.so


using export MALLOC_CHECK_=2 to add extra memory validations.

Aug 7 2013, 11:20 am
I mentioned this on the other thread, but for completeness' sake: 499.1202 is available now, and it addresses the issues the valgrind log discovered that could have led to heap corruption. There could well be other problems lurking, but the ones the 499.1201 valgrind log detected have been dealt with.
Aug 7 2013, 12:30 pm
This is pretty confusing...

So I make use of your swapmaps library. Just now, in 499.1202, it crashed with:
Warning: type read from file (/BankClass) is not defined
runtime error: cannot append to list
proc name: Read (/atom/Read)
  source file: SwapMaps.dm,480
  usr: Molo (/mob/Player)
  src: Molo (/mob/Player)
  call stack:
Molo (/mob/Player): Read(players/m/molo48.sav (/savefile))
Molo (/mob/Player): Read(players/m/molo48.sav (/savefile))
Molo48 (/client): base LoadMob("Molo")
Molo48 (/mob/BaseCamp/ChoosingCharacter): Choose Character()
Molo48 (/mob/BaseCamp/ChoosingCharacter): Topic("src=\[0x3000223];action=loginL...", /list (/list))
Molo48 (/client): Topic("src=\[0x3000223];action=loginL...", /list (/list), Molo48 (/mob/BaseCamp/ChoosingCharacter))
Molo48 (/client): Topic("src=\[0x3000223];action=loginL...", /list (/list), Molo48 (/mob/BaseCamp/ChoosingCharacter))
proc name: Write (/atom/Write)
  source file: SwapMaps.dm,458
  usr: Molo (/mob/Player)
  src: Molo (/mob/Player)
  call stack:
Molo (/mob/Player): Write(players/m/molo48.sav (/savefile))
Molo (/mob/Player): Write(players/m/molo48.sav (/savefile))
Molo48 (/client): base SaveMob()
Molo (/mob/Player): .Save()
Molo (/mob/Player): Say("save me")


World.log:
Welcome BYOND! (4.0 Public Version 499.1202)
The BYOND hub reports that port 8000 is reachable.
BUG: Missing savefile directory .0 (cd=/players/lavitiz/mobs/lavitiz/mob)
BUG: Missing savefile directory .2 (cd=/players/molo48/mobs/molo/mob/.0/Lwearing)
BUG: Missing savefile directory .2 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .3 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .4 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .5 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .10 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .11 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .12 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .13 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .14 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .15 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .16 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .17 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .18 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .19 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .20 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .21 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .22 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .23 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .24 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .25 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .26 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .27 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .28 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .29 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .30 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .31 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .33 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .34 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .35 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .36 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .37 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .38 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .39 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .40 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .41 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .42 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .43 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .44 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .45 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .46 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .47 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .48 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .49 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .50 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .51 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .52 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory .53 (cd=/players/molo48/mobs/molo/mob/.0/pmsRec)
BUG: Missing savefile directory ../Lwearing/.2 (cd=/players/molo48/mobs/molo/mob/.0/contents)
BUG: Finished erasure with refcount=1 (ref=23:0) DM (Cronicles.dm:627)
BUG: Crashing due to an illegal operation!

Backtrace for BYOND 499.1202 on Linux [old glibc]:
Generated at Wed Aug  7 17:04:58 2013

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a8fe]
libbyond.so [0xf721a000, 0x0], 0x24d252
 [0xf7792000, 0xf7792410], [0xf7792000, 0xf7792410]
libbyond.so [0xf721a000, 0x0], 0x24d252
libbyond.so [0xf721a000, 0x0], 0x24d31c
libbyond.so [0xf721a000, 0x0], 0x24f56d
libbyond.so [0xf721a000, 0x0], 0x253ef8
libbyond.so [0xf721a000, 0x0], 0x230adc
libbyond.so [0xf721a000, 0x0], 0x2311f2
libbyond.so [0xf721a000, 0x0], 0x239ef2
libbyond.so [0xf721a000, 0x0], 0x2311d9
libbyond.so 0x2331e0, 0x233449
libbyond.so [0xf721a000, 0x0], 0x2c9197
libbyond.so [0xf721a000, 0x0], 0x2b47ff
libbyond.so 0x2c64c0, 0x2c65cb
libbyond.so [0xf721a000, 0x0], 0x2cfad2
libbyond.so [0xf721a000, 0x0], 0x2b2282
libbyond.so 0x2c64c0, 0x2c65cb
libbyond.so 0x2c6640, 0x2c66f3
libbyond.so 0x2c8890, 0x2c895c
libbyond.so [0xf721a000, 0x0], 0x23360c
libbyond.so [0xf721a000, 0x0], 0x231338
libbyond.so 0x2331e0, 0x233449
libbyond.so [0xf721a000, 0x0], 0x2c9197
libbyond.so [0xf721a000, 0x0], 0x2b47ff
libbyond.so 0x2c64c0, 0x2c65cb
libbyond.so 0x2c6640, 0x2c66f3
libbyond.so [0xf721a000, 0x0], 0x2c7bb2
libbyond.so [0xf721a000, 0x0], 0x2b237c
libbyond.so 0x2c64c0, 0x2c65cb
libbyond.so 0x2c6640, 0x2c66f3
libbyond.so [0xf721a000, 0x0], 0x2c7b06
libbyond.so [0xf721a000, 0x0], 0x2b237c
libbyond.so 0x2c64c0, 0x2c65cb
libbyond.so 0x2c6640, 0x2c66f3
libbyond.so 0x2c8890, 0x2c895c
libbyond.so 0x274b00, 0x275365
libbyond.so [0xf721a000, 0x0], 0x27617a
libbyond.so [0xf721a000, 0x0], 0x2894fa
libbyond.so 0x28a580, 0x28a5ff
libbyond.so 0x2d6fc0, 0x2d706c
libbyond.so 0x32a0f0, 0x32a3c6
libbyond.so [0xf721a000, 0x0], 0x329fa8
libbyond.so 0x32cc40, 0x32d31f
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a41e]
libc.so.6 0x19840, 0x19935 (__libc_start_main)
DreamDaemon [0x8048000, 0x8049b40], [0x8048000, 0x8049f01]


Note that /BankClass is defined - this is the first time I've seen error text give that warning for something that does actually exist.

The crash mangled the guy's savefile and my backup doesn't seem to cause the issue unfortunately.

I'm going to just assume this is my error and try a few things.
Aug 8 2013, 12:52 pm
It just crashed again on 1202, during a sleep it seems.

proc name: unreadmessagelooper (/mob/Player/proc/unreadmessagelooper)
  source file: PM.dm,116
  usr: null
  src: 
  call stack:
unreadmessagelooper()


while(1)
            sleep(2650)//Crashed on this line
            spawn()sql_update_ckey_in_table(src)
            unreadmsgs = 0


DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a8fe]
libbyond.so [0xf7170000, 0x0], 0x24d2d3
 [0xf76e8000, 0xf76e8410], [0xf76e8000, 0xf76e8410]
libbyond.so [0xf7170000, 0x0], 0x24d2d3
libbyond.so [0xf7170000, 0x0], 0x24f56d
libbyond.so 0x253980, 0x253b14
libbyond.so 0x2705d0, 0x270787
libbyond.so [0xf7170000, 0x0], 0x2cce79
libbyond.so 0x2c6640, 0x2c67ed
libbyond.so 0x2c8890, 0x2c895c
libbyond.so [0xf7170000, 0x0], 0x253289
libbyond.so 0x2620b0, 0x2620f5
libbyond.so [0xf7170000, 0x0], 0x29398e
libbyond.so [0xf7170000, 0x0], 0x2aa44c
libbyond.so [0xf7170000, 0x0], 0x2c51a9
libbyond.so [0xf7170000, 0x0], 0x2c527b
libbyond.so 0x35ba10, 0x35bb87
libbyond.so 0x32cc40, 0x32ceaa
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a41e]
libc.so.6 0x19840, 0x19935 (__libc_start_main)
DreamDaemon [0x8048000, 0x8049b40], [0x8048000, 0x8049f01]


These crashes didn't happen in 1200, I believe. So I've gone back to it to confirm
Aug 8 2013, 1:36 pm (Edited on Aug 8 2013, 1:47 pm)
I can say with pretty high certainty that nothing will crash in 1202 that wouldn't have in 1201. I forget the specific changes in 1201 but I don't think they were likely to be any different either. This is one of the problems with intermittent bugs. I'll try tracing this though and see if it brings up any new info.

[edit]
It actually looks like this and the trace from your first post are the same crash. I believe this is where the string tree is bring rebalanced. It's basically identical to the freeze in that both problems are being caused by (undoubtedly) the same heap corruption.

After talking with Jey earlier I decided to throw something into the next build that will do some sanity checking in the event of a freeze or a weird string deletion, which will hopefully help us move closer to finding the actual cause. Still, if there's any way at all you can get valgrind to run some tests, that's going to be the best way to discover corruption.
Aug 16 2013, 8:20 pm
MLAAS is also crashing pretty consistently on 1202 with the following being spit into the log file:

http://ncomproductions.com/spies.log

The runtime error is pointing to this proc:

            set_id()
                var
                    the_time = world.realtime
                    tmsp = time2text(the_time,"YYmmDDhhmmss")
                    unique = rand(10000,99999)
                id = "[tmsp].[unique]" // This is the line pointed at.


The line being pointed at is a string being set, so I think it's something to do with the string corruption stuff. The proc is called when new objects are created that don't already have an id via being loaded or whatever. I could probably test back a ways with MLAAS to see if I can find a version without the crash, as MLAAS' code hasn't been updated in a long while (except tonight, but it was after the crashes already started happening).

I'll see about using valgrind to get some more details if you need 'em.
Aug 31 2013, 4:37 am
I'm getting a very similar set of crashes when hosting Andromeda / DarkeDungeon with build 1202.

Both games will crash after a couple of minutes of running or when a player logs in and starts moving around.

DarkeDungeon sometimes doesn't report a crash log at all and just errors out with "Bug: crashing due to illegal operation!"


DarkeDungeon:

libbyond.so [0xb71b9000, 0x0], 0x2c9197
libbyond.so [0xb71b9000, 0x0], 0x2b47ff
libbyond.so 0x2c64c0, 0x2c65cb
libbyond.so 0x2c6640, 0x2c66f3
libbyond.so [0xb71b9000, 0x0], 0x2c7bb2
libbyond.so [0xb71b9000, 0x0], 0x2b237c
libbyond.so 0x2c64c0, 0x2c65cb
libbyond.so 0x2c6640, 0x2c66f3
libbyond.so [0xb71b9000, 0x0], 0x2c7a4c
libbyond.so [0xb71b9000, 0x0], 0x2b237c
libbyond.so [0xb71b9000, 0x0], 0x2c51a9
libbyond.so 0x2c5560, 0x2c5665
libbyond.so [0xb71b9000, 0x0], 0x28c652
libbyond.so 0x35ba10, 0x35bb87
libbyond.so 0x32cc40, 0x32ceaa
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a41e]
libc.so.6 0x16c00, 0x16ce6 (__libc_start_main)
DreamDaemon [0x8048000, 0x8049b40], [0x8048000, 0x8049f01]


Andromeda:

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a8fe]
libbyond.so [0xb7264000, 0x0], 0x24d288
[0xb77d6000, 0xb77d6600], [0xb77d6000, 0xb77d6600]
libbyond.so [0xb7264000, 0x0], 0x24d288
libbyond.so [0xb7264000, 0x0], 0x24f56d
libbyond.so 0x253980, 0x253b14
libbyond.so 0x262c10, 0x262dcf
libbyond.so 0x2603d0, 0x260400
libbyond.so 0x2e67e0, 0x2e6838
libbyond.so [0xb7264000, 0x0], 0x28a306
libbyond.so 0x28a580, 0x28a5ff
libbyond.so 0x2d6fc0, 0x2d706c
libbyond.so 0x32a0f0, 0x32a3c6
libbyond.so [0xb7264000, 0x0], 0x329fa8
libbyond.so 0x32cc40, 0x32d31f
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a41e]
libc.so.6 0x16c00, 0x16ce6 (__libc_start_main)
DreamDaemon [0x8048000, 0x8049b40], [0x8048000, 0x8049f01]

Aug 31 2013, 6:05 am
Since the op mentioned it, I decided to go try 499.1197 and so far everything is perfectly peachy. No crashes. Got to be something in the recent builds.
Aug 31 2013, 7:58 am
In response to Drakemoore