ID:1764499
 
BYOND Version:507.1273
Operating System:Ubuntu 14.04 64-bit
Web Browser:Chrome 39.0.2171.99
Applies to:Dream Daemon
Status: Open

Issue hasn't been assigned a status value.
Descriptive Problem Summary:
Since updating my server to 507 a couple weeks back, there's been a few crashes which didn't ever occur in 498.

Log has:
The BYOND hub reports that port 8000 is reachable.
BUG: Finished erasure with refcount=1 (ref=3:642) DM (Login.dm:3074)
BUG: Doubly premature return value!
BUG: Doubly premature return value!
BUG: Failed to decode message 54,5
BUG: Network connection for Lord_of_Chaos24 shutting down due to read error. (2,1)
BUG: Unexpected message 26,53
BUG: Finished erasure with refcount=1 (ref=3:629) DM (Login.dm:3069)
BUG: Finished erasure with refcount=1 (ref=3:631) DM (Login.dm:3069)
BUG: Finished erasure with refcount=1 (ref=3:636) DM (Login.dm:3069)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:637) DM (Login.dm:3069)
BUG: Finished erasure with refcount=1 (ref=3:648) DM (Login.dm:3069)
BUG: Finished erasure with refcount=1 (ref=23:0) DM (Login.dm:279)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:640) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:643) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:635) DM (Login.dm:2993)
BUG: Finished erasure with refcount=18 (ref=3:640) DM (Login.dm:2993)
BUG: Finished erasure with refcount=3 (ref=2:58823) DM (Login.dm:2993)
*** Error in `DreamDaemon': double free or corruption (!prev): 0x0a595778 ***

The BYOND hub reports that port 8000 is reachable.
BUG: Unexpected hub certificate (65535)
BUG: Crc mismatch in ReadCacheItem
BUG: Crc mismatch in ReadCacheItem
BUG: Doubly premature return value!
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=2 (ref=3:639) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:687) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:643) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:620) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:631) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:649) DM (Login.dm:2993)
BUG: Finished erasure with refcount=3 (ref=3:637) DM (Login.dm:2993)
BUG: Unexpected hub certificate (65535)
BUG: Finished erasure with refcount=4 (ref=3:651) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:650) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:635) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:333) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:697) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:646) DM (Login.dm:2993)
BUG: Unexpected hub certificate (65535)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:632) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:621) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:621) DM (Login.dm:2993)
BUG: Finished erasure with refcount=3 (ref=3:652) DM (Login.dm:2993)
BUG: Finished erasure with refcount=10 (ref=3:621) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:714) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:657) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:641) DM (Login.dm:2993)
BUG: Doubly premature return value!
*** Error in `DreamDaemon': double free or corruption (out): 0x0a9b0300 ***

The ..() is line 2993 of Login.dm:
mob/Player/Del()
Players.Remove(src)
..()


Trace 1 and Trace 2
Neither of those shows me the stack trace from the crash. I need that to diagnose the issue. Additionally I need to know which 507 you're using.
It says the version of 507.1273 in the issue header.

How do I get you the stack trace from the crash? It's not freezing so I can't attach gdb like in the past.
Trace 1 and Track 2 are what ~/.byond/cfg/trace.txt contained after each crash.

It just crashed again:

The BYOND hub reports that port 8000 is reachable.
BUG: Doubly premature return value!

Linux DD should be printing out the stack trace every time it crashes. The backtrace is printed out to stderr.
I'm using -logself.
I'm not getting any backtrace. It's not even printing any kind of error now...
Wut do:
~~~
World opened on network port 8000.
Welcome BYOND! (5.0 Beta Version 507.1273)

Or this code can be embedded:
IFRAME: http://www.byond.com/play/embed/74.91.122.249:8000

The BYOND hub reports that port 8000 is reachable.
~~~
- Then the server died. Shit wasn't saved, so it wasn't a clean shutdown.

Edit: Nothing related in kern.log or syslog either.
The -logself option will not show the backtrace. That gets printed out to stderr only.
-logself (send stderr to world.log)

The log in the original post is from the file I've set world.log to. It's previously shown backtraces for crashes, it's just not doing it this time.
The doubly premature return value appearing frequently in the logs is a little troubling. That happens when a response to a waiting proc comes in (that could be an input(), world.Export(), medals, or any number of other things) when the background proc has been deleted. It's not the kind of thing that should really come up much; things like prompts for instance get canceled properly. I know the background proc will be erased if its src is erased, which suggests something that's calling one of those procs, is being deleted prematurely, while it's still waiting.

All that being said, I don't see why that'd result in the refcount errors. It might help to have more insight on what's being deleted and which background proc is trying to return. I will say that most of the items with the refcount errors are mobs (type 2), but a couple appear to be objs (type 3), and in one case I'm seeing a mob.contents list (23) which is very odd.

Regarding the crash, have you considered attaching gdb before it crashes and just letting it play out? Maybe if it crashes while running in gdb it'll give you some useful info.
Running under gdb didn't appear too successful...


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Wed Jan 21 21:48:16 2015
Auto-safety mode: safe (working directory access)
World opened on network port 8000.
Welcome BYOND! (5.0 Beta Version 507.1273)
[New Thread 0xf6eeeb40 (LWP 12114)]
[Thread 0xf6eeeb40 (LWP 12114) exited]

Or this code can be embedded:
IFRAME: http://www.byond.com/play/embed/74.91.122.249:8000

The BYOND hub reports that port 8000 is reachable.
BUG: Finished erasure with refcount=4 (ref=3:618) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:617) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:628) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:632) DM (Login.dm:2993)
BUG: Unexpected hub certificate (65535)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:634) DM (Login.dm:2993)

Program received signal SIGPIPE, Broken pipe.
0xf7fdb430 in __kernel_vsyscall ()
(gdb) bt
#0 0xf7fdb430 in __kernel_vsyscall ()
#1 0xf7752d9b in write () from /lib/i386-linux-gnu/libpthread.so.0
#2 0xf7d993a0 in SocketLib::WriteSocket(long, void const*, long, char*) ()
from /usr/local/lib/libbyond.so
#3 0xf7d95c78 in ?? () from /usr/local/lib/libbyond.so
#4 0xf7d95e6c in ?? () from /usr/local/lib/libbyond.so
#5 0xf7d0d025 in ?? () from /usr/local/lib/libbyond.so
#6 0xf7d10ff7 in ?? () from /usr/local/lib/libbyond.so
#7 0xf7d11739 in ?? () from /usr/local/lib/libbyond.so
#8 0xf7d987b5 in ?? () from /usr/local/lib/libbyond.so
#9 0xf7d9b06d in SocketLib::WaitForSocketIO(long, unsigned char) ()
from /usr/local/lib/libbyond.so
#10 0x0804af24 in ?? ()
#11 0xf75aea83 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6
#12 0x0804a7b1 in ?? ()




~~~~

The BYOND hub reports that port 8000 is reachable.
BUG: Finished erasure with refcount=1 (ref=3:632) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:658) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:620) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:631) DM (Login.dm:2993)
BUG: Finished erasure with refcount=4 (ref=3:645) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:634) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=2 (ref=3:641) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:636) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:649) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:640) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:642) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=2:61463) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:620) DM (Login.dm:2993)
BUG: Finished erasure with refcount=4 (ref=3:620) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=2:58132) DM (Spells.dm:696)
BUG: Bad ref (2:58132) in DecRefCount
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:631) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:634) DM (Login.dm:2993)
BUG: Finished erasure with refcount=3 (ref=3:619) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:641) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=6 (ref=3:657) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=2 (ref=3:673) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:630) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:673) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:643) DM (Login.dm:2993)
BUG: Finished erasure with refcount=3 (ref=3:630) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:653) DM (Login.dm:2993)
BUG: Finished erasure with refcount=10 (ref=3:657) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:623) DM (Login.dm:2993)
BUG: Doubly premature return value!
BUG: Finished erasure with refcount=1 (ref=3:651) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:692) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:622) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:633) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:649) DM (Login.dm:2993)
BUG: Finished erasure with refcount=2 (ref=3:671) DM (Login.dm:2993)
*** Error in `DreamDaemon': double free or corruption (!prev): 0x0939ead0 ***


-core didn't produce any core that I could find...
Core file sent to you.

New error in a crash this time! :)

The BYOND hub reports that port 8000 is reachable.
BUG: Finished erasure with refcount=1 (ref=3:643) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:630) DM (Login.dm:2993)
BUG: Finished erasure with refcount=1 (ref=3:626) DM (Login.dm:2993)
BUG: Unexpected certificate (6)
BUG: Finished erasure with refcount=1 (ref=3:646) DM (Login.dm:2993)
BUG: Unexpected hub certificate (65535)
BUG: Doubly premature return value!
*** Error in `DreamDaemon': free(): invalid pointer: 0x0b4c2a9d ***