ID:1292048
 
BYOND Version:499
Operating System:Linux
Web Browser:Firefox 23.0
Applies to:Dream Daemon
Status: Open

Issue hasn't been assigned a status value.
Descriptive Problem Summary:

Numbered Steps to Reproduce Problem: Start up the game, let it run for like five to ten hours averaging around 40-60 players.

Code Snippet (if applicable) to Reproduce Problem:


Expected Results: The server should run perfectly fine provided the CPU stays stable (which would be anything below 90% for us).

Actual Results: Despite the server's connection and CPU not having any issues, DreamDaemon will crash after around a day or so of activity. Sometimes less. It's really hard to predict when the crashes happen.

Does the problem occur:
Every time? Or how often? Around every day.
On other computers? We've tried to host the game on another machine, and it seems to crash consistently, so something in the code itself might be causing the crash. However, DreamDaemon shouldn't be crashing at all.

When does the problem NOT occur? Never.

Did the problem NOT occur in any earlier versions? If so, what was the last version that worked? Not really.

Workarounds: None.

We're going to run the server with -trace and post the logs here once a crash occurs.
Consider the steps here: ID:103269 as far as dumping core.

Also, the issue in that case was due to a huge recursion with world.loop_checks=0, so you might want to see if you have that set that way as well.

DD should never crash, although without loop checking it is possible due to running out of stack. Of course, we could also have some bugs.
We actually do not set world.loop_checks to 0 anywhere in the code, so I doubt that is the issue.

However, are getting some weird error reports on certain savefiles in the game:

BYOND Error:(sfile.cpp,1138) failed to open file:
BYOND Error:(sfile.cpp,1138) /home/dmb/wano0132/saves/player/kihatomay.sav


I suspect it may have something to do with this, but I can't be 100% sure.
In response to Doohl
Hazordhu randomly experiences immense lag and 100% CPU after some time, too, but I expect it's from crappy 5-year-old code. I do use while() for loops, though.

I've seen sfile.cpp errors, too. And a bunch of the recently fixed refcount BUGs.
In response to Tom
Tom wrote:
Consider the steps here: ID:103269 as far as dumping core.

We're already dumping the core, so we can rule this out.

Also, the server is being hosted on 496.1138. We're hesitant to update due to these issues: http://www.byond.com/forum/?post=1291238

If we can get confirmation that they're no longer a problem in the latest version, then we'll go ahead and update.
Well if you are dumping core, that means we can probably debug DD itself. We can't do it if you are on an old build though, so we'll have to wait for the update. There are some steps there that elude me at this time but we'll take a look if you continue to get crashes on the latest.

The issue with the hub connectivity is resolved in 1193 (we were simply pointing users to the beta test hub, which is no longer running-- I had forgotten that affected hosting too). I don't know about those other things since no bug reports with any info were filed, but very little has changed in the server so I doubt it's a problem.

Also, some of the refcount stuff should be fixed in the latest linux builds (actually not fixed on windows yet because we haven't released it).
Okay, thanks, then we'll update. If you want to debug the DD we should be good to go ahead and do that? I'll e-mail you the files if so.
Well, run with the latest and let us know if it crashes and you get a core file.
Okay, sure thing--thanks for the help.
In response to Tom
Tom wrote:
Well, run with the latest and let us know if it crashes and you get a core file.

Am I correct to say the core gets dumped in the same folder as the rsc and dmb?
I'm not sure how Linux works, but I think Hazordhu's hosted on one. It's possible the code is causing random freezes, but as far as I can tell from the log, there aren't any runtime errors or any "suppressing runtime errors" messages. Using the "top" command in PuTTY (I don't completely know what I'm doing, with that), it shows Dream Daemon at 100% CPU and the game is frozen. I'm not sure if that counts as "crashing", since I have to "kill" it manually and start it back up.

Sometimes, after a while, Hazordhu's framerate gets chopped in half and becomes very slow. Sometimes, it just freezes without warning. I wonder if it's an issue with the host.
In response to Kaiochao
Kaiochao wrote:
it shows Dream Daemon at 100% CPU and the game is frozen. I'm not sure if that counts as "crashing", since I have to "kill" it manually and start it back up.

You've described my issue here.
In response to Tom
Tom wrote:
Well, run with the latest and let us know if it crashes and you get a core file.

A crash occurred, despite updating. What is the name of the 'core file'?

Our crashing also seems to be similar to Murrawhip's and Kaiochao's (?): http://www.byond.com/forum/?post=1264881#comment4924862
Well, this is a first. Previously, Dream Daemon would only freeze. This time, it just instantly crashed. I'm not sure what to say about the call stack.
{"
BUG: Crashing due to an illegal operation!
proc name: Del (/mob/Del)
source file: Pulling Carts.dm,12
usr: null
src:
call stack:
Del()
Del()
Del()
Login()
Derio1994 (/client): New()

Backtrace for BYOND 498.1158 on Linux
[old glibc]:
Generated at Sun Jun 16 09:34:57 2013

DreamDaemon
[0x8048000, 0x0], [0x8048000, 0x804a8ce]
libbyond.so 0x283080, 0x283091
[0xd97000, 0xd97410], [0xd97000, 0xd97410]
libbyond.so 0x283080, 0x283091
libbyond.so 0x27f5b0, 0x27f5cc
libbyond.so 0x23c4f0, 0x23c56c
libbyond.so 0x283200, 0x2844a4
libbyond.so 0x2852e0, 0x285ac3
libbyond.so 0x25e300, 0x25e69f
libbyond.so
[0x7d6000, 0x0], 0x2bad21
libbyond.so
[0x7d6000, 0x0], 0x2bda91
libbyond.so
[0x7d6000, 0x0], 0x29ff85
libbyond.so 0x2b42b0, 0x2b43bd
libbyond.so
[0x7d6000, 0x0], 0x2bd9d3
libbyond.so
[0x7d6000, 0x0], 0x29ff85
libbyond.so 0x2b42b0, 0x2b43bd
libbyond.so
[0x7d6000, 0x0], 0x2bd9d3
libbyond.so
[0x7d6000, 0x0], 0x29ff85
libbyond.so 0x2b42b0, 0x2b43bd
libbyond.so 0x2b4410, 0x2b44c2
libbyond.so 0x2b6690, 0x2b675c
libbyond.so 0x239a20, 0x239a9d
libbyond.so
[0x7d6000, 0x0], 0x2aa9ae
libbyond.so 0x2b42b0, 0x2b43bd
libbyond.so 0x2b4410, 0x2b44c2
libbyond.so 0x2b6690, 0x2b675c
libbyond.so 0x24ff60, 0x2503b6
libbyond.so 0x26e0f0, 0x26e649
libbyond.so
[0x7d6000, 0x0], 0x2baacd
libbyond.so
[0x7d6000, 0x0], 0x2bda91
libbyond.so
[0x7d6000, 0x0], 0x29ff85
libbyond.so 0x2b42b0, 0x2b43bd
libbyond.so 0x2b4410, 0x2b44c2
libbyond.so 0x2b6690, 0x2b675c
libbyond.so
[0x7d6000, 0x0], 0x267a9f
libbyond.so
[0x7d6000, 0x0], 0x273695
libbyond.so
[0x7d6000, 0x0], 0x276eb4
libbyond.so 0x277f00, 0x277f7f
libbyond.so 0x2c4ec0, 0x2c4f59
libbyond.so 0x314910, 0x314bbd
libbyond.so
[0x7d6000, 0x0], 0x3147c8
libbyond.so 0x3175a0, 0x317c7f
DreamDaemon
[0x8048000, 0x0], [0x8048000, 0x804a3ee]
libc.so.6 0x16c00, 0x16ce6 (__libc_start_main)
DreamDaemon
[0x8048000, 0x8049b04], [0x8048000, 0x8049ed1]
"}

The line it points to is just ..() in mob/Del().

The core file is 74,628 KiB. Should I send that?

No one's really enjoying the rollbacks caused by these extremely random and unpredictable crashes.
Hold onto those files and we'll get you help early this week.
Okay, Tom. Let me know what to do when you find the time.