I have two problem signatures for Dream Daemon. This is a problem that's been plaguing the newly released Project Crimson Roleplay for two weeks now. We've gone through extensive efforts to fix the problem on our side, from changing the way our game operates, tons of bug fixes, runtime fixes, running DreamDaemon verbose to find suppressed runtimes to handle. We've changed servers, changed operating systems, done virtually everything and we can to solve this problem of crippling 'lag.'
So to be clear with what we're dealing with, after a couple hours, or even a few hours of running, DreamDaemon begins to take up a massive amount of resources. It maxes out an entire core and runs itself into the dirt-- locking up and eventually crashing. We've checked to ensure nothing is spawning and going out of control, we've checked for infinite loops, we've optimized constantly, and nothing seems to solve it. The game makes extensive use of animations, and we suspect that might have something to do with it, but we're almost entirely certain it's a red herring.
Our debugging efforts have tracked an in-game CPU that rises to 500+, 1500+, eventually 8000+ before DreamDaemon locks up and comes to a complete freeze, then crashes. This happens at random seemingly, without any kind of discernable trigger, at different populations of players, and regardless of the host machine's capacity (We suspected we didn't have enough RAM at first and upgraded servers and RAM and it doesn't matter how much is available, DD kills itself).
Problem signature:
Problem Event Name: APPCRASH
Application Name: dreamdaemon.exe
Application Version: 5.0.512.1470
Application Timestamp: 5ceae321
Fault Module Name: byondcore.dll
Fault Module Version: 5.0.512.1470
Fault Module Timestamp: 5ceae2aa
Exception Code: c0000005
Exception Offset: 000f69d2
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 1033
Additional Information 1: 54e2
Additional Information 2: 54e23fb558bc27b5426ec592345ce16d
Additional Information 3: aa59
Additional Information 4: aa592dcc388ea38f2a413041580d6077
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=280262
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt
===========================================================
Problem signature:
Problem Event Name: APPCRASH
Application Name: dreamdaemon.exe
Application Version: 5.0.512.1470
Application Timestamp: 5ceae321
Fault Module Name: byondcore.dll
Fault Module Version: 5.0.512.1470
Fault Module Timestamp: 5ceae2aa
Exception Code: c0000005
Exception Offset: 000f69d2
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 1033
Additional Information 1: 54e2
Additional Information 2: 54e23fb558bc27b5426ec592345ce16d
Additional Information 3: aa59
Additional Information 4: aa592dcc388ea38f2a413041580d6077
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=280262
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt
=======================================================
Also have this:
runtime error: Cannot execute null.().
runtime error: Cannot execute null.().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.3991033912().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.3991033912().
runtime error: Cannot execute null.3991033912().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.3991033912().#
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
runtime error: Cannot execute null.71.223.38.137().
Numbered Steps to Reproduce Problem:
1. Host server.
2. 50 + Players online (2 hour runtime)
3. Let DreamDaemon explode on its own.
Code Snippet (if applicable) to Reproduce Problem:
N/A
Expected Results:
Actual Results:
Does the problem occur:
Every time? Or how often?
Every last time.
In other games?
Not that I'm aware. I think Baystation12 encountered something similar and posted about it, but their post wasn't helpful for us.
In other user accounts?
N/A
On other computers?
On other host machines, yes.
When does the problem NOT occur?
Always occurs
Did the problem NOT occur in any earlier versions? If so, what was the last version that worked? (Visit http://www.byond.com/download/build to download old versions for testing.)
We started with 5.10 I believe, and updated BYOND to no avail.
Workarounds:
Nothing.
The big question here of course is, what's using all that memory? Some memory reports from DD would be helpful. In Windows there's a menu command for this; in Linux it's a signal you send to the server. With that info, it would be possible to start discovering what kinds of objects are using the most memory and, ultimately, why. The best thing to do is to collect these reports over time, so you can track if the info is remaining fairly stable or if something is tracking upwards at an increasing rate--the latter being a sign that maybe your game has some references it should be, but isn't, throwing away. (And, there's no reason a BYOND bug couldn't be ruled out, like if some proc weren't decrementing reference counts that it's supposed to.)