ID:1866730
 
Resolved
A longtime crash that was manifesting in icon destructors was found. This was caused by the client garbage collector firing while the client was trying to resolve a client-side icon operation, when the garbage collector deleted one of the component icons prematurely.
BYOND Version:507
Operating System:Windows 7 Ultimate 64-bit
Web Browser:Firefox 38.0
Applies to:Dream Seeker
Status: Resolved (513.1499)

This issue has been resolved.
Descriptive Problem Summary:
Since the implementation of a new lighting system, clients have been crashing regularly, usually when jumping from area to area (rather than scrolling across the screen). Some people crash more regularly, some less.

Here are a few windows crash logs of the issue:

Faulting application name: dreamseeker.exe, version: 5.0.507.1286, time stamp: 0x55694a6e
Faulting module name: byondcore.dll, version: 5.0.507.1286, time stamp: 0x55694996
Exception code: 0xc0000005
Fault offset: 0x00025747
Faulting process id: 0x34b4
Faulting application start time: 0x01d09f2d88441f18
Faulting application path: C:\Games\byond\bin\dreamseeker.exe
Faulting module path: C:\Games\byond\bin\byondcore.dll
Report Id: 1fc61b3a-0b27-11e5-96f3-005056c00008

and

Faulting application name: dreamseeker.exe, version: 5.0.507.1284, time stamp: 0x5547ec45
Faulting module name: byondcore.dll, version: 5.0.507.1284, time stamp: 0x5547eb51
Exception code: 0xc0000005
Fault offset: 0x000258f7
Faulting process id: 0xe7ac
Faulting application start time: 0x01d09fd7363e08ba
Faulting application path: Z:\Games\BYOND\bin\dreamseeker.exe
Faulting module path: Z:\Games\BYOND\bin\byondcore.dll
Report Id: c253f5bf-0bca-11e5-8472-6cf049e2e355
The lighting system in question uses overlays that are a subtype of /atom/movable, one for each turf. The overlays use a white turf-sized icon and multiply blending to recolor their turf and its contents. The color of the overlay is changed with an animate call whenever the lighting of a tile needs to change.

I'm not sure what the exact reason for using /atom/movable was, but I would guess it was simply easier than making a new /obj and making sure everything else ignored it.

The crashing occurs unpredictably, with some people crashing far more frequently than others, and most crashes occurring when trying to load an entire new screen of turfs. It seems like the frequency of crashing may be related to latency to the server.
I'll check out those details soon, and hopefully can get that patched into 508.1287 if I can find the cause.
I just experienced a crash on the same server that Zomgponies and Krausus are on, here's the info from mine:

Problem signature:
Problem Event Name: APPCRASH
Application Name: dreamseeker.exe
Application Version: 5.0.507.1286
Application Timestamp: 55694a6e
Fault Module Name: byondcore.dll
Fault Module Version: 5.0.507.1286
Fault Module Timestamp: 55694996
Exception Code: c0000005
Exception Offset: 0005b1e5
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1033
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
The second crash is in icon manipulation routines; specifically, it's in the routine that clears the icon, so this is probably happening as an icon is being destroyed. Still looking into the first case. I suspect they're related.

[edit]
The first crash appears to be in the animation message handler. I suspected animation was involved from the beginning so that makes sense. Looking into that further.
Animation crashes are probably the most important ones as those would relate directly to our new lighting system. Either way, both should return runtimes at worst, client crashes aren't really debuggable from our end.
It's impossible for a client-side issue to return a runtime error.
I experienced a crash similar to what ZomgPonies and Krausus have described, it is also on the same server as both. The Windows crash log is below.

Faulting application name: dreamseeker.exe, version: 5.0.507.1286, time stamp: 0x55694a6e
Faulting module name: byondcore.dll, version: 5.0.507.1286, time stamp: 0x55694996
Exception code: 0xc0000005
Fault offset: 0x00025747
Faulting process id: 0x103c
Faulting application start time: 0x01d0a17b72c2a0f1
Faulting application path: C:\Program Files (x86)\BYOND\bin\dreamseeker.exe
Faulting module path: C:\Program Files (x86)\BYOND\bin\byondcore.dll
Report Id: 90d217d3-0d6f-11e5-a5ca-c86000b25dbb
Crashes used to be quite frequent but less so now but still an annoyance

Faulting application name: dreamseeker.exe, version: 5.0.507.1286, time stamp: 0x55694a6e
Faulting module name: byondcore.dll, version: 5.0.507.1286, time stamp: 0x55694996
Exception code: 0xc0000005
Fault offset: 0x0005af89
Faulting process ID: 0x63e4
Faulting application start time: 0x01d0a949c592a3ee
Faulting application path: C:\Program Files (x86)\BYOND\bin\dreamseeker.exe
Faulting module path: C:\Program Files (x86)\BYOND\bin\byondcore.dll
Report ID: 4b155d69-1540-11e5-829b-645106a3060b
Faulting package full name:
Generally when moving fast (as a ghost, or if on meth), or teleporting around.



Faulting application name: dreamseeker.exe, version: 5.0.507.1284, time stamp: 0x5547ec45
Faulting module name: byondcore.dll, version: 5.0.507.1284, time stamp: 0x5547eb51
Exception code: 0xc0000005
Fault offset: 0x000258f7
Faulting process id: 0x1c08
Faulting application start time: 0x01d0a9ddf430b82a
Faulting application path: C:\Program Files (x86)\byond\bin\dreamseeker.exe
Faulting module path: C:\Program Files (x86)\byond\bin\byondcore.dll
Report Id: 479771de-15d2-11e5-88cd-bcaec56b6a3b

There are sanity checks in 508 against the animation issue, so I would suggest trying this in 508.1289. If new crash reports appear, please post them here.
Crashes appeared to happen less often with byond 508.1289, but they appear to be happening more often again on 508.1290.
Same server as all the others (Paradise).

This report happened as soon as I focused another window on top of dream seeker, so anything that may have prompted it was missed:
Faulting application name: dreamseeker.exe, version: 5.0.508.1290, time stamp: 0x5588c0b5
Faulting module name: byondcore.dll, version: 5.0.508.1290, time stamp: 0x55888cc5
Exception code: 0xc0000005
Fault offset: 0x00057159
Faulting process id: 0x2834
Faulting application start time: 0x01d0ad678d4eb0f5
Faulting application path: C:\Program Files (x86)\BYOND\bin\dreamseeker.exe
Faulting module path: C:\Program Files (x86)\BYOND\bin\byondcore.dll
Report Id: f09c911c-195a-11e5-8ffc-001d60bef81e
1289 seemed to fix the crashing issue that was plaguing us - only managed to get one crash on it:

Faulting application name: dreamseeker.exe, version: 5.0.508.1289, time stamp: 0x557a2ea6
Faulting module name: dreamseeker.exe, version: 5.0.508.1289, time stamp: 0x557a2ea6
Exception code: 0xc0000005
Fault offset: 0x0002b629
Faulting process id: 0x2068
Faulting application start time: 0x01d0aa3a4724baf0
Faulting application path: C:\Games\byond\bin\dreamseeker.exe
Faulting module path: C:\Games\byond\bin\dreamseeker.exe

It seems 1290, however, is a bit less stable...

Faulting application name: dreamseeker.exe, version: 5.0.508.1290, time stamp: 0x5588c0b5
Faulting module name: byondcore.dll, version: 5.0.508.1290, time stamp: 0x55888cc5
Exception code: 0xc0000005
Fault offset: 0x0005bb15
Faulting process id: 0x1148
Faulting application start time: 0x01d0ad649221f0a9
Faulting application path: C:\Games\byond\bin\dreamseeker.exe
Faulting module path: C:\Games\byond\bin\byondcore.dll

Faulting application name: dreamseeker.exe, version: 5.0.508.1290, time stamp: 0x5588c0b5
Faulting module name: ntdll.dll, version: 6.1.7601.18869, time stamp: 0x55636317
Exception code: 0xc0000005
Fault offset: 0x00034400
Faulting process id: 0x1b44
Faulting application start time: 0x01d0ad6727ec5a28
Faulting application path: C:\Games\byond\bin\dreamseeker.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
I suspect 508.1289 and 1290 probably aren't significantly different here. I'll run some traces on this info and see what I can find.
The ntdll.dll crash in 1290 is untraceable; the crash in byondcore.dll happens in the icon destructor, which likely means the problem is somewhere else.

The 1289 crash on the other hand, in dreamseeker.exe, yielded better data. I found that a sanity check in the hit test routine was being done in the wrong place. This does not explain the icon destructor issue but I kind of think these are all related.

I'm still trying to track this down. If you can come up with any kind of test project that reproduces this reliably (mayhe try to use that lighting system in an otherwise basic project, and stress it?) that would be a huge help.
Possibly related due to the usage of the same lighting system, (the crashes only occur when using this specific system, it's highly efficient but it apparently causes crashes), we had one round in which almost all players had nearly continuous client crashes caused by movement of the user it seemed.

It doesn't happen on a per round basis, but appears that when some trigger is activated (that I have no way of identifying yet), the rest of the round will cause these continuous crashes for all players.

(I'm not sure how to generate one of the crash reports shown above)
Problem signature:
Problem Event Name: APPCRASH
Application Name: dreamseeker.exe
Application Version: 5.0.508.1294
Application Timestamp: 55aadcc9
Fault Module Name: byondcore.dll
Fault Module Version: 5.0.508.1294
Fault Module Timestamp: 55aadbde
Exception Code: c0000005
Exception Offset: 0005ba75
OS Version: 6.1.7601.2.1.0.768.3
Locale ID: 1033
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
I have crashed before on paradise, and under both wine and windows, but not very frequently. Maybe twice before in as long as I can remember. I don't have crashlogs, but I can offer that I live in west coast america, and paradise is hosted in sweden. I've often noticed that moving around as a ghost or teleporting (which can happen at sanic speeds due to low ticklag) can lead to the client having to noticeably catch up with all the lighting overlays and objects while it displays the fullbright undertiles/atoms. These are usually the point that my client would crash.
Is it possible that, while the client is catching up to lighting arrays, it gets instructions to animate an icon or overlay or such that hasn't been loaded yet?
Lummox JR resolved issue with message:
A longtime crash that was manifesting in icon destructors was found. This was caused by the client garbage collector firing while the client was trying to resolve a client-side icon operation, when the garbage collector deleted one of the component icons prematurely.
Adding a note that NethIafins was instrumental in sussing out the cause of this bug.