Fairly rare Access Violation crash of dd.exe

ID:2919399
 
Apr 3, 12:41 am (Edited on Apr 3, 12:50 am)
Hello there

We seem to be occasionally encountering the exception displayed in the title.

Unhandled exception 0xc0000005
byondcore.dll + 0x935D5    insertIcon + 0xB95
byondcore.dll + 0x11C52B    SteamAvailableCheck + 0x8FBB
byondcore.dll + 0x1553A3    LocalDB::HubToJS + 0x1493
byondcore.dll + 0x13C075    SteamAvailableCheck + 0x28B05
byondcore.dll + 0x147E0A    SteamAvailableCheck + 0x3489A
byondcore.dll + 0x12579A    SteamAvailableCheck + 0x1222A
byondcore.dll + 0x124079    SteamAvailableCheck + 0x10B09
byondcore.dll + 0x139A9B    SteamAvailableCheck + 0x2652B
byondcore.dll + 0x147E0A    SteamAvailableCheck + 0x3489A
byondcore.dll + 0x12579A    SteamAvailableCheck + 0x1222A
byondcore.dll + 0x124079    SteamAvailableCheck + 0x10B09
byondcore.dll + 0x139A9B    SteamAvailableCheck + 0x2652B
byondcore.dll + 0x147E0A    SteamAvailableCheck + 0x3489A
byondcore.dll + 0x12579A    SteamAvailableCheck + 0x1222A
byondcore.dll + 0x124079    SteamAvailableCheck + 0x10B09
byondcore.dll + 0x139A9B    SteamAvailableCheck + 0x2652B
byondcore.dll + 0x147E0A    SteamAvailableCheck + 0x3489A
byondcore.dll + 0x12579A    SteamAvailableCheck + 0x1222A
byondcore.dll + 0x124079    SteamAvailableCheck + 0x10B09
byondcore.dll + 0x139A9B    SteamAvailableCheck + 0x2652B
byondcore.dll + 0x147E0A    SteamAvailableCheck + 0x3489A
byondcore.dll + 0x12579A    SteamAvailableCheck + 0x1222A
byondcore.dll + 0x124079    SteamAvailableCheck + 0x10B09
byondcore.dll + 0x139A9B    SteamAvailableCheck + 0x2652B
byondcore.dll + 0x147E0A    SteamAvailableCheck + 0x3489A
byondcore.dll + 0x12579A    SteamAvailableCheck + 0x1222A
byondcore.dll + 0x124079    SteamAvailableCheck + 0x10B09
byondcore.dll + 0x139A9B    SteamAvailableCheck + 0x2652B
byondcore.dll + 0x12FF3C    SteamAvailableCheck + 0x1C9CC
byondcore.dll + 0x15B359    LocalDB::HubToJS + 0x7449
byondcore.dll + 0x27AA28    TimeLib::Event_io + 0x98
USER32.dll + 0x4636B    AddClipboardFormatListener + 0x4B
USER32.dll + 0x36E0C    CallWindowProcW + 0xB2C
USER32.dll + 0x35F4B    DispatchMessageW + 0x23B
USER32.dll + 0x35D20    DispatchMessageW + 0x10
dd.exe + 0x3E5C
dd.exe + 0x16A2
dd.exe + 0x4938
KERNEL32.DLL + 0x205C9    BaseThreadInitThunk + 0x19
ntdll.dll + 0x678BD    RtlGetAppContainerNamedObjectPath + 0xED
ntdll.dll + 0x6788D    RtlGetAppContainerNamedObjectPath + 0xBD


Recent proc calls:
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/proc/Blend
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/New
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/New
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/New
/icon/New
/icon/proc/RscFile
/icon/proc/Icon
/icon/proc/Blend
/atom/proc/QueueOverlayUpdate
/icon/proc/RscFile
/datum/controller/subsystem/overlays/proc/GetIconAppearance
/datum/controller/subsystem/overlays/proc/AppearanceListEntry
/datum/controller/subsystem/overlays/proc/GetAppearanceList
/atom/proc/add_overlay
/icon/proc/Blend
/icon/New
/icon/proc/Icon
/icon/proc/Blend
/atom/proc/QueueOverlayUpdate
/icon/proc/RscFile
/datum/controller/subsystem/overlays/proc/GetIconAppearance
/datum/controller/subsystem/overlays/proc/AppearanceListEntry
/datum/controller/subsystem/overlays/proc/GetAppearanceList
/atom/proc/add_overlay
/icon/proc/Blend
/icon/New
/icon/proc/Blend
/obj/item/organ/external/proc/apply_colouration
/icon/New
/obj/item/organ/external/proc/is_hidden_by_markings
/datum/species/get_bodytype
/obj/item/organ/external/get_icon
/obj/item/organ/external/proc/is_hidden_by_sprite_accessory
/obj/item/organ/external/proc/is_stump
/icon/proc/Icon
/icon/proc/Blend
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/New
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/proc/Insert
/icon/proc/Icon
/icon/proc/Blend
/icon/proc/Icon
/icon/proc/Insert


Several crashes seem to have the same exception and rough state in the proc history / the current executing proc.

We're running a Windows server via dd.exe. We encountered this on 515.1633 and 515.1634.

I'm unsure if this is potentially a BYOND bug or if this might be an issue with our server or code, so i'll post it here for now. Feel free to move this if this is infact a bug.

This is very difficult to troubleshoot. It seems to happen randomly, within the timespan of about 3-6 days, and thereafter occurs frequently until completely restarting our Windows VM. But, after then, several days of stability is attainable again.

first time poster, please excuse any mistakes.
Apr 3, 9:36 am
This appears to be very similar to an issue I recently worked on with another user's SS13 codebase where they had a proc called getFlatIcon(). Even though that functionality now is handled by RenderIcon(), they had their reasons for not wanting to use it. There's a very, very pernicious bug where DDMI icons resolve to real icons on the server side, which will happen in the event of overly complex operation stacks.

In the recent case I investigated, their issue was resolved by fixing an error in their code. There was a particular mob whose body parts were handled a little differently from others, and it exposed an error in the way they were caching certain /icon objects based on the body part.

The fix is discussed here: https://discord.com/channels/725444629172060262/ 725458598213451879/1214263780762980393

The short version of the problem is that they were adding icons to their internal cache at a different point in the code before they fixed it. They were making changes to the cached /icon objects via their call to their render_mods_unto() proc, when instead it needed to be moved to before the icons were added to that cache list.

I suspect your game is using some similar code and running into a similar situation. There's still an underlying BYOND bug, but unfortunately until I have a much more reliable test case to reproduce it (one that acts the exact same way every time, so in a world with no randomness), it won't be going anywhere.

The short-term fix for you is to:

1) Reduce or eliminate reliance on icon math.
2) Use RenderIcon() wherever possible.
3) If you're using any kind of caching of generated /icon objects from your icon math, check that the icons you put in the cache aren't going through any more changes.
Apr 4, 3:20 pm
In response to Lummox JR
We inacted a bunch of changes to our overlays/icon stuff and hope to see if that helps make the issue go away. If not, i'll try farm up some logs and the like to see if i can in any way help you diagnose it, but this one seems like a doozey, especially on your end, heh.

The advice here helped a bunch though, thankyou.
Apr 18, 9:17 am
In response to Lummox JR
Continuing to dig through our code.

The vibe i'm getting is any form of cached icon being referred somewhere else in code, being then altered (Via .blend or any other icon altering operations) is the (extremely rare) causer of this crash.

In one case in our code, when we're creating the icon for a mob, we grab the icon of the 'chest organ' which is considered the core icon of a mob and then start doing things with it.

I think a mistake we make is;
var/icon/mob_icon = chest_organ.get_icon()


where chest_organ.get_icon() returns object chest_organ's own icon, but the actual reference to it's icon.

I assume instead i should be making a 'copy' of this icon instead, as later operations on this same reference is modifying a 'cached' icon? Like so?:

var/icon/mob_icon = new(chest_organ.get_icon())


This leads me to perhaps speculate on what might be the core issue here. If an icon variable is being referenced anywhere else in code (Say by a var/global/list_of_icons or even an object's var/an_icon), and it is then edited outside of this, perhaps this is what causes the crash?

Or, for example;

GOOD:
var/global/a_list = list()

/proc/something()
   var/icon/A = new()
   A.Blend(something_to_blend)
   a_list[1] = A
   ///a_list[1] and A must now be considered READ ONLY


BAD:
var/global/a_list = list()

/proc/something()
   if(a_list[1])
      a_list[1].Blend(something_to_blend)
   else
      var/icon/A = new()
      a_list[1] = A


where, subsequent executions of /proc/something() in the BAD case carry a very very small chance to cause the access violation (perhaps one in a million)