Random world.Reboot() time crashes and hangs that MIGHT be sleep related.

ID:2324176
 
Dec 12 2017, 12:37 am
Resolved
Multiple reboots caused crashing.
BYOND Version:512.1397
Operating System:Windows Server 2012 R2 64bit
Web Browser:Chrome 63.0.3239.84
Applies to:Dream Daemon
Status: Resolved (512.1399)
This issue has been resolved.
Descriptive Problem Summary:
I had to revert our 512 testing server to 512.1395 from 512.1398 because of an issue with repeated lockups and/or crashes near world.reboot() time that also happened in 512.1397.


A coder said they once had the same issue in a project that involved a lot of sleeps.

The timing of the bug would make sense. I didn't test 1396 thou because of the issues.

Once I have more time I'll try to get this down to a test case and/or analyze our logs to make sure they always happened near world.Reboot().

One other hint is it seemed to happen shortly after updating the game code. the coder who had the issue said it happened when they had updated the code to their testing project and restarted.

Since this involves live swapping dmbs (something the ref for world.Reboot() says is supported), it could be relating to references that are sticking around becoming invalid.
Dec 12 2017, 6:51 am
So to possibly add to this, I've noticed that if I start a server, close it, then start one again, Dream daemon stops working and will crash. (Version 512.1398)

This occurs with my full sized game however I did get it to happen once or twice with a very small scale test server (20x20 map, 2 code files)

To note, I've only recently tested this due to reports my game is going down after reboots and I don't currently have any logs.
Dec 12 2017, 10:46 am
If you can get me crash details for those reboots, that would be a very very big help.

Logically I would expect the proc scheduler change to be the cause of this, but I'd like to be able to narrow down how and why it's happening.
Dec 12 2017, 1:17 pm
this is from my own computer. (running 1398)

I had stopped the world, made an edit, and than restarted the started

Faulting application name: dreamdaemon.exe, version: 5.0.512.1398, time stamp: 0x5a2aff06
Faulting module name: ntdll.dll, version: 6.1.7601.23403, time stamp: 0x56f5847f
Exception code: 0xc0000005
Fault offset: 0x0002e064
Dec 12 2017, 1:34 pm
Got it to happen with vs attached
    ntdll.dll!_RtlFreeHeap@12()  + 0x3f bytes
        kernel32.dll!_HeapFree@12()  + 0x14 bytes
    msvcr120.dll!free(void * pBlock)  Line 51 C
>   byondcore.dll!74754a26()
    [Frames below may be incorrect and/or missing, no symbols loaded for byondcore.dll]
    byondcore.dll!747f1cd1()
    byondcore.dll!747f235c()
    byondcore.dll!747f1fd8()
    byondcore.dll!7472a2f2()
    dreamdaemon.exe!010b07f9()
    mfc120.dll!CCmdTarget::OnCmdMsg(unsigned int nID, int nCode, void * pExtra, AFX_CMDHANDLERINFO * pHandlerInfo)  Line 356 + 0x5 bytes  C++
    mfc120.dll!CWnd::ReflectChildNotify(unsigned int uMsg, unsigned int wParam, long lParam, long * pResult)  Line 3464 + 0x15 bytes    C++
    mfc120.dll!CWnd::SendChildNotifyLastMsg(long * pResult)  Line 3378 + 0x16 bytes C++
    mfc120.dll!CWnd::ReflectLastMsg(HWND__ * hWndChild, long * pResult)  Line 3416  C++



    dreamdaemon.exe C:\Program Files (x86)\BYOND\bin\dreamdaemon.exe    N/A N/A Binary was not built with debug information.        1    5.00.512.1398   12/8/2017 13:07    010A0000-010DD000 [4296] dreamdaemon.exe: Native
    ntdll.dll   C:\Windows\SysWOW64\ntdll.dll   N/A N/A Symbols loaded (source information stripped).   C:\Users\kyle\AppData\Local\Temp\SymbolCache\wntdll.pdb\DC57E4561F7C4103A0AD04A2DE3EE88B2\wntdll.pdb    2    6.1.7600.16385 (win7_rtm.090713-1255)  3/25/2016 10:33    773D0000-77550000  [4296] dreamdaemon.exe: Native
    kernel32.dll    C:\Windows\SysWOW64\kernel32.dll    N/A N/A Symbols loaded (source information stripped).   C:\Users\kyle\AppData\Local\Temp\SymbolCache\wkernel32.pdb\733549B0F2264A4B9FA433E6107B89D42\wkernel32.pdb   3    6.1.7601.23403 (win7sp1_ldr.160325-0600)   3/25/2016 10:36    76B20000-76C30000   [4296] dreamdaemon.exe: Native
    KernelBase.dll  C:\Windows\SysWOW64\KernelBase.dll  N/A N/A Symbols loaded (source information stripped).   C:\Users\kyle\AppData\Local\Temp\SymbolCache\wkernelbase.pdb\CFB359894E674608B142253215E6F4061\wkernelbase.pdb  4    6.1.7601.23403 (win7sp1_ldr.160325-0600)   3/25/2016 10:36    75040000-75087000   [4296] dreamdaemon.exe: Native
    byondwin.dll    C:\Program Files (x86)\BYOND\bin\byondwin.dll   N/A N/A Binary was not built with debug information.        5    5.00.512.1398   12/8/2017 13:06    74990000-74B46000 [4296] dreamdaemon.exe: Native
    byondcore.dll   C:\Program Files (x86)\BYOND\bin\byondcore.dll  N/A N/A Binary was not built with debug information.        6    5.00.512.1398   12/8/2017 13:05    74670000-74985000   [4296] dreamdaemon.exe: Native
    byondext.dll    C:\Program Files (x86)\BYOND\bin\byondext.dll   N/A N/A Binary was not built with debug information.        7    5.00.512.1398   12/8/2017 13:03    74490000-7466B000   [4296] dreamdaemon.exe: Native
    msvcr120.dll    C:\Program Files (x86)\BYOND\bin\msvcr120.dll   N/A N/A Symbols loaded. C:\Users\kyle\AppData\Local\Temp\SymbolCache\MicrosoftPublicSymbols\msvcr120.i386.pdb\16F5E2EF340A453ABC8B8F67DC6FD8082\msvcr120.i386.pdb    18    12.00.21005.1 built by: REL 10/4/2013 18:43    743A0000-7448E000  [4296] dreamdaemon.exe: Native
Dec 12 2017, 1:55 pm
And have a reproduction project

world
    fps = 60      // 25 frames per second
    icon_size = 32    // 32x32 icon size by default

    view = 6     // show up to 6 tiles outward from center (13x13 view)
    
    New()
        ..()
        spawn()
            for (var/i in 1 to 1000)
                spawn(rand(1,world.fps)*world.tick_lag)
                    if (prob(5))
                        Reboot()


run that and you'll get a crash within a few seconds
Dec 12 2017, 2:06 pm
Thanks for that! That test project was the smoking gun I needed.

Looks like multiple reboots triggered the problem, and it was because a pointer used for the new proc queue was being freed but never reset to null. Reallocs to the pointer after that caused the crash.
Dec 12 2017, 2:06 pm
Lummox JR resolved issue with message:
Multiple reboots caused crashing.