More crashes Both in r1286 and r1290

Page: 1 2
ID:1883464
 
Jun 27 2015, 7:36 am (Edited on Jun 27 2015, 10:16 am)
Not a bug
BYOND Version:507
Operating System:windows server 2012
Web Browser:Firefox 39.0
Applies to:Dream Daemon
Status: Not a bug
This is not a bug. It may be an incorrect use of syntax or a limitation in the software. For further discussion on the matter, please consult the BYOND forums.
http://ss13.pomf.se/uploads/report.html

Faulting application name: dreamdaemon.exe, version: 5.0.508.1290, time stamp: 0x55888dac
Faulting module name: byondcore.dll, version: 5.0.508.1290, time stamp: 0x55888cc5
Exception code: 0xc0000005
Fault offset: 0x001518c9
Faulting process id: 0x118c
Faulting application start time: 0x01d0b0e23ad5b068
Faulting application path: C:\Program Files (x86)\BYOND\bin\dreamdaemon.exe
Faulting module path: C:\Program Files (x86)\BYOND\bin\byondcore.dll
Report Id: 0b1277c6-1cd9-11e5-93fc-001e8cf19ccb
Faulting package full name:
Faulting package-relative application ID:

----

Fault bucket 107426595238, type 1
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: dreamdaemon.exe
P2: 5.0.508.1290
P3: 55888dac
P4: byondcore.dll
P5: 5.0.508.1290
P6: 55888cc5
P7: c0000005
P8: 001518c9
P9:
P10:

Attached files:
C:\Users\ss13\AppData\Local\Temp\WER92E5.tmp.WERInternalMetadata.xml

These files may be available here:
C:\Users\ss13\AppData\Local\Microsoft\Windows\WER\ReportArchive\AppCrash_dreamdaemon.exe_aa56c1fb22dd141ed949a3a85d579f9f6a9127_f8009621

Analysis symbol:
Rechecking for solution: 0
Report Id: 0b1277c6-1cd9-11e5-93fc-001e8cf19ccb
Report Status: 0
Hashed bucket: 1fb351d2d130a8861bb78f895a1e48e6


Faulting application name: dreamdaemon.exe, version: 5.0.507.1286, time stamp: 0x5559633f
Faulting module name: byondcore.dll, version: 5.0.507.1286, time stamp: 0x55596246
Exception code: 0xc0000005
Fault offset: 0x000bd590
Faulting process id: 0x21d0
Faulting application start time: 0x01d0afcc1eee4af8
Faulting application path: C:\Program Files (x86)\BYOND\bin\dreamdaemon.exe
Faulting module path: C:\Program Files (x86)\BYOND\bin\byondcore.dll
Report Id: ea109227-1c2b-11e5-93fc-001e8cf19ccb
Faulting package full name:
Faulting package-relative application ID:

----

Fault bucket 74263499033, type 1
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: dreamdaemon.exe
P2: 5.0.507.1286
P3: 5559633f
P4: byondcore.dll
P5: 5.0.507.1286
P6: 55596246
P7: c0000005
P8: 000bd590
P9:
P10:

Attached files:
C:\Users\ss13\AppData\Local\Temp\WERF45E.tmp.WERInternalMetadata.xml

These files may be available here:
C:\Users\ss13\AppData\Local\Microsoft\Windows\WER\ReportArchive\AppCrash_dreamdaemon.exe_71afe1140a815a6adb6e82e6bba2d7daedbdd2_e755f6b0

Analysis symbol:
Rechecking for solution: 0
Report Id: ea109227-1c2b-11e5-93fc-001e8cf19ccb
Report Status: 0
Hashed bucket: c8a04501b2ef72c2f33af6a005ba8774

This happens in both latest 508 and 507.
Jun 27 2015, 8:10 am
Ok these are happening with a high frequency now, wonder if somebody has figured out a way to crash the server.
Jun 27 2015, 11:47 am
I've tracked the 507.1286 crash so far, though it doesn't make sense. The crash is happening in a comparison between a null value and a number. Specifically, it's null >= number, right after null is changed to 0. But the spot of the crash makes zero sense, as at that point it's not accessing anything but a built-in value.

Still looking into the 1290 issue.
Jun 27 2015, 9:15 pm
The 1290 crash appears to be happening in an overlay or underlay append operation. The checks against an out-of-memory situation are inadequate there; I believe you're simply running out of memory.
Jun 27 2015, 11:04 pm
In response to Lummox JR
Lummox JR wrote:
The 1290 crash appears to be happening in an overlay or underlay append operation. The checks against an out-of-memory situation are inadequate there; I believe you're simply running out of memory.

yeah after posting i noticed every time it crashed it was at 1.9gb ram, but it was like a random spike of memory usage that kept coming on randomly.
Jun 27 2015, 11:38 pm
See if you can get reports on the numbers of various objects in use. In Dream Daemon you can get a memory report. It does sound to me like someone possibly found a loophole in your codebase that causes lots of things to spawn, eventually using up the available memory.
Jun 28 2015, 10:13 am
Yeah i was trying to yesterday, eventually found out that the ctrl+m hotkey for dumping the memory information works a few times then stops. But i'll see if i can grab it.
Jul 16 2015, 2:41 pm (Edited on Jul 16 2015, 4:14 pm)
 
Faulting application name: dreamdaemon.exe, version: 5.0.508.1293, time stamp: 0x55a585d2
Faulting module name: byondcore.dll, version: 5.0.508.1293, time stamp: 0x55a584dc
Exception code: 0xc0000005
Fault offset: 0x000cc6f8
Faulting process id: 0x1870
Faulting application start time: 0x01d0c00df5a78bb8
Faulting application path: C:\Program Files (x86)\BYOND\bin\dreamdaemon.exe
Faulting module path: C:\Program Files (x86)\BYOND\bin\byondcore.dll
Report Id: 7ccdb499-2c02-11e5-93fc-001e8cf19ccb
Faulting package full name:
Faulting package-relative application ID:

Crashing ~5 minutes after world start. Can't dump memory stats as once it occurs it's along side full core usage so DD is unresponsive. But memory usage went from 300mb to 2gb in 5 seconds.

Logged all atom creation and it doesnt seem like its caused by atom/new
Jul 16 2015, 4:26 pm
Hrm. I wonder what it's doing to get the memory that high. It's too bad this isn't running under Linux because that'd pop out a stack trace that would include the last procs called.
Jul 16 2015, 4:52 pm (Edited on Jul 16 2015, 4:57 pm)
gonna see if i can get a stack trace with procdump
edit: http://ss13.pomf.se/uploads/ dreamdaemon.exe_150716_185531.dmp
Jul 16 2015, 6:14 pm
Jul 16 2015, 9:47 pm
I can't use the .dmp files; it's really not feasible to try to get a trace that way (especially with them being so freakishly huge), and anyway it won't tell me the last procs that ran, which is a better clue.
Jul 16 2015, 11:09 pm
In response to Lummox JR
Lummox JR wrote:
I can't use the .dmp files; it's really not feasible to try to get a trace that way (especially with them being so freakishly huge), and anyway it won't tell me the last procs that ran, which is a better clue.

is there nothing at all that can be done at this present time on windows, or possibly if you can add something to get a 'sane' stack trace when it goes into crash time?
Jul 18 2015, 11:45 pm
It seems to come from our code for shuttles, as "moving" a shuttle (moving all movable atoms to the new area and recreating the turfs) will cause the memory leak after a few instances of it being called. Here is a posting of the code.

proc/move_research_shuttle()
    writepanic("[__FILE__].[__LINE__] \\/proc/move_research_shuttle() called tick#: [world.time]")
    if(research_shuttle_moving) return
    research_shuttle_moving = 1
    spawn(research_shuttle_tickstomove*10)
        var/area/fromArea
        var/area/toArea
        if (research_shuttle_location == 1)
            fromArea = locate(/area/shuttle/research/outpost)
            toArea = locate(/area/shuttle/research/station)
        else
            fromArea = locate(/area/shuttle/research/station)
            toArea = locate(/area/shuttle/research/outpost)
            var/list/search = fromArea.search_contents_for(/obj/item/weapon/disk/nuclear)
            if(!isemptylist(search))
                research_shuttle_moving = 0
                return

        var/list/dstturfs = list()
        var/throwy = world.maxy

        for(var/turf/T in toArea)
            dstturfs += T
            if(T.y < throwy)
                throwy = T.y

        // hey you, get out of the way!
        for(var/turf/T in dstturfs)
            // find the turf to move things to
            var/turf/D = locate(T.x, throwy - 1, 1)
            //var/turf/E = get_step(D, SOUTH)
            for(var/atom/movable/AM as mob|obj in T)
                AM.Move(D)

            if(istype(T, /turf/simulated))
                del(T)
        //Do I really need to explain this loop?
        for(var/atom/A in toArea)
            if(istype(A,/mob/living))
                var/mob/living/unlucky_person = A
                unlucky_person.gib()
            // Weird things happen when this shit gets in the way.
            if(istype(A,/obj/structure/lattice) \
                || istype(A, /obj/structure/window) \
                || istype(A, /obj/structure/grille))
                qdel(A)

        fromArea.move_contents_to(toArea)
        if (research_shuttle_location)
            research_shuttle_location = 0
        else
            research_shuttle_location = 1

        for(var/mob/M in toArea)
            if(M.client)
                spawn(0)
                    if(M.buckled)
                        shake_camera(M, 3, 1) // buckled, not a lot of shaking
                    else
                        shake_camera(M, 10, 1) // unbuckled, HOLY SHIT SHAKE THE ROOM
            if(istype(M, /mob/living/carbon))
                if(!M.buckled)
                    M.Weaken(3)

        research_shuttle_moving = 0
    return



/area/proc/move_contents_to(var/area/A, var/turftoleave=null, var/direction = null)
    writepanic("[__FILE__].[__LINE__] (no type)([usr ? usr.ckey : ""])  \\/area/proc/move_contents_to() called tick#: [world.time]")
    //Takes: Area. Optional: turf type to leave behind.
    //Returns: Nothing.
    //Notes: Attempts to move the contents of one area to another area.
    //       Movement based on lower left corner. Tiles that do not fit
    //       into the new area will not be moved.

    if(!A || !src) return 0

    var/list/turfs_src = get_area_turfs(src.type)
    var/list/turfs_trg = get_area_turfs(A.type)

    var/src_min_x = 0
    var/src_min_y = 0
    for (var/turf/T in turfs_src)
        if(T.x < src_min_x || !src_min_x) src_min_x = T.x
        if(T.y < src_min_y || !src_min_y) src_min_y = T.y

    var/trg_min_x = 0
    var/trg_min_y = 0
    for (var/turf/T in turfs_trg)
        if(T.x < trg_min_x || !trg_min_x) trg_min_x = T.x
        if(T.y < trg_min_y || !trg_min_y) trg_min_y = T.y

    var/list/refined_src = new/list()
    for(var/turf/T in turfs_src)
        refined_src += T
        refined_src[T] = new/datum/coords
        var/datum/coords/C = refined_src[T]
        C.x_pos = (T.x - src_min_x)
        C.y_pos = (T.y - src_min_y)

    var/list/refined_trg = new/list()
    for(var/turf/T in turfs_trg)
        refined_trg += T
        refined_trg[T] = new/datum/coords
        var/datum/coords/C = refined_trg[T]
        C.x_pos = (T.x - trg_min_x)
        C.y_pos = (T.y - trg_min_y)

    var/list/fromupdate = new/list()
    var/list/toupdate = new/list()

    moving:
        for (var/turf/T in refined_src)
            var/area/AA = get_area(T)
            var/datum/coords/C_src = refined_src[T]
            for (var/turf/B in refined_trg)
                var/datum/coords/C_trg = refined_trg[B]
                if(C_src.x_pos == C_trg.x_pos && C_src.y_pos == C_trg.y_pos)

                    var/old_dir1 = T.dir
                    var/old_icon_state1 = T.icon_state
                    var/old_icon1 = T.icon
                    var/image/undlay = image("icon"=B.icon,"icon_state"=B.icon_state,"dir"=B.dir)
                    undlay.overlays = B.overlays.Copy()
                    var/prevtype = B.type

                    var/turf/X = B.ChangeTurf(T.type)
                    for(var/key in T.vars)
                        if(key in ignored_keys) continue
                        if(istype(T.vars[key],/list))
                            var/list/L = T.vars[key]
                            X.vars[key] = L.Copy()
                        else
                            X.vars[key] = T.vars[key]
                    if(ispath(prevtype,/turf/space))//including the transit hyperspace turfs
                        if(ispath(AA.type, /area/syndicate_station/start) || ispath(AA.type, /area/syndicate_station/transit))//that's the snowflake to pay when people map their ships over the snow.
                            X.underlays += undlay
                        else if(T.underlays.len)
                            for(var/Over in T.underlays)
                                X.underlays += T.underlays
                        else
                            X.underlays += undlay
                    else
                        X.underlays += undlay
                    X.dir = old_dir1
                    X.icon_state = old_icon_state1
                    X.icon = old_icon1 //Shuttle floors are in shuttle.dmi while the defaults are floors.dmi

                    var/turf/simulated/ST = T

                    if(istype(ST) && ST.zone)
                        var/turf/simulated/SX = X

                        if(!SX.air)
                            SX.make_air()

                        SX.air.copy_from(ST.zone.air)
                        ST.zone.remove(ST)

                    /* Quick visual fix for some weird shuttle corner artefacts when on transit space tiles */
                    if(direction && findtext(X.icon_state, "swall_s"))

                        // Spawn a new shuttle corner object
                        var/obj/corner = new()
                        corner.loc = X
                        corner.density = 1
                        corner.anchored = 1
                        corner.icon = X.icon
                        corner.icon_state = replacetext(X.icon_state, "_s", "_f")
                        corner.tag = "delete me"
                        corner.name = "wall"

                        // Find a new turf to take on the property of
                        var/turf/nextturf = get_step(corner, direction)
                        if(!nextturf || !istype(nextturf, /turf/space))
                            nextturf = get_step(corner, turn(direction, 180))


                        // Take on the icon of a neighboring scrolling space icon
                        X.icon = nextturf.icon
                        X.icon_state = nextturf.icon_state


                    for(var/obj/O in T)

                        // Reset the shuttle corners
                        if(O.tag == "delete me")
                            X.icon = 'icons/turf/shuttle.dmi'
                            X.icon_state = replacetext(O.icon_state, "_f", "_s") // revert the turf to the old icon_state
                            X.name = "wall"
                            del(O) // prevents multiple shuttle corners from stacking
                            continue
                        if(!istype(O,/obj)) continue
                        O.forceMove(X)
                    for(var/mob/M in T)
                        if(!M.move_on_shuttle)
                            continue
                        M.forceMove(X)

//                  var/area/AR = X.loc

//                  if(AR.lighting_use_dynamic)                         //TODO: rewrite this code so it's not messed by lighting ~Carn
//                      X.opacity = !X.opacity
//                      X.SetOpacity(!X.opacity)

                    toupdate += X

                    if(turftoleave)
                        fromupdate += T.ChangeTurf(turftoleave)
                    else
                        if(ispath(AA.type, /area/syndicate_station/start))
                            T.ChangeTurf(/turf/unsimulated/floor)
                            T.icon = 'icons/turf/snow.dmi'
                            T.icon_state = "snow"
                        else
                            T.ChangeTurf(get_base_turf(T.z))
                            if(istype(T, /turf/space))
                                switch(universe.name)   //for some reason using OnTurfChange doesn't actually do anything in this case.
                                    if("Hell Rising")
                                        T.overlays += "hell01"
                                    if("Supermatter Cascade")
                                        T.overlays += "end01"


                    refined_src -= T
                    refined_trg -= B
                    continue moving

    var/list/doors = new/list()

    if(toupdate.len)
        for(var/turf/simulated/T1 in toupdate)
            for(var/obj/machinery/door/D2 in T1)
                doors += D2
            /*if(T1.parent)
                air_master.groups_to_rebuild += T1.parent
            else
                air_master.mark_for_update(T1)*/

    if(fromupdate.len)
        for(var/turf/simulated/T2 in fromupdate)
            for(var/obj/machinery/door/D2 in T2)
                doors += D2
            /*if(T2.parent)
                air_master.groups_to_rebuild += T2.parent
            else
                air_master.mark_for_update(T2)*/

    for(var/obj/O in doors)
        O:update_nearby_tiles()
Jul 18 2015, 11:45 pm
I actually managed to reproduce this locally without DD crashing, http://i.imgur.com/QVCtaBY.png

Interesting enough DS cpu usage spiked to an entire core's worth prior to the memory leak.

http://ss13.pomf.se/uploads/2015-07-19_01-39-17.txt
'fresh' server vs memory leaked server

usage immediately spiked from 360mb to 1.9gb with nothing in between, all from just moving the ingame shuttle a few times. http://i.imgur.com/Su9PnST.png scary
Jul 19 2015, 12:09 am
That code is a bit much to follow. Mind breaking down what's happening step by step? I assume there's a specific part of this that's responsible for the leak, and that being the case it'd help a ton if you could narrow that down into a demo.
Jul 19 2015, 12:28 am
In response to Lummox JR
Lummox JR wrote:
That code is a bit much to follow. Mind breaking down what's happening step by step? I assume there's a specific part of this that's responsible for the leak, and that being the case it'd help a ton if you could narrow that down into a demo.

can you explain what exactly an id array is in the memory information context? cause that seems to be the thing taking up the extra 1.6gb
Jul 19 2015, 12:41 am (Edited on Jul 19 2015, 12:47 am)
proc/move_research_shuttle()

In this instance, a shuttle has two designated areas where it can be moved to. If it is at one located, the shuttle is 'moved' to the other when this is called. This is done using areas.

We locate() the area typepath for the location where the shuttle is going to be moved and locate() the area typepath where the shuttle currently is so we have the two areas as references. Everything in the new area of the shuttle is simply moved out of the way or deleted to make room for it.

It then calls the following proc

/area/proc/move_contents_to()

Here is where we actually move everything to the new area. Everything before the loop begins seems to be merely lining up the turfs lists so that the same index for both will refer to the same shuttle spot at both locations.

In summary, after the loop begins all atom/movables are moved to the equivalent tile in the new area, and the turf is changed to be identical to the old turf and the turfs vars are copied over to the new turf. Much of the code in this loop is merely for the purpose of trying to recreate identical turfs

In the order shown, the loop is basically
1. Copying all of the turfs vars, icon_state, dir, etc. Applying them to the changed turf at the new location.
2. Creating new air datum if the turf is supposed to have air
3. Creating shuttle corners if the shuttle has a triangular corner
4. Forcibly moving all objects to the new shuttle location
5. Forcibly moving all mobs to the new shuttle location.
6. The rest of the loop has some stuff for specific scenarios that isn't relevant here

After the loop we are merely updating the surrounding structures/objects/turfs like doors and tiles for properties that depend on the surrounding tiles.


The memory leak we are experience is modeled below
'fresh' server:
server mem usage:
Prototypes:
obj: 1536064 (9106)
mob: 1539328 (204)
proc: 12582876 (21353)
str: 6190403 (117480)
appearance: 7929503 (8525)
id array: 12616392 (41250)
map: 67513100 (255,255,6)
objects:
mobs: 329792 (548)
objs: 25065744 (71021)
datums: 21015520 (343542)
lists: 23603968 (465187)


Memory leaked server:
server mem usage:
Prototypes:
obj: 1536028 (9106)
mob: 1539292 (204)
proc: 12583744 (21351)
str: 5857964 (110555)
appearance: 13367816 (36813)
id array: 1636052528 (71197)
map: 69418244 (255,255,6)
objects:
mobs: 334928 (515)
objs: 25199712 (71021)
datums: 21057952 (343542)
lists: 24013528 (465187)

The appearances and id arrays skyrockets after merely 'moving' these shuttles back and forth a number of times. This code is not new, but it has undergone some very minor revisions in the past few weeks.
Jul 19 2015, 12:53 am
Now I think what is unique about this code in terms of the larger codebase as relevant to appeareances/id arrays
1. that it uses locate() for an area typepath which is very rarely used in general,
2. that it creates a lists of these coordinates datums like such to align the specific shuttle positions
    var/list/refined_trg = new/list()
    for(var/turf/T in turfs_trg)
        refined_trg += T
        refined_trg[T] = new/datum/coords
        var/datum/coords/C = refined_trg[T]
        C.x_pos = (T.x - trg_min_x)
        C.y_pos = (T.y - trg_min_y)

and
3. that it copies over all the turf variables to make an appearance like such
                    var/old_dir1 = T.dir
                    var/old_icon_state1 = T.icon_state
                    var/old_icon1 = T.icon
                    var/image/undlay = image("icon"=B.icon,"icon_state"=B.icon_state,"dir"=B.dir)
                    undlay.overlays = B.overlays.Copy()
                    var/prevtype = B.type

                    var/turf/X = B.ChangeTurf(T.type)
                    for(var/key in T.vars)
                        if(key in ignored_keys) continue
                        if(istype(T.vars[key],/list))
                            var/list/L = T.vars[key]
                            X.vars[key] = L.Copy()
                        else
                            X.vars[key] = T.vars[key]
                    if(ispath(prevtype,/turf/space))//including the transit hyperspace turfs
                        if(ispath(AA.type, /area/syndicate_station/start) || ispath(AA.type, /area/syndicate_station/transit))//that's the snowflake to pay when people map their ships over the snow.
                            X.underlays += undlay
                        else if(T.underlays.len)
                            for(var/Over in T.underlays)
                                X.underlays += T.underlays
                        else
                            X.underlays += undlay
                    else
                        X.underlays += undlay
                    X.dir = old_dir1
                    X.icon_state = old_icon_state1
                    X.icon = old_icon1 //Shuttle floors are in shuttle.dmi while the defaults are floors.dmi
Jul 19 2015, 12:54 am
An ID array is a list of ID values for a certain type. The overlays and underlays lists are ID arrays of appearances. Each ID array is immutable, so regardless of type if two arrays have the exact same IDs in the exact same order, they'll only count as one.

In this line, Copy() is irrelevant:

undlay.overlays = B.overlays.Copy()

Because all you're doing is creating a temporary user-defined list via Copy(), and then assigning that to overlays converts it back to an ID array, which should be identical to B.overlays. I don't think this could cause a leak, though it likely increases churn.

It's possible that you might be creating new, unique ID arrays through something that's going on in this process. Although those get reference counted, they might not be recycled right away if the appearances they're attached to have been seen. Appearances that have been sent to a client aren't deleted until all clients report that they no longer need those appearances.

This is why I think that might be an issue:

X.underlays += undlay

undlay itself is a new image (though not likely to be unique the next time it's created for the same shuttle turf), with the turf's old overlays attached. So whatever X is, it's getting a list of underlays that's composed of the main image appearance plus the original overlays. I'm imagining a scenario where this kind of thing repeats, and the overlays list keeps growing each time.

Based on your numbers, it looks like not only is the number of ID arrays growing, but the size of each one is growing as well. I still can't follow your code all that well, but it seems to me that it's creating monster ID arrays.
Page: 1 2