Sorry to double post (again), I was advised to alert you that we animate color matrices extensively, and that it is a problem that has been reported in the past.

Still collecting that data you asked for.
Animating color matrices shouldn't cause a problem on the server, though. I wouldn't rule anything out at this point but the server doesn't actually do any animation itself, so if this were a problem it would probably be in how and when the color matrix is set.

This does however give me a thought. Is there an earlier, semi-recent version that didn't experience this issue? Automatic rehashing went into the internal DMHash class not that long ago, and color matrices use a hash to check for uniqueness so as to conserve memory. While I feel fairly comfortable with the code, it's something to look at at least.
You said world.cpu raises a lot.

Have you tried running a code profile once this happens. (protip, you can grant admins and server ops access to view the code profile in the client by adding their ckey to the admins list via world.SetConfig().)

This requires the world be compiled in debugging mode, but knowing what kinds of procs are all of the sudden taking longer would give you a hint. When you do it, wait for the major slowdown, then run the code profile, or reset it if it was already running, wait a bit, hit refresh, then sort by self cpu (realtime is useless for this).

The procs with the obviously high cpu time are more likely than not going to be working with whatever has the memory leak.

This absolutely looks like some kind of corruption or explosion in one of the collections byond uses to dedupe and cache certain complex structures.

The other way to find the code is to try and reproduce the memory growth in local testing, then start speeding up any of the regular timed tasks until you can make the memory growth also speed up, that will tell you what game systems interact with the buggy byond system, and give lummox a hint on what byond system is buggy.
Sorry this took so long to respond to. Been really busy and trying different methods of circumventing the problem-- one of which is rebooting the server hourly which keeps the memory from spilling and becoming unmanageable.

Tonight is the first time since my last analysis where I could sit down and really record any data on the issue and I've found that DD now goes up to almost 2 gigabytes in memory usage. Since the last time, I've re-enabled a lot of the old 'animate' procs I had cut out to try and reduce/eliminate lag. Before, the server would rise from roughly 270MB of memory to 500-600 and then stall out into the lockup. Tonight, I've caught it at that level, then watched it rapidly increase.









The astonishing thing is just that the server han't locked up yet, and I've been trying to let it eexplode to get a crash report but I've already put the playerbase through enough with crashes and such so I aborted the process early.

This is the server memory usage at the start:
Server mem usage:
prototypes:
obj: 686 KB (3,217)
mob: 12 KB (768)
proc: 4.07 MB (3,654)
str: 687 KB (16,646)
appearance: 300 KB (9,212)
filter: 256 B (0)
id array: 7.68 MB (12,101)
map: 30 MB (1000,1000,4)
objects:
mobs: 368 KB (560)
objs: 20.3 MB (131,648)
datums: 56.9 KB (686)
images: 79.8 KB (1,039)
lists: 1.92 MB (27,258

This is the server memory usage after 1713 mb.
Server mem usage:
prototypes:
obj: 686 KB (3,217)
mob: 12 KB (768)
proc: 4.07 MB (3,653)
str: 794 KB (18,776)
appearance: 579 KB (17,631)
filter: 256 B (0)
id array: 7.69 MB (12,849)
map: 30 MB (1000,1000,4)
objects:
mobs: 676 KB (767)
objs: 22 MB (135,679)
datums: 1.01 MB (2,667)
images: 1.53 MB (21,214)
lists: 2.59 MB (34,773)



The first time I seen a drastic increase in memory in any department too, but it's also after re-enabling a lot of the animations.
Animations will take up some room on the server, but they shouldn't stick around after their loops end. However, if they loop forever there's a chance they could clog things up if you just leave them lying around.

Can you show me some info about the animations you reenabled?
This is what a lot of our animations may look like. Color changes/flashes, pixel shifts, fade ins, fade outs.
proc/thunderpunch(var/datum/move/move)
if(current_hp < 1)
return

if(world.time < last_animation)
return

if(world.time < last_attack + 30)
return

spawn()

for(var/mob/pokemon/r in get_step(src, dir))

if(world.time < r.last_animation)
return

if(!move.check_pp()) return

if(world.time > r.last_hit + 20)
r.last_hit = world.time
r.last_attack = world.time - 15
else continue

if(r.current_hp < 1) continue

last_attack = world.time

initiate_combat(r)

shift_pokemon_forward(16)

var/obj/k = new(r.loc)
k.layer = 8
if(!saved_icons["punch1"])
var/icon/i = icon('37. Thrash.dmi')
i.Scale(16, 16)
i.Scale(32, 32)
saved_icons["punch1"] = i
k.icon = saved_icons["punch1"]
k.icon_state = "1"
k.pixel_y = 8

var/obj/k2 = new(r.loc)
k2.icon = k.icon
k2.icon_state = "2"
k2.layer = 7
k2.alpha = 100
k2.pixel_y = 8

var/matrix/m = matrix()
m.Scale(1.5, 1.5)

animate(k2, transform = m, 2.5)

spawn(2.5) del(k2)

spawn(2) del(k)

animate(src, color = list(-1,0,0,0, 0,-1,0,0, 0,0,-1,0, 0,0,0,1, 1,1,1,0), 1)
animate(color = null, 0.5)

animate(r, color = list(-1,0,0,0, 0,-1,0,0, 0,0,-1,0, 0,0,0,1, 1,1,1,0), 1)
animate(color = null, 0.5)

view() << sound('Hit.wav')

sleep(1)

r.vibrate()

sleep(2)

//thunder

view() << sound('9. Thunder Punch.wav')

for(var/rep = 0, rep < 4, rep++)
var/obj/t = new(r.loc)
t.pixel_x = 0
t.pixel_y = 64 - (32 * rep)
t.layer = 7
if(!saved_icons["thunder_punch"])
var/icon/i = icon('9. Thunder Punch.dmi')
i.Scale(16, 16)
i.Scale(32, 32)
saved_icons["thunder_punch"] = i
flick(saved_icons["thunder_punch"], t)

spawn(4) del(t)

sleep(0.5)

sleep(5)

r.take_damage(move, src)

view() << sound('Damage.wav')


Below is something specific that had its animations/colors disabled before that was turned on over the past week.

proc/poisonstrike(var/datum/move/move)
if(current_hp < 1)
return

if(world.time < last_animation)
return

if(world.time < last_attack + 30)
return

spawn()

for(var/mob/pokemon/r in get_step(src, dir))

if(world.time < r.last_animation)
return
if(!move.check_pp()) return
if(world.time > r.last_hit + 20)
r.last_hit = world.time
r.last_attack = world.time - 15
else continue

if(r.current_hp < 1) continue

last_attack = world.time

initiate_combat(r)

for(var/n = 0, n < 1, n++)
var/obj/f = new(loc)
f.layer = 6
f.pixel_y = 4

if(dir == EAST)
f.pixel_x = 0
f.icon_state = "2"
if(dir == WEST)
f.pixel_x = -16
f.icon_state = "2"

if(dir == NORTH)
f.pixel_y = 16
f.icon_state = "1"
if(dir == SOUTH)
f.pixel_y = 0
f.icon_state = "1"

if(!saved_icons["poison"])
var/icon/i = icon('40. Poison.dmi')
i.Scale(16, 16)
i.Scale(32, 32)
saved_icons["poison"] = i
f.icon = saved_icons["poison"]

if(dir == EAST) animate(f, pixel_x = 40, pixel_y = rand(2, 8), 2)
if(dir == WEST) animate(f, pixel_x = -40, pixel_y = rand(2, 8), 2)

if(dir == NORTH) animate(f, pixel_x = rand(-2, 2), pixel_y = 40, 2)
if(dir == SOUTH) animate(f, pixel_x = rand(-2, 2), pixel_y = -40, 2)

spawn(2) if(f) del(f)
sleep(0.5)

sleep(1)

view() << sound('40. Poison.wav')

var/obj/ex = new(r.loc)
ex.layer = 7
if(!saved_icons["poison"])
var/icon/i = icon('40. Poison.dmi')
i.Scale(16, 16)
i.Scale(32, 32)
saved_icons["poison"] = i
ex.icon = saved_icons["poison"]
ex.icon_state = "3"
ex.alpha = 150
spawn(0.5) del(ex)

r.vibrate()

sleep(3)

animate(r, color = rgb(255, 0, 255), 3)
animate(color = null, 3)

for(var/n = 0, n < 4, n++)
var/obj/f = new(r.loc)
f.layer = 6
f.pixel_x = n * 4 + rand(-16, 16)
f.pixel_y = rand(-16, 0) + 8
if(!saved_icons["poison"])
var/icon/i = icon('40. Poison.dmi')
i.Scale(16, 16)
i.Scale(32, 32)
saved_icons["poison"] = i
f.icon = saved_icons["poison"]
f.icon_state = "4"

animate(f, pixel_y = f.pixel_y + 32, 10)

spawn(11) del(f)

view() << sound('40. Toxic.wav')

sleep(1)

sleep(16)

r.take_damage(move, src)

view() << sound('Damage.wav')
The call to vibrate() would seem to be relevant in that code as well, if that does any animation. There doesn't appear to be any animation that loops forever.

I'm seeing a number of things in that code that could use cleanup, but none that should cause the kind of insane memory growth you're seeing. I recommend doing the cleanup regardless because it's important.

1) You're doing icon math that appears to be completely unnecessary. Most scaling can be done now with transform and PIXEL_SCALE. Because icon math is basically all done on the client now except when the server needs a "live" copy for some reason, this is unlikely to spike your memory. (Much of this code is also needlessly repeated. Since you know you'll need the icon from your list cache anyway, do the cache check at the beginning of the proc, to see if Scale() needs to be called, and then the saved_icons[whatever] will be valid from that point on. You could do one better by handling all that in a separate proc that returns the icon, and just keeping that icon in a local var.)

2) In multiple places, you're calling del() on temporary objects that really would be better served by moving them to null and letting the garbage collector destroy them.

3) In several places you're using "magic number" layers, where you'd be better off using #define to declare the layers you'll use early in your project. Then you can change them up more easily at any time.

4) Those if(dir) lines are awful. At the very least I would recommend using the else command in there, so you're not doing unnecessary if() checks, or a switch() would be a nice change. Better still is to take advantage of bitflags to set the vars you want, but looking at the code I think yours is one of the rare cases that wouldn't work well.

5) When you call view() you should be calling view(src). Remember these are procs, where usr is unreliable, but view() uses usr as its default point of reference. In fact I would suggest the damage sound should be sent to view(r) since that's where the damage occurs, and it should be sent before r.take_damage() is called in case r gets deleted.

It's clear that whatever is eating the memory is not one of the simple things like objs or lists or appearances growing out of control. All of those are well within normal levels.

One thing the memory reporter doesn't currently look at is sleeping procs. Is it possible you have something that's generating a crapload of sleepers, especially something with long sleeps/spawns? You're calling sleep() and spawn() a very great deal in this code, and I would suggest a lot of those calls are unnecessary. (For instance, the last sleep() in each of those for() loops could be put at the beginning of the loop and skipped if !n.) While this specific code doesn't show a problem, it's conceivable that another part of your code is causing procs to sleep some inordinate amount of time and that their memory is what's drowning your project.
Using the status debug thing would give an idea of sleeping procs. On my phone at the moment so I don't have the command handy but I think it's in the DD menu now.
It's "Status Report" in the same menu as your memory stats by the way, it'll output what procs are sleeping and how many instances of said proc are sleeping.
I'm not too well versed in the crash reports. I just encountered this:
Problem signature:
Problem Event Name: APPCRASH
Application Name: dreamdaemon.exe
Application Version: 5.0.512.1470
Application Timestamp: 5ceae321
Fault Module Name: byondcore.dll
Fault Module Version: 5.0.512.1470
Fault Module Timestamp: 5ceae2aa
Exception Code: c0000005
Exception Offset: 000eab47
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 1033
Additional Information 1: 5861
Additional Information 2: 5861822e1919d7c014bbb064c64908b2
Additional Information 3: 3c39
Additional Information 4: 3c390ead1f6678e1205760b69cf3f2fc

I don't think it's a related crash, but you guys are suer helpful and I don't want to likee, fill up the forums like my own personal help desk, so unless you guys feel I should make another thread, I just thought I'd post that here. Back on topic however:



I've collected some status reports.
Connections: [0x5000000] [0x5000001] [0x5000002] [0x5000003] [0x5000004] [0x5000005] [0x5000006] [0x5000007] [0x5000008] [0x5000009] [0x500000a] [0x500000b] [0x500000c] [0x500000d] [0x500000e] [0x500000f] [0x5000010] [0x5000011] [0x5000012] [0x5000013] [0x5000014] [0x5000015] [0x5000016] [0x5000017] [0x5000018] [0x5000019] [0x500001a] [0x500001b] [0x500001c] [0x500001d] [0x500001e] [0x500001f] [0x5000020] [0x5000021] [0x5000022] [0x5000023] [0x5000024] [0x5000025] [0x5000026] [0x5000027] [0x5000028]
Run Time: 285.35s
Sleeping procs:
/datum/elevator_tower/New 6
/obj/overlay/leaves/New 3
/turf/bush/Entered 394
/mob/monster/proc/wild_life 10
/mob/proc/fade_out_xp_bar
/mob/proc/fade_out_hp_bar 2
/mob/monster/proc/take_damage
/mob/monster/proc/large_appearance 8
/mob/proc/area_animation 23
/mob/monster/proc/play_area_music_in 3
/world/New
/mob/player/proc/knockout 4
/mob/NPC/overworld_monster/Dark_Dragon/New
/world/proc/time_cycle
/world/New


This was earlier today. I've since gone in and changed bush which had a sleep(600) proc on it that fired when people walked into a certain object, and I've decreased it to a sleep(60); I'm going to put animations into the icon itself so I can remove that sleep call entirely, but so far this is what it's looking like:

Connections: [0x5000000] [0x5000001] [0x5000002] [0x5000003] [0x5000004] [0x5000005] [0x5000006] [0x5000007] [0x5000008] [0x5000009] [0x500000a] [0x500000b] [0x500000c] [0x500000d] [0x500000e] [0x500000f] [0x5000010] [0x5000011] [0x5000012] [0x5000013] [0x5000014] [0x5000015] [0x5000016] [0x5000017] [0x5000018] [0x5000019] [0x500001a] [0x500001b] [0x500001c] [0x500001d] [0x500001e] [0x500001f] [0x5000020] [0x5000021] [0x5000022] [0x5000023] [0x5000024] [0x5000025] [0x5000026] [0x5000027] [0x5000028] [0x5000029] [0x500002a] [0x500002b] [0x500002c] [0x500002d] [0x500002e] [0x500002f] [0x5000030] [0x5000031] [0x5000032] [0x5000033] [0x5000034] [0x5000035] [0x5000036] [0x5000037] [0x5000038] [0x5000039] [0x500003a] [0x500003b] [0x500003c] [0x500003d] [0x500003e] [0x500003f] [0x5000040] [0x5000041] [0x5000042] [0x5000043]
Run Time: 1647.1s
Sleeping procs:
/mob/monster/proc/wild_life 25
/mob/proc/fade_display_name 11
/datum/elevator_tower/New 6
/client/proc/xalertbox
/obj/title_bg/Click
/turf/bush/Entered 71
/obj/overlay/leaves/New 3
/obj/card_obj/proc/flash_white
/mob/monster/proc/tailwhip
/mob/monster/proc/bite 4
/mob/proc/fade_out_hp_bar 2
/mob/player/proc/sendout_poke
/mob/monster/proc/large_appearance 29
/mob/monster/proc/poisonsting 5
/mob/proc/area_animation 25
/mob/proc/title_screen 4
/client/New
/mob/monster/proc/capture_animation
/mob/verb/Say
/mob/proc/display_hp_bar
/mob/monster/proc/play_area_music_in
/world/New
/proc/progress_loop 2
/mob/NPC/overworld_monster/Dark_Dragon/New
/world/proc/time_cycle
/world/New
/mob/verb/roleplay2 3
/mob/NPC/relearner/Click
/mob/verb/AdminReply
/mob/verb/edit_ic_color


I've taken the server off of its hourly reboot cycle and now I'm trying to see how long it'll take this to become unplayable.


P.S: Thanks for all the help so far ugys and all the support-- and thank you for those tips Lummox, I'll be working on those soon enough.
The number of sleeping procs dosn't look bad, but what's up with the sleeping client/New()? Or for that matter, obj/overlay/leaves/New()? The latter especially looks really odd.
I removed the client/New(), it was related to the smiley library; and the leaves proc is just this

obj/overlay/leaves/New()
spawn(5) del(src)


I'm going to just put modify the .dmi's of the files that need these leaves to include an animation specfically for this so I don't have to use procs with spawns/sleeps for timing.


So far I'm still having the same issue, but I'm tidying things up to make sure it's not an element of my code leading to this memory issue.
Page: 1 2