ID:1544357
 
(See the best response by Ter13.)
How would one go about counting images?
First thing that came to mind:

for(var/image/I)

That doesn't work. Why? Who knows.

The catch is that i'm trying to figure out whether BYOND GC's them or not.
That means no adding them to a list upon New(), etc.

Got any ideas?
I'm actually not sure that /images conform to the standard rules of world locality, but part of the reason that the snippet you posted won't work, is that you need a container specifier.

for(var/turf/t in world)


Would by default check for each turf in world.contents.

The same would be true for checking for every /obj/item in a mob's contents:

for(var/obj/item/i in m)


The same would also be true for checking in any other list of your making. The trick here, is the VM interprets some objects you pass as a container specifier as having a contents list, and therefore allows listing the object itself as a shortcut.

Again, though, I'm not sure you can search for each type of /image in world, because they may not be treated like regular atoms as far as contents go.

Give it a shot using a list specifier in your for loop though, and let us know what you come up with. Oh, and as far as I've noticed, images seem to garbage collect fine provided you don't leave handles to them laying around live.
First off, what are you trying to count the images from? For example, if you add an image to a client's screen you can do:

for(var/image/I in src.client.screen)


You need a target to loop through.

To count images, simply make a variable and then add one for each image found.

var/Counter
for(var/image/I in src.client.screen)
Counter ++
world << "Final Count: [Counter]"


If you'd like to count every image in existence, no matter the container, then you'd have to add the image to a central list on creation as well; then loop through that list.

Edit: Curse you Ter, posted while I was writing. :P
In response to Reformist
Reformist wrote:
First off, what are you trying to count the images from? For example, if you add an image to a client's screen you can do:

> for(var/image/I in src.client.screen)
>

You need a target to loop through.

To count images, simply make a variable and then add one for each image found.

> var/Counter
> for(var/image/I in src.client.screen)
> Counter ++
> world << "Final Count: [Counter]"
>

If you'd like to count every image in existence, no matter the container, then you'd have to add the image to a central list on creation as well; then loop through that list.

Edit: Curse you Ter, posted while I was writing. :P

130WPM FTW.

Spot on, though. You've got my vote.
Ter13 wrote:
I'm actually not sure that /images conform to the standard rules of world locality, but part of the reason that the snippet you posted won't work, is that you need a container specifier.

Not necessarily, this:

/mob/verb/count_datums()
var/count = 0
for(var/datum/D) count++
world << "There are [count] datums"
return

works just fine, returns ~1400 even though for(var/datum/D in world) doesn't return anything.
(It's pretty terrible that there's 1400 datums in a 32x32 testworld with 1 mob and 7 objects, but that's an SS13-specific problem)
And "in world" doesn't work on images either, but that's not really surprising considering they're not atoms.


Reformist wrote:
If you'd like to count every image in existence, no matter the container, then you'd have to add the image to a central list on creation as well; then loop through that list.

See OP, already thought of this, but if BYOND has an image GC this would interfere with it by creating refs.

As for the background on this, here's some worrying data from a graphics benchmark:



Images can and will demolish performance if not GC'd properly, so i'd like to be sure.

This is an interesting problem. How are you creating the images that aren't being cleared, may I ask?

I'm not sure this will work, but every object has a specific reference number that you can access via "\ref[object]". It'll give you an id number, so if you manufacture your own references and then locate() by reference number, you might be able to count the number of images available. It will be slow as all get-out, but if you need to count images, that might well be your only way if the for() method doesn't work.
Ter13 wrote:
This is an interesting problem. How are you creating the images that aren't being cleared, may I ask?

Was doing some rough stress tests with procs like the example below.
Noticed it was scaling miserably (1st 1000 images would run in .1s, 5th 1000 would take over 2000s) and thought to investigate further.

/mob/verb/test_images_with_deletion()
var/list/imgs = list()
var/image/img = null
var/init = 0
file("code/modules/graphics/test.txt") << "Begin series - with deletion"
for(var/i=1,i<=25,i++)
init = world.timeofday
for(var/j=1,j<=100,j++)
img = image(icon='icons/test.dmi',loc=usr,icon_state="test",layer=20)
usr << img
animate(img,pixel_x=(rand(-32,32)),pixel_y=(rand(-32,32)),time=20,easing=CIRCULAR_EASING)
imgs += img
file("code/modules/graphics/test.txt") << "[i]: [world.timeofday-init]"
file("code/modules/graphics/test.txt") << "End series - with deletion."
for(var/I in imgs) del(I)
world << "Finished"
return


Ter13 wrote:
I'm not sure this will work, but every object has a specific reference number that you can access via "\ref[object]". It'll give you an id number, so if you manufacture your own references and then locate() by reference number, you might be able to count the number of images available. It will be slow as all get-out, but if you need to count images, that might well be your only way if the for() method doesn't work.

Tried the following bit:
var/global/list/all_images = list()

/image/New()
all_images += "\ref[src]"
return ..()

1. Doesn't work (length of all_images never goes above 0)
2. Crashes Daemon whenever a player join the world.
Happens at the point where it attempts to load their body images (it loads their clothes first with no problems, since they're overlays from icons)
Also, this crash is a new one. The world is unresponsive and frozen, but Daemon doesn't actually crash (throwing "stopped working" on Windows) until I end the world by pressing "stop".

Still stumped.
Yeah, that last bit makes perfect sense /image can't be extended like a normal atom. It's not a normal type you have access to.

/mob/verb/test_images_with_deletion()
var/list/imgs = list()
var/image/img = null
var/init = 0
file("code/modules/graphics/test.txt") << "Begin series - with deletion"
for(var/i=1,i<=25,i++)
init = world.timeofday
for(var/j=1,j<=100,j++)
img = image(icon='icons/test.dmi',loc=usr,icon_state="test",layer=20)
usr << img
animate(img,pixel_x=(rand(-32,32)),pixel_y=(rand(-32,32)),time=20,easing=CIRCULAR_EASING)
imgs += img
file("code/modules/graphics/test.txt") << "[i]: [world.timeofday-init]"
file("code/modules/graphics/test.txt") << "End series - with deletion."
for(var/I in imgs) del(I)
world << "Finished"
return


There are actually a number of factors behind why this is taking so long to run.

For one, imgs is a list declared in a proc. That means it has a static length, not a dynamic length. The bigger that list gets, the more often it has to reallocate itself in memory.

Second, when you delete an object forcibly, it needs to check every other datum for references to itself. The more datums in the world, the longer that takes.

Third, your images aren't going to lose scope and fall off on their own, because there are at least three cases of reference to them here.

1) The images list of clients it has been exposed to.
2) The referenced loc of the image.
3) The list in the proc's scope.

So yeah, it's very obvious that this should be taking continuously longer the more iterations you run, largely due to that list you are hanging on to alone, much less the forcible deletion of the objects.

Try initializing that list with a set length:

var/l[2500]
for(var/iter=0;iter<25;iter++)
for(var/count=0;count<100;count++)
l[iter * 100 + count + 1] = /*image*/


In order for images to be garbage collected naturally, they need to be set to a null location, they need to have all viewers removed, and they need to not be stored in a list anywhere or referenced by anything. You have to cause the reference-count to hit zero.
As long as the image is not at a null loc and is referenced somewhere it doesn't get GC'd.

It should be noted here that overlays don't count as a reference.

When you add an object to the overlays list, it hijacks the appearance of the object, type, or icon you pass into the overlays list.

An appearance is an internal type that stores information about a few visual variables of the atom class.

So, interestingly enough:

var/obj/o = new()
o.icon = 'someicon.dmi'
o.icon_state = "someiconstate"
o.layer = FLOAT_LAYER
src.overlays += o


The above snippet would instantiate an object, then after the scope of the variable: o expires when the function terminates, the object will have zero references. The object will be garbage collected.

In other words, the above snippet is comparatively wasteful compared to this:

obj/someoverlay
icon = 'someicon.dmi'
icon_state = "someiconstate"
layer = FLOAT_LAYER

proc
addOverlay(var/mob/m)
m.overlays += /obj/someoverlay


BYOND's internal management of certain classes is... Interesting to say the least. And for the record, what reformist said about his tests with the datum counter makes a lot of sense... /image objects are not normal objects. I'm not actually sure they sit in the contents list of their location, but rather, internally, I think they are stored in another way. Might have to ask Lummox how they work, because I know they don't comply with standard rules that other atoms abide by.

They've largely been out of compliance since BYOND has existed, much like Client and World not being true datums, for instance.
Ter13 wrote:
There are actually a number of factors behind why this is taking so long to run.
For one, imgs is a list declared in a proc. That means it has a static length, not a dynamic length. The bigger that list gets, the more often it has to reallocate itself in memory.

Interesting, didn't know that. Will keep in mind.

Ter13 wrote:
Third, your images aren't going to lose scope and fall off on their own, because there are at least three cases of reference to them here.
So yeah, it's very obvious that this should be taking continuously longer the more iterations you run, largely due to that list you are hanging on to alone, much less the forcible deletion of the objects.

Yes, this was meant as a benchmark for manual control, not GC.
The point was that letting the image count grow means eating a colossal hit to performance.
So needless to say, I'm not satisfied with "hopefully BYOND GC's images properly, and hopefully we're not leaving hanging refs that interfere with it".

I'd like to be sure things are working right, which means somehow finding a way to count the images.

Ter13 wrote:
BYOND's internal management of certain classes is... Interesting to say the least.
Hehe, actually put up a separate question about appearances a few minutes ago.
Let me see what I can do. I think I can come up with a wrapper strategy.
Best response
var
__gc_loop = 0
__gc_timer = 50
list/__init_images = list()

proc
__image(icon=null,loc=null,icon_state=null,layer=null,dir=null)
var/image/i = image(icon,loc,icon_state,layer,dir)
. = "\ref[i]"

if(. in __init_images)
return i
else
__init_images += .
world << .

if(!__gc_loop)
__gc_loop = 1
spawn(__gc_timer)
__check_image_gc()
return i

__check_image_gc()
set background = 1
set waitfor = 0

var/image/i
var/count = 1

while(count<=__init_images.len)
i = locate(__init_images[count])
if(i)
count++
else
__init_images -= __init_images[count]

if(__init_images.len)
__gc_loop = 1
spawn(__gc_timer)
__check_image_gc()


This is the counting strategy I came up with. Replace all isntances of "image( with "__image(" for temporary testing, and you should be fine.

I should note, however, that I ran some tests to make sure this works. BYOND exceeds my expectations on the subject of garbage collection of images. It's borderline instant, provided you:

1) Remove the image from all stored variables and let all local scopes containing references to it terminate.
2) Set the image's loc to null
3) Remove the image from all clients' images list.

In all honesty, I think your issue is most likely not image counts increasing, but possibly poor memory management and excessive use of manual deallocation of objects using the del keyword.

I saw you mention SS13. SS13 is horribly written. It's improved, yeah, but SS13 has a tendency to manually delete objects by the thousands. In practice, I strongly recommend avoiding the del keyword as much as possible if you want fast, clean code.
The problem is this:
some_list += "\ref[src]"

As far as I can tell, it's freezing Dream Daemon.
Since you got it to work, my guess is it's an SS13-specific problem.

Ter13 wrote:
In all honesty, I think your issue is most likely not image counts increasing, but possibly poor memory management and excessive use of manual deallocation of objects using the del keyword.
I saw you mention SS13. SS13 is horribly written. It's improved, yeah, but SS13 has a tendency to manually delete objects by the thousands. In practice, I strongly recommend avoiding the del keyword as much as possible if you want fast, clean code.

True, in clean code at least. In SS13's mess of spaghetti, it's tough.
Making sure an object has no refs is much easier said than done.

So in the end, it looks like there's not much point going further with this.
Though image counting isn't really feasible, it's not a huge loss.
I doubt there's many hanging references anyway, since 99% of images created are used to make overlays, then left to GC.

On top of that, it looks like there's absolutely zero reason to use them for graphics purposes.
The whole reason i started down this road was because I thought using images would reduce overhead and increase performance.
I just ran the same benchmarks using movable atoms instead.
Night and day.

The performance drop with number of movable atoms is linear, compared to quadratic for images.
Also, it's much flatter. I can push upwards of 25,000 without deletion, and at that point generating and animating 2500 more takes ~ 1 second.
For images, it takes ~ 2.5 minutes at 7500 without deletion.
Finally, 10,000 movable atoms can be del()d in roughly a second.
Trying to del() 10,000 images crashes Daemon.

Bottom line:

With 50,000 movable atoms, procs that run in under 5 seconds, and 100% CPU usage, i'm clearly hitting my laptop's limitations.

With 10,000 images, 20% CPU usage, procs running upwards of 15 minutes, and graphs showing painfully obvious quadratic growth, i'm clearly hitting BYOND's.
Yeah, the sole point of using images is that you have per-client control of who can see them. For any other purpose, use atomic objects, or use prototypes on the subject of overlays.

EDIT:

some_list += "\ref[src]"


Can you show me what you are doing here?

...I don't think you should be using "\ref[src]" for the subject we're speaking about. [src] can literally never, in the context of DM, be a /image object... So... If you are freezing because of that line... You are having another problem.
It's this here:

var/global/list/all_images = list()

/image/New()
all_images += "\ref[src]"
return ..()

First (and only) way that springs to mind to count images.
Creating and counting them doesn't help: it's the ideal case.

I'd like to track the number of images during average gameplay, since that could reveal problems like many not getting GCd due to hanging refs, or just plain-old inefficient code/bad loops creating a few thousand per tick (wouldn't be surprised).
Yeah, you can't do that. You just can't override image/New(). It's not allowed, because it's not an exposed type. The function I showed you that uses a wrapper function around image() is pretty much the only way you are going to pull it off.
You can use a #define macro to seamlessly replace image() with __image() or whatever.
Yeah, the wrapper should work. Will look into this further sometime.
For now, it looks like there's no reason to work with thousands of persistent images instead of movable atoms, so not a high priority.
Spoke too soon. Though at this point, I really shouldn't be surprised.
Turns out, there are quite a few hanging refs to images in SS13 code, so this is definitely is a high priority.

Going to search & destroy, but it'd be nice to know exactly what we're dealing with here.

A wrapper should work in theory, but since the DM preprocessor (afaik) doesn't support variable args, it's not quite that simple.
Ctrl-H is more than willing to do the job, but that feels like using a handsaw because I can't find a Dremel bit.

Any ideas?
Yeah, unfortunately that's about your options. I'm sorry there's not a better solution for your case, but some times with DM, you run into a situation where the best way to do something is to do it right the first time, which doesn't help you in a lot of cases...