Not a bug
BYOND Version:513.1511
Operating System:Windows 10 Home 64-bit
Web Browser:Firefox 72.0
Applies to:Dream Seeker
Status: Not a bug

This is not a bug. It may be an incorrect use of syntax or a limitation in the software. For further discussion on the matter, please consult the BYOND forums.
Descriptive Problem Summary:

Numbered Steps to Reproduce Problem:
1.) Have at least two objects with KEEP_TOGETHER appearance flag on screen. In the example below it's obj/a and obj/b.
2.) At least one of those objects has an overlay that uses that object as a render_target in a filter. In the example below it's im created in New.
3.) Possibly wait a while for the bug to appear. In the minimal example attached below it appears with 100% consistency however.

Code Snippet (if applicable) to Reproduce Problem:
appearance_flags = KEEP_TOGETHER
icon = 'icons.dmi'
render_target = "\ref[src]"
var/image/im = image('icons.dmi', icon_state="blue")
im.filters += filter(type="alpha", render_source=render_target)
overlays += im
icon_state = "a"
icon_state = "b"

Expected Results:
obj/a is displayed with an overlay with icon_state "blue" masked to the shape of obj/a. Same with obj/b.

Actual Results:
obj/a is displayed with an overlay with icon_state "blue" masked to the shape of obj/b and not obj/a. obj/b doesn't have any visible overlays

Does the problem occur:
Every time? Or how often? Every time in the example attached above.
In other games? N/A
In other user accounts? Yes.
On other computers? Yes.

When does the problem NOT occur?
Removing the KEEP_TOGETHER flag makes the problem not occur. Displaying image/im not as an overlay but through different means also makes the problem not occur.

Did the problem NOT occur in any earlier versions? If so, what was the last version that worked? (Visit to download old versions for testing.)
I believe it's been occuring since the introduction of render targets.

I guess not using overlays for this stuff but something else, like vis_contents.
Lummox JR resolved issue (Not a bug)
You can't use an object as a both render target and a render source in one of its own filters, or in this case one of its overlays' filters. The behavior is undefined and will never produce what you want.

The problem in this case is that the overlay is part of the main object's KEEP_TOGETHER group. If it had KEEP_APART or a different plane you'd be fine, since it would become a separate graphical entity. However since it is not, here's what the render process looks like:

1) Start rendering obj. It has KEEP_TOGETHER children so it needs a temporary surface. It has a render target that's used somewhere, so we'll save it to a slate and then render that slate to the temp surface.
2) Draw the obj's KEEP_TOGETHER underlays.
3) Draw the icon for the obj.
4) Draw the obj's KEEP_TOGETHER overlays.
4a) One of those overlays has a filter, so it'll need a temporary surface.
4b) Draw the icon on the temporary surface.
4c) Apply the filter, which uses a render source. That render source is the slate we started writing in step 1, so we're not finished with it yet.

Since you want to mask to the shape of the parent object, I believe what you should be using is the new BLEND_INSET_OVERLAY blend mode that was created for exactly this purpose.
Thanks for the quick and extensive reply!