(See the best response by Spunky_Girl.)
Pretty simple question: is it possible to use one image as an alpha mask of another? With the stipulations that: 1) No /icon objects are used, 2) Both images have opaque and transparent pixels.

Here's basically what I'm trying to do, in psuedo-code:
. = ..()
plane = assignAnUnusedPlane()
var effectGraphic = 'some big graphic that fills the screen'
var effectMask = 'some small graphic like a circle or star'
effectGraphic.plane = plane
effectMask.plane = plane
applyMask(effectGraphic, effectMask)
animate(effectMask, transform = scaleX5, time = fast)

The desired effect is that the graphic would appear at the center of the screen as a small circle or star, and then quickly spread out to the full screen.

I'm currently able to achieve this, except it suffers from one of several defects: 1) I can use BLEND_ADD which looks decent, but the effect becomes translucent instead of the hard BLEND_OVERLAY that I want. 2) If the effectGraphic has no transparent pixels, I can make it work perfectly; otherwise, compositing leaves behind opaque white.

Any ideas?
Basically, you need to have the alpha mask be the base and BLEND_MULTIPLY the real "base" texture onto that. I just so happen to have already done this so here you go. Yeah it's a lot of hoops to jump through but hey, at least it works. Replace the lines with //whatever comments to whatever you need.

    var/obj/debuggery = new(loc) //whatever you want to apply this to, for demo purposes we're making a new object

var/image/blank = new()

var/mutable_appearance/together = new(blank)
together.appearance_flags = KEEP_TOGETHER
together.blend_mode = BLEND_OVERLAY

var/image/alpha_mask = image() //whatever
alpha_mask.color = list(0,0,0,1, 0,0,0,0, 0,0,0,0, 0,0,0,0, 1,1,1,0)
alpha_mask.appearance_flags = KEEP_TOGETHER
var/mutable_appearance/texture = new(src) //whatever, for demo purposes I use my mob's texture but it could be anything
texture.icon_state = TEXTURE_TO_MAP //whatever, change the icon too if you need to
texture.blend_mode = BLEND_MULTIPLY
texture.color = list(0,0,0,0, 0,0,0,0, 0,0,0,0, 1,0,0,0, 0,0,0,1)
alpha_mask.overlays += texture
texture.icon_state = icon_state
alpha_mask.overlays += texture
together.overlays += alpha_mask
texture.color = list(1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,0,0, 0,0,0,1)
together.overlays += texture

debuggery.overlays += together

Thanks for the reply! I tried this as-is, and the result was an all white graphic, though the alpha masking part did work. Am I seeing something different from what you're seeing?
I was able to get your code working after correcting a small error. The problem is that it uses overlays to compose the final image, so the individual components can't be animated afterward. I tried to edit it to use planes, but I still can't separate one image/overlay into objs in a plane. Anyone have any ideas?

// Create Effect Plane, Alpha Mask, and Texture
var /obj/effect = new(locate(8,8,1))
effect.plane = 10
effect.appearance_flags = KEEP_TOGETHER | PLANE_MASTER
effect.blend_mode = BLEND_OVERLAY
var /obj/alphaMask = new(locate(8,8,1))
alphaMask.icon = 'alpha_mask.dmi'
alphaMask.plane = 10
var /obj/texture = new(locate(8,8,1))
texture.icon = 'texture.dmi'
texture.plane = 10
// Setup Alpha mask from graphic, using RED as mask color.
alphaMask.color = list(0,0,0,1, 0,0,0,0, 0,0,0,0, 0,0,0,0, 1,1,1,0)
alphaMask.appearance_flags = KEEP_TOGETHER
// Binary OR the texture's alpha with the alpha mask to get final alpha channel
texture.blend_mode = BLEND_MULTIPLY
texture.color = list(0,0,0,0, 0,0,0,0, 0,0,0,0, 1,0,0,0, 0,0,0,1)
alphaMask.overlays.Add(texture) // Problem Here
// Reset Color info on texture
texture.color = null
// Multiply Texture over alpha mask
alphaMask.layer = texture.layer - 1
In response to IainPeregrine
Best response
I typically only interact with the overlays/underlays lists by only adding/removing to/from them. Any other interaction, I do through secondary "overlay"/"underlay" lists.
addLay(var/atom/a, var/under=0, var/temp=0)
if(!a) return
//basically, add to one of the 4 lists based upon the arguments passed into this proc
removeLay(var/atom/a, var/under=0, var/temp=0)
if(!a) return
//do the reverse of addLay()
src.overlays = null
src.underlays = null
for(var/atom/a in src.t_over) src.overlays += a
for(var/atom/a in src.overs) src.overlays += a
//etc for the underlays

This code is purely just for proof of concept and not intended for copypasta.

So for what you're asking, you'd just interact with the secondary over/under lays lists, then call reloadLays(). I, personally, do the "t_over" and "t_under" tmp var lists because some VFX you'd never want saved to the player. It's not 100% necessary of course.
Thank you for the reply. After working with MisterPerson's code I think I have a better understanding of the constraints of my project, and any solution with underlays or overlays isn't going to work.

What I'm actually trying to achieve is applying an alpha mask to an entire plane. The effects on the plane consist of independent objects moving around and transforming on their own, and I want to restrict what portion of that the player sees using an alpha mask. Seeing as how this question was technically answered, I'll probably make another topic with a more specific question.
In response to IainPeregrine
Ah, sorry. I guess I misunderstood what the question was.