var/r_n = rand(1,255)

var/list/new_c = list(0.1,0,0, 0,0.1,0, 0,0,0.1, 1,1,1)
M.color += new_c
M.color -= new_c

Problem description:

So I've never worked with color matrixes before but from reading quite a few posts and reference files, they seem to be the only way to efficiently do what I'm trying to on this code.

What I'm TRYING to do is, every time this runs, fade the color of mob M's icon slightly toward white with it getting progressively whiter each time it's run.

And after approximately 10 seconds, undoing 1 step of this fade toward white.

What I'm getting right now is when this proc runs, the target's icon just turns completely white and when the spawn(100) line runs, nothing happens.

To be perfectly honest, I'm kinda doing this a little blind, the reference page for color matrixes makes my head hurt just by looking at it, it shows a few examples of the format of the code but not examples of how those formats actually function. and it's very difficult to search the forum these days since the search times out pretty much every time I try, so I'm relying on the posts I can find while searching on google.

Does anyone have any experience working with these that could point me in the right direction?
The += and -= operators won't work with color matrices, because you have to remember when atom.color is a matrix, it's read as a list. (When it's not a matrix, it's an RGB or RGBA string.) You can't add and subtract to a color matrix like you can with a transform matrix, since this is a list. Also, adding and subtracting a value like 0.1 tends to produce increasingly inaccurate results anyway.

The matrix you have in your post is multiplying all RGB values by 0.1 and then adding 1 to each (full white), so that won't do what you want. Instead I'd multiply each color by the amount of non-whiteness you want (0.9) and then add the amount of whiteness (0.1). The matrix would look like:

list(0.9,0,0, 0,0.9,0, 0,0,0.9, 0.1,0.1,0.1)

Far easier than trying to add and subtract stuff, the brighten-and-later-undo thing you're trying to do would be better served by using a var to keep track of the number of "steps" toward white you've gone, and then recalculating the matrix whenever that number of steps changes.

mob/var/tmp/brightens = 0

var c = 0.9 ** brightens
// another way to go is max(0, 1-brightens*0.1)
return list(c,0,0, 0,c,0, 0,0,c, 1-c,1-c,1-c)

color = BrightnessMatrix(++brightens)
color = BrightnessMatrix(--brightens)

There are probably other approaches you could take, like having a list of colors if you also want to temporarily redden, golden, darken, etc. Alternatively, you could also do a list of matrices and multiply them all together.