I'm trying to create a faction system of sorts for my roguelike and I'm not quite sure how to go about making it work the way I envisioned it. Here is a scenario:
Skeletons and rats would be their own group. They would not attack each other. Skeletons and rats will attack goblins though.
Guards will attack any sort of monster. Guards will not attack players, unless players attack guards.
Turtles do their own thing. No one would really bother to attack turtles.
Basically I'd need to be able to easily define different groups and add/remove allegiances so groups will attack/ignore each other.
The second, more complicated question is about how to model AI that uses this information. That depends entirely on how complex your game is and how you want to run things. This DreamMakers article contains a discussion about a state-machine based approach to AI that might be worth considering.