Ok, I want to design a tactics battle system, sort of on the lines of FFTA. I have a quite large consern about this though:

First off, I want it to be on-the-map style (IE: A monster is on the map, and when the player's get within view of the monster, depending on if it is friendly or not it will initiate a battle). The problem is, if one person is battling on-the map, another could walk up from behind and randomly join the battle. This could be problamatic in the long run, if the game becomes very popular and the dungeons have 10-20 players in them. If I was to make it a single-player game it wouldn't nearly be as fun, as a party of friends couldn't play together.

I have thought of a couple ways around this, but they seem very impractical to implament. First thing was the client.screen turns into the battle and the player's turn invisible to other players (that is, if they are not in the same adventuring party) and the battle will commence from there. I plan on having flashy graphics for the spells though, which could play out bad if I try to use them on-screen (possible issues: latency, CPU deficiency on the host computer, maybe others). I like this idea better than the next one though:

Another way I thought of would be to have the same dungeon-map on multiple layers, and check if there is a player party already on this layer and if that was true than move to the next layer up. Problem with this is I would have to have an upward of 20+ maps of the same Dunegon, which I plan on being quite large in size. This could practically kill my host computer depending on how many dungeons I implament.

I was wondering if there is a better & more practical way of doing something like this. I know there is probbably a better way out there, but I haven't thought of one yet. I thank you in advance for any help regarding this subject ^.^
You could try making it so once the monster has a target, no other monsters or players could come up to it like a barrier type thing. More specifically, make an 'inbattle" var and if another player is within a couple steps of another playiner in a battle, their attacks are disabled. Just spitballing but...
In response to Tatenen
I could see a couple of concerns with this, one in particular would be that I would have to make maps with many different ways to exit, with many extra passages (which isn't much of a problem, but with multple people in each dungeon the passages could be in use.) And if, for say, another person wants/needs to go down that passage and fight a specific monster, they would have to wait till the other person is finished with that battle.

Another way I thought of afterwards is to make it so only one person/party can access the dungeon at once, but this could cause the same problems as mentioned above, with quests for specific items dropped by a specific creature, or the need/want to kill said creature for themself or a friend.
In response to KingCold999
Ok well that makes your problem a bit more clear. How about making a var for every mission like "mission1unavailable" or shortened..."m1u" and when you speak with the NPC they give you something else to do or just tell you some words of wisdom until the mission is available again.
In response to Tatenen
Thanks for that, that isn't the whole problem but it does help a little bit on one of the problems I would experience. There would still be the problem of random adventurers (players) wondering into a battle they shouldn't or where they would be unwanted.
In response to KingCold999
well like i said before you could just have their abilities disabled if they're within attacking distance. Assuming you haven't coded this part yet it, it wouldn't be too much of a hassle to add in.
In response to Tatenen
I actually have started this. The attacks aren't avaliable until the battle starts. If there is no battle, they don't get the verbs. But the battle begins as soon as a monster (any mob could be a monster, even shopkeepers btw) decides to be offensive to a player/party.

I also have thought of a problem with the "Quest Unavaliable" answer. Since quests are going to be a large portion of this game, the "Disable Quest" might have some problems. What if someone takes days, even weeks, to complete a given quest? The other people can't progress farther into the game without this, yet they can't do the quest utill this person finishes.
Sounds to me like what you need to do is instancing.

I've found that Lummox JR's SwapMaps is one handy library to use to do this.
In response to KingCold999
Off the top my head, Give them a time limit and make alternate quests, make 2 others that complete the same level.