So from my experience watching them add stuff to EverQuest, and from thinking about my use of savefiles, I've come to the conclusion that binary data files are a wonderful curse for a developer.

They are wonderful because they are compact and fast.

They are a curse because they make iterative development very difficult.

Consider this common scenario:

Players are playing version A of the game.

Developer begins work on version B, which changes how the spell system works and tweaks some stats around.

In the meantime, the players find some things that really need to be fixed in version A.

But the developer won't be ready with version B for three more weeks, and these fixes to A need to be done Right Now so the game isn't ruined.

So the developer decides to take a copy of A and tweak it, which will create a version C. Then he realizes that he will need to integrate any changes he makes into version B, and because the files storing spell data and stats are binaries, he can't merge them.

Not wanting to do the work twice, the developer decides to put the needed changes into version B, turn off the "new stuff" for now, and release it to players.

And of course a whole bunch of stuff from version B that was supposed to be turned off accidentally slips in, and some of it the developer had no choice but to put out even though it wasn't ready yet...

The result: Regular releases of revs of the game that are filled with unintentional stuff and not ready for prime time, because of the difficulties in having multiple versions of binary data.

I see this over and over and over again...

Some of it can be avoided by modular design of game components. If you work ahead of time to make sure all game systems are autonomous from other systems, you can swap in or remove a new spell system with much less difficulty.

But you still often run into the problem of wanting to have two versions of binaries, or to be able to merge binaries...

Anyway I'm lead to two conclusions:

1) It's really really important to make your game modular so you can plug-in and take out pieces without screwing up the rest of the game.

2) If at all possible, the source files should not be binary. The source should be text, and the binary files should be derived from the text. That way, you can use standard file merging tools to combine data from two releases. In addition, you avoid the large number of problems related to trying to put binary files under versioning and source control.

I wanted to throw this out there to see if Dantom could think of any clever ways for a developer to have their binary savefiles be derived from text-based files...
This is a pretty weighty topic that deserves more than this little response, but I'll give you some input. I'm pretty much in agreement with your conclusions about savefiles, namely that they should be modular and easily-editable. Unfortunately, as you've noted, this is often in conflict with the other big criteria that savefiles be efficient. The .dmp file is a perfect example of this: its text readibility has saved me (and others) numerous times from headache, but on the other hand it is kind of a pain to merge into running programs because it is so inefficient to parse.

I think, though, that the DM /savefile format provides a pretty nice compromise. If you take full advantage of the directory structure (storing each variable in its own subdirectory) you can pretty much organize it like a text file without forsaking the storage efficiency. Consider a simple mob save:


Now suppose you decide to remove the "strength" var. In a literal binary storage system, this would be disastrous, because your format would completely change. But in this organization no damage is done, aside from the fact that you may have an obsolete category sitting in the savefile. Similarly, if you decide to later add a new stored mob var, the old savefiles will be backwards-compatible, since the categories are read in individually.

I guess the point I'm trying to make here is that you can think of savefiles as text files that are organized as in the above example. Although you can't literally read them as text, you essentially can treat them like this when you are parsing the files from DM. This enables you to make functions to merge and edit savefiles as if you had an editor.

Hopefully I'm addressing your concerns somewhat. I am feeling somewhat babbley so I'll stop now :)