I switched over the live NEStalgia servers to 506.1236 with threading on this afternoon. Right away players started reporting issues with weird visual glitches and NPCs who's names or dialogue text was garbled or switched. There were even NPCs who had relocated to maps that they shouldn't be on. Some players ended up invisible and others had their icons replaced by letters/numbers from that same font.dmi file.
Every player was seeing the same thing, so it wasn't a cache problem (several of them cleared their caches and restarted the game just to double check). Most of the issues seemed to be isolated to mobs, though I can't confirm that 100%.
I just restarted with threads off and so far everything looks fine. I'll report back if anything pop ups with threads off.
Here is an example of the bug when talking to an NPC (her icon is replaced by a "4" from the font.dmi file, and the name is now CR2: http://silkgames.com//random/1236_bug.png
Here is what it should look like: http://silkgames.com/random/1236_normal.png
Some examples from other NEStalgia players: http://silkgames.com/forum/viewtopic.php?f=35&t=10095
Also, when I tried to shutdown the largest of the three servers I received a Windows "Dream Daemon has stopped working" error with the following details:
Problem signature:
Problem Event Name: BEX
Application Name: dreamdaemon.exe
Application Version: 5.0.506.1236
Application Timestamp: 530fed9b
Fault Module Name: StackHash_e9ed
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 00000000
Exception Offset: e9009e88
Exception Code: c0000005
Exception Data: 00000008
OS Version: 6.0.6002.2.2.0.272.7
Locale ID: 1033
Additional Information 1: e9ed
Additional Information 2: 576404d65bac9d3a8587162f65ca8aab
Additional Information 3: fbf9
Additional Information 4: 690dd99e998a169cb19443508b68bdd0
It seemed to indicate that the server had been killed by something called "Data Execution Protection" and pointed to an entry about it in the Windows help files.
ID:1511051
 Mar 4 2014, 2:36 pm
|
|||||||||||||
| |||||||||||||
Does this happen with map-threads off?
| |
No, I don't think it does. At least for me this happened with threads on.
| |
The fact that Silk's report mentioned this happens for all players makes me wonder if his issue, at least, is happening without map threads. The fact that mobs seem to be the main target got me looking in a particular area where I found something iffy, so I've made a change for the next beta (506.1237) that I hope will take care of that. Please retest there.
Silk, if you could check that with map threads on and off, that'd be a big help. If it's fixed in both that's of course good news. | |
I'm going to be hosting a test server for NEStalgia in the next couple hours, so I'll try 506.1237 with threads on(and map threads off if necessary) and let you know how it goes.
| |
The new version 506.1237 had the same exact issues with threads on. I just rebooted with "map threads off" and we haven't seen any problems so far, but I'll report back if we do.
| |
Just confirming that there are no issues when "map threads off" is toggled.
| |
Scratch that, the testers encountered the same issues with "map threads off".
| |
Okay, I made a couple major discoveries with threads that will impact the 506.1238 build, so when you get a chance please retest in that build.
- Apparently, there were some situations where server procs could still run on the client thread. This would explain world-wide problems that would occur even without map-threads. - There was a lingering (and difficult to debug!) issue in map-threads that caused wrong Appearances to be shown, and this probably accounted for the majority of cases in NEStalgia (and probably SS13) when map-threads were enabled. This was because Appearances had a non-threadsafe refcount increase, and that led to some premature deletion and recycling. The server still thought the old Appearances were known to certain clients, though, and didn't bother sending them updates. Hence why an NPC could look like a 4, for instance. That could also play into NPC names on dialogue popups being wrong if they're displayed by adding a bunch of individual Appearances onto a main one as overlays, though I'm not sure where I'd look in NEStalgia's code to confirm that. As a followup question for Silk: Did you actually have the hyphen so it was map-threads? The "map-threads" item can't be two separate words, either in the command line argument or the daemon.txt file. If you didn't have the hyphen, then probably everything you saw as far as map oddities is accounted for by my map-threads fix. If I'm right about NPC names, anyway. In any case since I've added fixes outside of map-threads as well, it'd be good to know if the problems are behind us. | |
Hmmm... I'm pretty sure that I did not hyphen "map-threads". Sorry about that, I don't know how I managed to overlook it.
I'll host another test server tomorrow with the new build and report the results back to you after it's been up for a few hours. Thanks! | |
Is this issue supposed to be resolved in 506.1238? The lack of a changelog message makes it hard to tell.
| |
Some threading issues were found and fixed in 506.1238. I suspect they did resolve this, but I can't say so for sure. Sometimes changes get left out of the notes if they don't correspond to a specific bug or feature.
| |
I started up our test server in 506.1238 with threads on a little over an hour ago and everything was going great for awhile. No appearances of the visual bugs, and everyone noticed a marked improvement in server speed and performance.
The server did just randomly freeze up, however. It was similar to the crashes that I experienced with previous builds with multi-threading on (like when I went to go shut down the server before). The version of NEStalgia that we are running is reliable and is not prone to crashing. I'll double check with the testers to see what they were doing before the crash happened. I also just restarted the server, so I'll keep you posted on how things go from here. Here is what the DD looked like: http://silkgames.com/random/dd_crash_1238.png And the error text: Description: A problem caused this program to stop interacting with Windows. Problem signature: Problem Event Name: AppHangB1 Application Name: dreamdaemon.exe Application Version: 5.0.506.1238 Application Timestamp: 531fc9e5 Hang Signature: 8eb0 Hang Type: 0 OS Version: 6.0.6002.2.2.0.272.7 Locale ID: 1033 Additional Hang Signature 1: 19498c1fa43d469c03f90dca7cf98b8d Additional Hang Signature 2: 04a8 Additional Hang Signature 3: af2990e8171a659588a987dcd3859516 Additional Hang Signature 4: 8eb0 Additional Hang Signature 5: 19498c1fa43d469c03f90dca7cf98b8d Additional Hang Signature 6: 04a8 Additional Hang Signature 7: af2990e8171a659588a987dcd3859516 | |
Dang I wish that gave some info on where in the program it hung. If you could break in with a debugger even to just get an offset, that'd be a huge help.
| |
Sure, I can do that the next time that it hangs. I'm not familiar with that process though... can you point me to the right tools/instructions?
You can also remote connect to the server the next time it happens to do that yourself. I think that I send you the latest login info, but I can send it again if that is easier. | |
Just an update: the test server has now been up for six hours with 10~ players online consistently. There have been no new issues reported, and no crashes.
| |
Unfortunately I'm not prone to break into processes with the debugger myself in Windows, so I don't have anything to offer in the way of instructions. Usually if DS/DD has a hang problem I can just run it in the debugger itself till the problem occurs, then hit pause and hope it leads me to the offending code. I know this is doable with third-party debuggers and probably some Microsoft tools; I've just never had to look into it personally.
| |
I've had the game crash on me a couple different times when I've left the DS open for awhile during my personal testing sessions, i.e. when nothing apart from the background procs were running on the server. This only happens with threads on, and I haven't tried "map-threads off" yet.
I haven't had time (and probably won't have time in the near future) to do the proper research to download and use a Windows debugger. If you want to host the next NEStalgia closed test server on one of your Linux boxes then let me know and I'll send you the host files when the time comes. | |
http://i60.tinypic.com/2vja5gj.png
We did not have a crash, though, and I am not sure about the specifics of this, but it may have something to do with the new mapping updating?