The first error indicates that you updated your game's resources but failed to upload the updated rsc file (added a single resource from the looks of it). The other errors I'm not sure of, but they could both be related to failed resource loading.
| |
I actually have a huge issue with the refcount issue. I'm not sure if it causes problems, but my log is littered with it:
BUG: Finished erasure with refcount=1 (ref=2:6595) DM (Save.dm:25) BUG: Finished erasure with refcount=1 (ref=2:6597) DM (Save.dm:25) BUG: Finished erasure with refcount=1 (ref=2:6601) DM (Save.dm:25) BUG: Finished erasure with refcount=1 (ref=2:6589) DM (Save.dm:25) BUG: Finished erasure with refcount=81036 (ref=3:310) DM (Save.dm:25) BUG: Finished erasure with refcount=1532 (ref=3:331) DM (Save.dm:25) BUG: Finished erasure with refcount=71836 (ref=3:332) DM (Save.dm:25) BUG: Finished erasure with refcount=2493 (ref=3:328) DM (Save.dm:25) BUG: Finished erasure with refcount=22058 (ref=3:330) DM (Save.dm:25) BUG: Finished erasure with refcount=24620 (ref=3:333) DM (Save.dm:25) BUG: Finished erasure with refcount=30783 (ref=3:329) DM (Save.dm:25) BUG: Finished erasure with refcount=49 (ref=3:330) DM (Save.dm:25) BUG: Finished erasure with refcount=541 (ref=3:327) DM (Save.dm:25) BUG: Finished erasure with refcount=1122 (ref=3:330) DM (Save.dm:25) That's like 1/100's of one of my logs with the world running for maybe a day. I take severe advantage of the garbage collector and other memory abuse to keep my cpu usage to a minimum. So, I can only assume it's related to that, if this gets resolved, or if there's a known reason for it, that would help me out. I can try a version (though I won't be able to try it for a while, due to my entry into a contest) that will lower my mem usage, and see if there's a comparable difference. | |
I need a reliable test case on those refcount issues, something that causes it every time or most of the time that can be easily done again and again for testing.
This refcount issue isn't the same as the one mentioned earlier; that was clients, and this is objs and mobs. It could however be related. | |
*** glibc detected *** DreamDaemon: malloc(): smallbin double linked list corrupted: 0x0d925e40 ***
======= Backtrace: ========= /lib/libc.so.6[0xb776fd96] /lib/libc.so.6(__libc_malloc+0x67)[0xb7770fb7] /lib/libc.so.6(__strdup+0x30)[0xb77751a0] /usr/local/lib/libbyond.so[0xb7dabdf5] /usr/local/lib/libbyond.so[0xb7e0bd24] /usr/local/lib/libbyond.so(_Z15ExecProc_Attach5ValuehthS_PS_m PFvS_PvES1_+0x10d)[0xb7e235ed] /usr/local/lib/libbyond.so(_Z29CallByNameWithCallback_Attach5 ValuehmS_PS_mPFvS_PvES1_+0xb2)[0xb7e236f2] /usr/local/lib/libbyond.so[0xb7e24a5d] /usr/local/lib/libbyond.so[0xb7e155b7] /usr/local/lib/libbyond.so(_Z15ExecProc_Attach5ValuehthS_PS_m PFvS_PvES1_+0x10d)[0xb7e235ed] /usr/local/lib/libbyond.so(_Z29CallByNameWithCallback_Attach5 ValuehmS_PS_mPFvS_PvES1_+0xb2)[0xb7e236f2] /usr/local/lib/libbyond.so[0xb7e24bc5] /usr/local/lib/libbyond.so[0xb7e155b7] /usr/local/lib/libbyond.so(_Z15ExecProc_Attach5ValuehthS_PS_m PFvS_PvES1_+0x10d)[0xb7e235ed] /usr/local/lib/libbyond.so(_Z29CallByNameWithCallback_Attach5 ValuehmS_PS_mPFvS_PvES1_+0xb2)[0xb7e236f2] /usr/local/lib/libbyond.so[0xb7e24bc5] /usr/local/lib/libbyond.so[0xb7e155b7] /usr/local/lib/libbyond.so[0xb7e072af] /usr/local/lib/libbyond.so[0xb7e21fc9] /usr/local/lib/libbyond.so(_Z8TickProcl+0x125)[0xb7e22485] /usr/local/lib/libbyond.so[0xb7de943c] /usr/local/lib/libbyond.so(_ZN7TimeLib11SystemAlarmEv+0x177)[ 0xb7eb1ab7] /usr/local/lib/libbyond.so(_ZN9SocketLib15WaitForSocketIOElh+ 0x26a)[0xb7e843fa] DreamDaemon[0x804a3ee] /lib/libc.so.6(__libc_start_main+0xdc)[0xb771ae9c] DreamDaemon(__gxx_personality_v0+0x3cd)[0x8049ed1] ======= Memory map: ======== 08048000-0804d000 r-xp 00000000 08:06 27484173 /usr/local/byond/bin/DreamDaemon 0804d000-0804e000 rw-p 00005000 08:06 27484173 /usr/local/byond/bin/DreamDaemon 08974000-13f50000 rw-p 08974000 00:00 0 [heap] b70ea000-b70ed000 rw-p b70ea000 00:00 0 b70ee000-b70fe000 r-xp 00000000 08:06 27247415 /lib/libresolv-2.5.so b70fe000-b70ff000 r--p 0000f000 08:06 27247415 /lib/libresolv-2.5.so b70ff000-b7100000 rw-p 00010000 08:06 27247415 /lib/libresolv-2.5.so b7100000-b7102000 rw-p b7100000 00:00 0 b7102000-b7106000 r-xp 00000000 08:06 27247413 /lib/libnss_dns-2.5.so b7106000-b7107000 r--p 00003000 08:06 27247413 /lib/libnss_dns-2.5.so b7107000-b7108000 rw-p 00004000 08:06 27247413 /lib/libnss_dns-2.5.so b7108000-b7112000 r-xp 00000000 08:06 27247491 /lib/libnss_files-2.5.so b7112000-b7113000 r--p 00009000 08:06 27247491 /lib/libnss_files-2.5.so b7113000-b7114000 rw-p 0000a000 08:06 27247491 /lib/libnss_files-2.5.so b7116000-b7118000 rw-p b7116000 00:00 0 b7500000-b7521000 rw-p b7500000 00:00 0 b7521000-b7600000 ---p b7521000 00:00 0 b76c2000-b76c4000 rw-p b76c2000 00:00 0 b76c5000-b76c7000 rw-p b76c5000 00:00 0 b76c9000-b76ca000 rw-p b76c9000 00:00 0 b76d0000-b76d1000 rw-p b76d0000 00:00 0 b76d6000-b76d7000 rw-p b76d6000 00:00 0 b76da000-b76db000 rw-p b76da000 00:00 0 b76e4000-b76e5000 rw-p b76e4000 00:00 0 b76e8000-b76ff000 rw-p b76e8000 00:00 0 b76ff000-b7702000 r-xp 00000000 08:06 27247526 /lib/libdl-2.5.so b7702000-b7703000 r--p 00002000 08:06 27247526 /lib/libdl-2.5.so b7703000-b7704000 rw-p 00003000 08:06 27247526 /lib/libdl-2.5.so b7704000-b7705000 rw-p b7704000 00:00 0 b7705000-b7858000 r-xp 00000000 08:06 27247466 /lib/libc-2.5.so b7858000-b785a000 r--p 00153000 08:06 27247466 /lib/libc-2.5.so b785a000-b785b000 rw-p 00155000 08:06 27247466 /lib/libc-2.5.so b785b000-b785e000 rw-p b785b000 00:00 0 b785e000-b7869000 r-xp 00000000 08:06 27247499 /lib/libgcc_s-4.1.2-20080825.so.1 b7869000-b786a000 rw-p 0000a000 08:06 27247499 /lib/libgcc_s-4.1.2-20080825.so.1 b786a000-b7891000 r-xp 00000000 08:06 27247449 /lib/libm-2.5.so b7891000-b7892000 r--p 00026000 08:06 27247449 /lib/libm-2.5.so b7892000-b7893000 rw-p 00027000 08:06 27247449 /lib/libm-2.5.so b7893000-b7971000 r-xp 00000000 08:06 27274372 /usr/lib/libstdc++.so.6.0.8 b7971000-b7974000 r--p 000dd000 08:06 27274372 /usr/lib/libstdc++.so.6.0.8 b7974000-b7976000 rw-p 000e0000 08:06 27274372 /usr/lib/libstdc++.so.6.0.8 b7976000-b797c000 rw-p b7976000 00:00 0 b797c000-b7b8b000 r-xp 00000000 08:06 27484171 /usr/local/byond/bin/libext.so b7b8b000-b7b8c000 rw-p 0020e000 08:06 27484171 /usr/local/byond/bin/libext.so b7b8c000-b7f92000 r-xp 00000000 08:06 27484175 /usr/local/byond/bin/libbyond.so b7f92000-b7f93000 rw-p 00406000 08:06 27484175 /usr/local/byond/bin/libbyond.so b7f93000-b7faf000 rw-p b7f93000 00:00 0 b7faf000-b7fb0000 rw-p b7faf000 00:00 0 b7fb0000-b7fcb000 r-xp 00000000 08:06 27247520 /lib/ld-2.5.so b7fcb000-b7fcc000 r--p 0001a000 08:06 27247520 /lib/ld-2.5.so b7fcc000-b7fcd000 rw-p 0001b000 08:06 27247520 /lib/ld-2.5.so bf945000-bf9e3000 rw-p 7ffffff60000 00:00 0 [stack] I'm lost, this is so weird. It doesn't crash on Windows. | |
Does Windows run on same system?
There's chance that some drivers corrupt memory or your RAM is dying. | |
The stack trace happened on every VPS I ever bought, the only though they did all run CentOS 5.7, I tried 6 but that crashed even faster. I'm now hosting it from my new laptop which doesn't seem to crash.
| |
Possibly memory management bug, or they have identical hardware. Can't be just coincidence.
| |
Zaoshi wrote:
Possibly memory management bug, or they have identical hardware. Can't be just coincidence. I think it would be beneficial to keep this topic on-track. And without meaning any offense, you're suggesting things that don't add to the thread in any way. If he's run the world on a variety of different machines, then the chance of faulty hardware is extremely, extremely unlikely. At the same time, whether they have identical hardware or not is likely irrelevant, and whether Windows runs in another virtualized instance or not has no influence at all. Darker Legends: If you profile your game, can you do so at increments and then post the one you did last, before it crashed? Oh, and if you plan to post long snippets, would you mind doing so by going to f.ex www.pastie.org and pasting it in there? Then you can link to it. On pastie.org, you'll want to select the blank text entry under Language. That way its easier to keep this conversation on-track, without huge stack dumps in the middle :) I do also wonder if you have luck with the game not crashing on a non-CentOS distribution; I haven't used CentOS much myself, but it tends to have a very conservative view on package updates. Does it crash with no players on? I'm sure one of us has a server we can run it on, in that case, to check. | |
The stack dumps on this issue have already been established as useless, because the memory corruption happens at some point before the crash. To be clear, the first one was helpful, because it showed that this is a heap corruption problem. But the crash traces are incapable of showing the cause of that corruption. A tool like Valgrind would be needed to intercept the corruption as it happens.
My assumption is that there is a legitimate BYOND bug causing this, and it's not a hardware issue. That said, it's difficult to rule hardware out completely. I've suspected for some time that some bugs are more prevalent in Linux environments than in Windows and vice-versa (accounting for interface bugs which are always Windows). | |
I seriously do not understand how Valgrind works, does this mean the issue will just be ignored?
| |
Darker Legends wrote:
I seriously do not understand how Valgrind works, does this mean the issue will just be ignored? There are quite a few easy-to-use tutorials on Valgrind. You don't necessarily need to understand the output, just show it. From http://cs.ecs.baylor.edu/~donahoo/tools/valgrind/, try running: valgrind -v --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes DreamDaemon your-game.dmb >& ./out.log That should start your game as usual in DD. Connect to it, play, etc, until it crashes. When it does, check the out.log file, post it on pastie.org and return here with a link to it. | |
Darker Legends wrote:
I seriously do not understand how Valgrind works, does this mean the issue will just be ignored? What I'm basically saying is, we have no data. There is nothing that can be done about the issue till we have it. The first crash trace you provided showed that the heap is getting corrupted. The subsequent traces have showed the same thing, so it's not giving me any new information to act on. The cause of heap corruption is more or less impossible to suss out with a stack trace alone, because the crash is happening sometime after the corruption. Ultimately the only way to move forward on this issue is to use a tool that can watch for the heap corruption when it happens, which will provide a smoking gun. The only other way this could get fixed would be if the cause was discovered in the course of fixing a different bug or just randomly stumbling across the issue while looking at the code. Such things are unlikely but possible, though it would probably be difficult to confirm it was the same issue. | |
Darker Legends:
If you provide me with the .dmb and .rsc and whatever else is needed, I will run the game under Valgrind and host it for you. However, you'll need to be active on the game yourself, or do whatever it is necessary to cause the crash, because I don't have the time to look into that. | |
That's the catch, it randomly crashes. I have no idea what triggers it. I could give you root access on my VPS if you can install Valgrind for me.
| |
Darker Legends wrote:
That's the catch, it randomly crashes. I have no idea what triggers it. I could give you root access on my VPS if you can install Valgrind for me. You haven't even installed Valgrind? Its an RPM package on CentOS. Read Yum how-to. You'll want to first search for valgrind (yum list valgrind) and then install the appropriate package. Then run valgrind as I've already posted here.. I'm not going to hop onto your VPS to do things for you, sorry :) You've been given all of the tools to check this out yourself. Steps: 1) Read the above yum tutorial to learn how to install packages on CentOS. 2) Install the valgrind package. 3) Run valgrind as I've posted above 4) Get the game to crash 5) Post the valgrind output at pastie.org and link to it here. | |
I have installed Valgrind but the command you gave doesn't run DreamDaemon.
| |
Darker Legends wrote:
I have installed Valgrind but the command you gave doesn't run DreamDaemon. So if you were me, and you needed to help you figure out what was wrong, do you think the error description you just gave me is sufficient to do so? What does typing what I wrote give, exactly? You of course need to be in the directory of the .dmb file, and you need to replace your-game.dmb with the name of the dmb file. | |
I am in the directory and I did replace the name, it's just that nothing happens. EDIT I got it fixed, apparently BYOND refused to load some library.
| |
BUG: Finished erasure with refcount=1 (ref=5:2) DM (:0)
BUG: Sequence number 9727 expected but 2 received