Descriptive Problem Summary:
Ever since we applied the latest patch (https://github.com/VOREStation/VOREStation/pull/6274) to the server, it has been crashing regularly at the end of every round, after round shut down during world restart (using the built in BYOND world restart). We have done some hot fixes and reverts in an attempt to isolate the problem to no avail. Memory usage was ruled out as DD only uses about 1.2 GB of memory.
Previously running 512.1484, but updated too 512.1488 in an early attempt to fix the issues.
Numbered Steps to Reproduce Problem:
1: Run a full round
2: Have transfer happen
3: Wait until the server attempts to reboot to start another round
Expected Results:
DD to not crash when the server restarts.
Actual Results:
DD crashes after every restart.
Does the problem occur:
Every time the server attempts to restart.
Current Github Version:
https://github.com/VOREStation/VOREStation
Commit: 051cd4028c7ec596a11e1906ad0efb5b3df0b030
Logs:
World rebooted at 2019-12-11T22:10:57
BUG: Crashing due to an illegal operation!
Backtrace for BYOND 512.1488 on Linux:
Generated at Wed Dec 11 22:11:19 2019
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bd64]
libbyond.so [0xf711f000, 0x0], 0x24dc98
[0xf76fd000, 0xf76fdbd0], [0xf76fd000, 0xf76fdbd0]
libbyond.so [0xf711f000, 0x0], 0x24dc98
libbyond.so [0xf711f000, 0x0], 0x271f7f
libbyond.so [0xf711f000, 0x0], 0x27395e
libbyond.so [0xf711f000, 0x0], 0x273cf3
libbyond.so 0x3ab120, 0x3ab23a
libbyond.so 0x36cd80, 0x36cf82
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804af94]
libc.so.6 0x18540, 0x18637 (__libc_start_main)
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a7d1]
Recent proc calls:
/datum/species/proc/handle_logout_special
/proc/key_name
/proc/log_access_out
/mob/proc/update_client_z
/datum/controller/subsystem/nanoui/proc/close_user_uis
/datum/controller/subsystem/nanoui/proc/user_logout
/mob/Logout
/mob/living/Logout
/mob/living/carbon/human/Logout
/datum/admin_help_tickets/proc/ClientLogout
/client/Del
/datum/species/proc/handle_logout_special
/proc/key_name
/proc/log_access_out
/mob/proc/update_client_z
/datum/controller/subsystem/nanoui/proc/close_user_uis
To help the BYOND developers debug this, please send the above trace as part
of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1
--- Memory Map Start ---
14859: DreamDaemon /opt/vorestation/source vorestation.dmb -invisible -threads on -map-threads on -port 2303 -trusted
0000000008048000 36K r-x-- DreamDaemon
0000000008051000 4K rw--- DreamDaemon
0000000009f3f000 49108K rw--- [ anon ]
00000000f5975000 1324K rw--- [ anon ]
00000000f5bca000 628K rw--- [ anon ]
00000000f5c67000 352K rw--- [ anon ]
00000000f5cf5000 172K rw--- [ anon ]
00000000f5e49000 776K rw--- [ anon ]
00000000f5f4c000 2944K rw--- [ anon ]
00000000f62c0000 8640K rw--- [ anon ]
00000000f6b30000 12K r-x-- libdl-2.23.so
00000000f6b33000 4K r---- libdl-2.23.so
00000000f6b34000 4K rw--- libdl-2.23.so
00000000f6b35000 1728K r-x-- libc-2.23.so
00000000f6ce5000 8K r---- libc-2.23.so
00000000f6ce7000 4K rw--- libc-2.23.so
00000000f6ce8000 12K rw--- [ anon ]
00000000f6ceb000 100K r-x-- libpthread-2.23.so
00000000f6d04000 4K r---- libpthread-2.23.so
00000000f6d05000 4K rw--- libpthread-2.23.so
00000000f6d06000 8K rw--- [ anon ]
00000000f6d08000 112K r-x-- libgcc_s.so.1
00000000f6d24000 4K rw--- libgcc_s.so.1
00000000f6d25000 332K r-x-- libm-2.23.so
00000000f6d78000 4K r---- libm-2.23.so
00000000f6d79000 4K rw--- libm-2.23.so
00000000f6d7a000 4K rw--- [ anon ]
00000000f6d7b000 1460K r-x-- libstdc++.so.6.0.21
00000000f6ee8000 4K ----- libstdc++.so.6.0.21
00000000f6ee9000 20K r---- libstdc++.so.6.0.21
00000000f6eee000 4K rw--- libstdc++.so.6.0.21
00000000f6eef000 12K rw--- [ anon ]
00000000f6ef2000 28K r-x-- librt-2.23.so
00000000f6ef9000 4K r---- librt-2.23.so
00000000f6efa000 4K rw--- librt-2.23.so
00000000f6f0a000 2128K r-x-- libext.so
00000000f711e000 4K rw--- libext.so
00000000f711f000 5884K r-x-- libbyond.so
00000000f76de000 8K rw--- libbyond.so
00000000f76e0000 104K rw--- [ anon ]
00000000f76fa000 12K r---- [ anon ]
00000000f76fd000 4K r-x-- [ anon ]
00000000f76fe000 140K r-x-- ld-2.23.so
00000000f7721000 4K r---- ld-2.23.so
00000000f7722000 4K rw--- ld-2.23.so
00000000ffb6d000 132K rw--- [ stack ]
total 76292K
--- Memory Map End ---
ID:2528945
 Dec 11 2019, 7:58 pm (Edited on Dec 11 2019, 8:23 pm)
|
|||||||||||||
| |||||||||||||
We have not attempted to run the server on 513, but we are currently updating the code to be 513 compliant and will try again.
| |
Still crashes on the latest 513.
World rebooted at 2019-12-13T05:44:23 BUG: Crashing due to an illegal operation! Backtrace for BYOND 513.1502 on Linux: Generated at Fri Dec 13 05:44:45 2019 DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bd94] libbyond.so [0xf71a1000, 0x0], 0x252970 [0xf77a6000, 0xf77a6b90], [0xf77a6000, 0xf77a6b90] libbyond.so [0xf71a1000, 0x0], 0x252970 libbyond.so [0xf71a1000, 0x0], 0x27ff6f libbyond.so [0xf71a1000, 0x0], 0x28195e libbyond.so [0xf71a1000, 0x0], 0x281cf3 libbyond.so 0x3c1c40, 0x3c1d5a libbyond.so 0x37e8d0, 0x37ead2 DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804afc4] libc.so.6 0x18540, 0x18637 (__libc_start_main) DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a801] Recent proc calls: /datum/species/proc/handle_logout_special /proc/key_name /proc/log_access_out /mob/proc/update_client_z /datum/controller/subsystem/nanoui/proc/close_user_uis /datum/controller/subsystem/nanoui/proc/user_logout /mob/Logout /mob/living/Logout /mob/living/carbon/human/Logout /datum/admin_help_tickets/proc/ClientLogout /client/Del /datum/species/proc/handle_logout_special /proc/key_name /proc/log_access_out /mob/proc/update_client_z /datum/controller/subsystem/nanoui/proc/close_user_uis To help the BYOND developers debug this, please send the above trace as part of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1 --- Memory Map Start --- 24682: DreamDaemon /opt/vorestation/source/vorestation.dmb -invisible -threads on -map-threads on -port 2303 -trusted 0000000008048000 36K r-x-- DreamDaemon 0000000008051000 4K rw--- DreamDaemon 00000000f6baf000 12K rw--- [ anon ] 00000000f6bb2000 12K r-x-- libdl-2.23.so 00000000f6bb5000 4K r---- libdl-2.23.so 00000000f6bb6000 4K rw--- libdl-2.23.so 00000000f6bb7000 1728K r-x-- libc-2.23.so 00000000f6d67000 8K r---- libc-2.23.so 00000000f6d69000 4K rw--- libc-2.23.so 00000000f6d6a000 12K rw--- [ anon ] 00000000f6d6d000 100K r-x-- libpthread-2.23.so 00000000f6d86000 4K r---- libpthread-2.23.so 00000000f6d87000 4K rw--- libpthread-2.23.so 00000000f6d88000 8K rw--- [ anon ] 00000000f6d8a000 112K r-x-- libgcc_s.so.1 00000000f6da6000 4K rw--- libgcc_s.so.1 00000000f6da7000 332K r-x-- libm-2.23.so 00000000f6dfa000 4K r---- libm-2.23.so 00000000f6dfb000 4K rw--- libm-2.23.so 00000000f6dfc000 4K rw--- [ anon ] 00000000f6dfd000 1460K r-x-- libstdc++.so.6.0.21 00000000f6f6a000 4K ----- libstdc++.so.6.0.21 00000000f6f6b000 24K rw--- libstdc++.so.6.0.21 00000000f6f71000 12K rw--- [ anon ] 00000000f6f74000 28K r-x-- librt-2.23.so 00000000f6f7b000 8K rw--- librt-2.23.so 00000000f6f7d000 60K r---- ld.so.cache 00000000f6f8c000 2128K r-x-- libext.so 00000000f71a0000 4K rw--- libext.so 00000000f71a1000 6044K r-x-- libbyond.so 00000000f7788000 12K rw--- libbyond.so 00000000f778b000 100K rw--- [ anon ] 00000000f77a4000 8K r---- [ anon ] 00000000f77a6000 4K r-x-- [ anon ] 00000000f77a7000 140K r-x-- ld-2.23.so 00000000f77ca000 8K rw--- ld-2.23.so 00000000ffcd9000 132K rw--- [ stack ] total 12576K --- Memory Map End --- | |
Well, that crash is in the same place, so it's consistent. That's something. I'll try to find if there's any conceivable way this could be happening without heap corruption, but it's likely I won't get very far unless I can reproduce it myself. Does this happen even if you start a round without any players (except maybe an admin to login and reboot it), or does it require that the round had some intense activity?
| |
I have not attempted to reproduce it in a sandbox environment, but it happens regardless of round activity when DD tries to restart the world after the round ends. You should be able to reproduce it by starting a round, having it end, then wait for the world to restart itself.
| |
Can you link me to a zip of the exact version of the code you're running? I'll download it and see if I can reproduce this locally then.
| |
Same issue here except we dont have memory map enabled. but here is our issue. this is from YW which is a branch of vorestation.
we run 512.1460 World rebooted at 2019-12-28T19:12:39 BUG: Crashing due to an illegal operation! Backtrace for BYOND 512.1460 on Linux: Generated at Sat Dec 28 19:13:43 2019 DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bd64] libbyond.so [0xf79a5000, 0x0], 0x23fbc8 linux-gate.so.1 [0xf7f81000, 0xf7f820a0], [0xf7f81000, 0xf7f820a0] libbyond.so [0xf79a5000, 0x0], 0x23fbc8 libbyond.so [0xf79a5000, 0x0], 0x262e8f libbyond.so [0xf79a5000, 0x0], 0x26481e libbyond.so [0xf79a5000, 0x0], 0x264bb3 libbyond.so 0x3932c0, 0x3933da libbyond.so 0x355230, 0x355432 DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804af94] libc.so.6 0x18d90, 0x18e81 (__libc_start_main) DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a7d1] Recent proc calls: /datum/species/proc/handle_logout_special /proc/key_name /proc/log_access_out /mob/proc/update_client_z /datum/controller/subsystem/nanoui/proc/close_user_uis /datum/controller/subsystem/nanoui/proc/user_logout /mob/Logout /mob/living/Logout /mob/living/carbon/human/Logout /datum/admin_help_tickets/proc/ClientLogout /client/Del /proc/key_name /proc/log_access_out /mob/proc/update_client_z /datum/controller/subsystem/nanoui/proc/close_user_uis /datum/controller/subsystem/nanoui/proc/user_logout | |
actually. this didn't crash when initiating an immediate reboot
| |
There's zero excuse to be running 512.1460 anymore. 512.1488 is the current version and 1460 is way, way out of date. I'm not looking into any crashes from versions that old.
| |
may be worth nothing that handle_logout_special is repeated twice and the rest is in order
| |
noting that the crash is not exclusive to linux at all. just had it happen on our windows server
| |
The strongest case of heap corruption I've found has been in icon operations, several important sources of which were cleared up in 513. Those mostly impact the client, but I think it's possible for there to be a server dimension in some weird cases.
If you run the server in 513 does the same issue happen?