ID:2088350
 
BYOND Version:509
Operating System:Linux
Web Browser:Chrome 50.0.2661.102
Applies to:Dream Daemon
Status: Open

Issue hasn't been assigned a status value.
Descriptive Problem Summary:
This bug is completely random but it has been happening a lot more recently. Shell Server keeps going down but the games it hosts remain live. The problem is that people can't manage their games if they don't have the verbs in their games to manage them now.

BUG: Crashing due to an illegal operation!

Backtrace for BYOND 509.1318 on Linux:
Generated at Mon May 23 14:31:52 2016

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bb24]
libbyond.so [0xb7280000, 0x0], 0x320ba9
[0xb77ca000, 0xb77ca600], [0xb77ca000, 0xb77ca600]
libbyond.so [0xb7280000, 0x0], 0x320ba9
libbyond.so [0xb7280000, 0x0], 0x3211fe
libbyond.so [0xb7280000, 0x0], 0x2e8b37
libbyond.so [0xb7280000, 0x0], 0x2db339
libbyond.so [0xb7280000, 0x0], 0x2dcd8c
libbyond.so [0xb7280000, 0x0], 0x2dca86
libbyond.so [0xb7280000, 0x0], 0x2de378
libbyond.so [0xb7280000, 0x0], 0x2de845
libbyond.so 0x2e0970, 0x2e10ed
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804ae34]
libc.so.6 0x16d60, 0x16e46 (__libc_start_main)
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a731]

Recent proc calls:
/proc/StatLoop
/proc/StatLoop
/mob/Logout
/mob/Stat
/mob/proc/isTyping
/mob/proc/isTyping
/mob/proc/labelSet
/mob/Stat
/mob/Stat
/proc/timestamp
/proc/StatLoop
/proc/StatLoop
/mob/Stat
/mob/proc/isTyping
/mob/proc/isTyping
/mob/proc/labelSet

To help the BYOND developers debug this, please send the above trace as part
of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1



Numbered Steps to Reproduce Problem:
I have absolutely no clue. But, I'm starting to question the people who are reporting the server manager being down to see if there's any correlation with what they were doing last on the server manager.

Did the problem NOT occur in any earlier versions? If so, what was the last version that worked? (Visit http://www.byond.com/download/build to download old versions for testing.) It has honestly always occurred and I've done other bug reports on this.

This report isn't really helpful but it's better than making no report at all. Hopefully I get some valuable information that I can post here and this issue gets resolved.

If you need any code snippets for the recent procedures, let me know. I'm open to posting them.
509.1318 is not the stable version even of 509. That's an outdated beta and should no longer be used at all.

I would however recommend switching to 510, as it's far enough along that there's a chance whatever caused this was already fixed.
I have been waiting for 510 to be released as stable. Even if it is already stable. I just wanted to reduce the amount of times I had to update. Because then I'd be in this same situation where I am on a beta release and there is a stable release one minor version ahead. :P If it's not coming by the end if this month then I guess I will update.
It will definitely come by then, but you're already on a beta version. I can't in good conscience do a trace on a version this old.
There's no rush. Like I said, the report isn't all that helpful. But this has been an issue since 2014. Some parts of it has been fixed and the frequency of it has been reduced. When 510 become stable I'll update and if crashes occur again I'll post an update. If I did not make this report, the odds of me posting a report about it in 510 would be very low. It's better to just get it started, right?

I guess this could fall under the section that says "does this problem occur on other BYOND versions" as well.
The problem with a random crash is that it could be caused by just about anything. For all either of us knows, the issue you had in 2014 has long been fixed, and this one is a different crasher entirely. And for all either of us knows, this one may have been fixed too.

If this had been on 509.1319, I'd have done a quick trace. I probably already have some saved data on that. But 1318 is not just an old version, but an old beta.
I updated to 510 and will keep you posted.
BUG: Crashing due to an illegal operation!
Thu May 26 15:27:53 2016

Backtrace for BYOND 510.1345 on Linux:
Generated at Thu May 26 15:27:53 2016

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bb24]
libbyond.so 0x32d630, 0x32d752
[0xb7773000, 0xb7773600], [0xb7773000, 0xb7773600]
libbyond.so 0x32d630, 0x32d752
libbyond.so [0xb7208000, 0x0], 0x258479
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804c253]
[0xb7773000, 0xb7773500], [0xb7773000, 0xb7773500]
[0xb7773000, 0xb7773400], [0xb7773000, 0xb7773405]
libc.so.6 0x2ab50, 0x2ab9b (sigprocmask)
libc.so.6 0x2a720, 0x2a78b (_longjmp)
libpthread.so.0 [0xb6e88000, 0x0], 0xd134
libbyond.so [0xb7208000, 0x0], 0x2f6bcb
libbyond.so [0xb7208000, 0x0], 0x32eb4f
[0xb7773000, 0xb7773500], [0xb7773000, 0xb7773500]

Recent proc calls:
/proc/displayServers
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/displayServers
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/displayServers
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/displayServers
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/displayServers
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/displayServers

To help the BYOND developers debug this, please send the above trace as part
of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1


Honestly, it always feels like these crashes happen when you have those few selected players who spam commands to try and get things to work. After viewing the rest of my logs, it seems like someone was trying to upload the same savefile aboutttttt.... I'd have to say 700+ times over the course of 8 hours.

It also seems that their game was started up and shut down more than 5 times back to back just before the hangup. It's not verifiable. But, this is an indicator for me:
Process 5990 dead!
Process 7484 dead!
Process 13138 dead!
Process 12273 dead!


A lot of the processes here use shell(), as you may remember. And that was definitely a leading cause in the crashes last time; something which I fixed considerably by wrapping the shell() in some if statements to check for null references (even though it shouldn't run if there is null anyways... since there is an if statement further above it. somehow it slipped through regardless or turned null as soon as shell() was running) and slowing down its usage.

Keep note, the above report was generated from SIGSEGV. The program hung up this time, stayed at 90%+, and did not naturally generate a report.
Okay, glad to know it was from SIGSEGV, because the trace occurred on a jump call and that confused the heck out of me--since it should be impossible to crash there.

However, since this didn't actually crash, that trace isn't giving me anything useful to go on. What you'll actually have to do instead is attach gdb to the hanging process, and use that to get the stack trace. You'll want to get the offsets in libbyond.so where it's at, not just raw memory addresses, and get those as far back as you can go. That will be very useful information.
Well, this is pretty new to me. I just installed gdb:
apt-get install gdb


And I did some research in the man page and on Stack Overflow on how to attach it:
gdb -p PID or gdb program-name PID or gdb attach PID


I don't know about that last one being valid. But, based on the man page, the first two seem to be valid.

Some questions I have would be how to offset it to libbyond.so and how do I get the ones in the back?
That you'd have to ask a Linux expert about.
Alrighty, one of my servers hung up again. The PID for that game was 1745. To attach GDB, this was the first command I initated:
gdb -p 1745


I then proceeded to get a list of all the frames in GDB by using the backtrace command (these are individual commands inside the gdb command-line for anyone else reading this):
bt


That gave this output:
(gdb) bt
#0 0xb7501750 in TimeLib::AddTimer(long*, timeval, unsigned char, void (*)(long, void*), void*) () from /usr/local/lib/libbyond.so
#1 0xb742c479 in ?? () from /usr/local/lib/libbyond.so
#2 0x0804c253 in ?? ()
#3 <signal handler called>
#4 __pthread_mutex_lock (mutex=0xbf8a78fc) at pthread_mutex_lock.c:50
#5 0x0804cd12 in ?? ()
#6 0x0804ae4d in ?? ()
#7 0xb6d1de46 in __libc_start_main () from /lib32/libc.so.6
#8 0x0804a731 in ?? ()


I also ran bt full to get some more information that I did not understand. That produced this output:
(gdb) bt full
#0 0xb7501750 in TimeLib::AddTimer(long*, timeval, unsigned char, void (*)(long, void*), void*) () from /usr/local/lib/libbyond.so
No symbol table info available.
#1 0xb742c479 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#2 0x0804c253 in ?? ()
No symbol table info available.
#3 <signal handler called>
No symbol table info available.
#4 __pthread_mutex_lock (mutex=0xbf8a78fc) at pthread_mutex_lock.c:50
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = 1
#5 0x0804cd12 in ?? ()
No symbol table info available.
#6 0x0804ae4d in ?? ()
No symbol table info available.
#7 0xb6d1de46 in __libc_start_main () from /lib32/libc.so.6
No symbol table info available.
#8 0x0804a731 in ?? ()
No symbol table info available.


You stated that you wanted it offset in to libbyond.so. Unfortunately, I don't know what it is you want specifically to see. But, seeing as how there's libbyond.so present in these backtraces, I figured there's more that could be done. So I learned that each of those #'s represent a frame. With a little research, I learned how to get info on a frame. As such, I ran the following command:
(gdb) frame 1 // This verifies that that frame number is the one you want by outputting what is below.
//The number "1" can be replaced with whatever the Frame # was when you typed "bt" as a command.
#1 0xb742c479 in ?? () from /usr/local/lib/libbyond.so

#1  0xb742c479 in ?? () from /usr/local/lib/libbyond.so
(gdb) info frame 1 // This gets info on frame #1
Stack frame at 0xbf8a74d0:
eip = 0xb742c479; saved eip 0x804c253
called by frame at 0xbf8a7500, caller of frame at 0xbf8a7450
Arglist at 0xbf8a74c8, args:
Locals at 0xbf8a74c8, Previous frame's sp is 0xbf8a74d0
Saved registers:
ebx at 0xbf8a74bc, ebp at 0xbf8a74c8, esi at 0xbf8a74c0, edi at 0xbf8a74c4, eip at 0xbf8a74cc


If this is not enough for you, I will have to try again next time and see if I can go deeper.

Edit: I know you said that SIGSEGV and SIGBUS backtrace reports didn't entirely help this situation. But, I might as well get every information as possible. It wouldn't hurt, now would it? Nothing needs to be done with the information.
Thu Jun  9 10:24:17 2016
World opened on network port 7000.
Welcome BYOND! (5.0 Public Version 510.1345)
The BYOND hub reports that port 7000 is reachable.

...
...
...

BUG: Crashing due to an illegal operation!
Wed Jun 15 16:28:54 2016

Backtrace for BYOND 510.1345 on Linux:
Generated at Wed Jun 15 16:28:54 2016

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bb24]
libbyond.so 0x32d630, 0x32d750
[0xb771d000, 0xb771d600], [0xb771d000, 0xb771d600]
libbyond.so 0x32d630, 0x32d750
libbyond.so [0xb71d4000, 0x0], 0x258479
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804c253]
[0xb771d000, 0xb771d500], [0xb771d000, 0xb771d500]
libpthread.so.0 0x8270, 0x8294
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804cd12]
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804ae4d]
libc.so.6 0x16d60, 0x16e46 (__libc_start_main)
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a731]

Recent proc calls:
/proc/displayServers
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/displayServers
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU

To help the BYOND developers debug this, please send the above trace as part of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1
It looks like this didn't get any new information. It's clear this is happening on a return from a background proc, which could be something like winget(), although shell() is the most likely explanation.

What I think is going on is that something is calling a proc like startup() or shell(), which is returning almost immediately, and it's repeating over and over. Out of curiosity, what do those last three procs (displayServers, getServer_CPU, getCPU_Usage) look like?
proc/getServer_CPU(var/S)
for(var/F in serverList[S])
if(worldRestoring || worldSaving)
return

var/serverChild/U = serverList[S][F]
if((istype(U,/serverChild))&&(U.PID))
if(shell("kill -0 [U.PID]") == 1)
text2file("Key:[U.Key]| Name: [U.Name]| Folder: [U.Folder] | Time: [time2text(world.realtime,"MMM DD, YYYY")] [timeOfDay]","Crash.txt")
del(U)
else
if(U)
U.CPU="[getCPU_Usage(U.PID)]%"
U.name = "[U.Name] - CPU: [U.CPU]"

proc/getCPU_Usage(pid)
var/cpu_file = "./data/[pid]_cpu.log"

var/cpu = shell("[TOP_PATH]top -p [pid] -b -n1 | [GREP_PATH]grep \"[pid]\" | [AWK_PATH]awk '{print $9}' > \'[cpu_file]\'")
cpu = text2num(copytext(file2text(cpu_file),1,lentext(file2text(cpu_file))))

fdel(cpu_file)

if(cpu > 0)
return cpu
else
return 0

proc/displayServers() // @Lummox - This is ran in stat. It should only run if the user has that tab open and is not AFK. If they are AFK it stops.
if(!worldRestoring&&!worldSaving&&!gettingCPUUsage)
SERVERS_VIEWED = 0
gettingCPUUsage=1
// authIDs=new/list
for(var/S in serverList)
var/list/SL=serverList[S]
if(!istype(SL,/list))
continue
if(SL.len<1)
continue
// var/serverChild/T = SL[SL[1]]
// if(T)
spawn()
getServer_CPU(S)
sleep(50)
gettingCPUUsage=0


The things to do with CPU Usage typically uses shell(). And it will only use it if there is a user to view it. I know it's shell() because this was definitely one of the cases beforehand. And it cleared up when I minimized the use of shell().

I am actually working on writing the entire program from scratch since it's fairly simple at this point though. It'd also resolve any conflicts we had when there were two people working on this at once without any agreements on conventions.

Edit: Something to note though is that null references seem to be slipping through.
runtime error: Cannot modify null.CPU.
proc name: getServer CPU (/proc/getServer_CPU)
source file: Server Statistics.dm,39
usr: (src)
src: null
call stack:
getServer CPU("furrychicken")


Is if(U) not enough? I'm starting to wonder if isnull() is a requirement in this instance. Regardless, days go by without it being a problem. Not sure if it's truly a result of the random null references.
I think the Stat() run here is super problematic. stat() is meant to behave differently when it sleeps; it should never ever be calling procs that wait.
In response to Lummox JR
Lummox JR wrote:
I think the Stat() run here is super problematic. stat() is meant to behave differently when it sleeps; it should never ever be calling procs that wait.

I'll see what I can do to change that tomorrow. In the meantime, I got another trace for you. I don't have much time to organize it for you. So, forgive me. But I tried to keep the commands as straightforward as possible so it's not jumbled.

:~# gdb -p 18964
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 18964
Reading symbols from /usr/local/byond/bin/DreamDaemon...(no debugging symbols found)...done.
Reading symbols from /usr/local/lib/libbyond.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libbyond.so
Reading symbols from /usr/local/lib/libext.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libext.so
Reading symbols from /lib32/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib32/librt.so.1
Reading symbols from /usr/lib32/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib32/libstdc++.so.6
Reading symbols from /lib32/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib32/libm.so.6
Reading symbols from /usr/lib32/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib32/libgcc_s.so.1
Reading symbols from /lib32/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib32/libpthread.so.0
Reading symbols from /lib32/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib32/libc.so.6
Reading symbols from /lib32/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib32/libdl.so.2
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib32/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib32/libnss_files.so.2
Reading symbols from /lib32/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib32/libnss_dns.so.2
Reading symbols from /lib32/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib32/libresolv.so.2
0xb75be75a in TimeLib::AddTimer(long*, timeval, unsigned char, void (*)(long, void*), void*) ()
from /usr/local/lib/libbyond.so
(gdb) bt full
#0 0xb75be75a in TimeLib::AddTimer(long*, timeval, unsigned char, void (*)(long, void*), void*) ()
from /usr/local/lib/libbyond.so
No symbol table info available.
#1 0xb74e9479 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#2 0x0804c253 in ?? ()
No symbol table info available.
#3 <signal handler called>
No symbol table info available.
#4 0xb75be9ba in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#5 0xb7585e3d in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#6 0xb7585f66 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#7 0xb74ff6d7 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#8 0xb748a096 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#9 0xb74d0a80 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#10 0xb74e8ace in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#11 0xb74f12e3 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#12 0xb74f267b in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#13 0xb74eae0c in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#14 0xb74f1463 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#15 0xb74f267b in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#16 0xb74a1247 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#17 0xb74a58b1 in ?? () from /usr/local/lib/libbyond.so
No symbol table info available.
#18 0xb75bf4ba in TimeLib::SystemAlarm() () from /usr/local/lib/libbyond.so
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#19 0xb75895c2 in SocketLib::WaitForSocketIO(long, unsigned char) () from /usr/local/lib/libbyond.so
No symbol table info available.
#20 0x0804ae34 in ?? ()
No symbol table info available.
#21 0xb6ddae46 in __libc_start_main () from /lib32/libc.so.6
No symbol table info available.
#22 0x0804a731 in ?? ()
No symbol table info available.
(gdb) info frame 0
Stack frame at 0xbf8c3a00:
eip = 0xb75be75a in TimeLib::AddTimer(long*, timeval, unsigned char, void (*)(long, void*), void*);
saved eip 0xb74e9479
called by frame at 0xbf8c3a80
Arglist at 0xbf8c39f8, args:
Locals at 0xbf8c39f8, Previous frame's sp is 0xbf8c3a00
Saved registers:
ebx at 0xbf8c39ec, ebp at 0xbf8c39f8, esi at 0xbf8c39f0, edi at 0xbf8c39f4, eip at 0xbf8c39fc
(gdb) info frame 1
Stack frame at 0xbf8c3a80:
eip = 0xb74e9479; saved eip 0x804c253
called by frame at 0xbf8c3ab0, caller of frame at 0xbf8c3a00
Arglist at 0xbf8c3a78, args:
Locals at 0xbf8c3a78, Previous frame's sp is 0xbf8c3a80
Saved registers:
ebx at 0xbf8c3a6c, ebp at 0xbf8c3a78, esi at 0xbf8c3a70, edi at 0xbf8c3a74, eip at 0xbf8c3a7c
(gdb) info frame 2
Stack frame at 0xbf8c3ab0:
eip = 0x804c253; saved eip 0xb77da500
called by frame at 0xbf8c4000, caller of frame at 0xbf8c3a80
Arglist at 0xbf8c3aa8, args:
Locals at 0xbf8c3aa8, Previous frame's sp is 0xbf8c3ab0
Saved registers:
ebx at 0xbf8c3aa4, ebp at 0xbf8c3aa8, eip at 0xbf8c3aac
(gdb) info frame 3
Stack frame at 0xbf8c4000:
eip = 0xb77da500 in __kernel_sigreturn; saved eip 0xb75be9ba
called by frame at 0xbf8c4050, caller of frame at 0xbf8c3ab0
Arglist at unknown address.
Locals at unknown address, Previous frame's sp is 0xbf8c4000
Saved registers:
eax at 0xbf8c3ae0, ecx at 0xbf8c3adc, edx at 0xbf8c3ad8, ebx at 0xbf8c3ad4, ebp at 0xbf8c3acc, esi at 0xbf8c3ac8,
edi at 0xbf8c3ac4, eip at 0xbf8c3aec
(gdb) info frame 4
Stack frame at 0xbf8c4050:
eip = 0xb75be9ba; saved eip 0xb7585e3d
called by frame at 0xbf8c40a0, caller of frame at 0xbf8c4000
Arglist at 0xbf8c4048, args:
Locals at 0xbf8c4048, Previous frame's sp is 0xbf8c4050
Saved registers:
ebx at 0xbf8c403c, ebp at 0xbf8c4048, esi at 0xbf8c4040, edi at 0xbf8c4044, eip at 0xbf8c404c
(gdb) info frame 5
Stack frame at 0xbf8c40a0:
eip = 0xb7585e3d; saved eip 0xb7585f66
called by frame at 0xbf8c4520, caller of frame at 0xbf8c4050
Arglist at 0xbf8c4098, args:
Locals at 0xbf8c4098, Previous frame's sp is 0xbf8c40a0
Saved registers:
ebx at 0xbf8c408c, ebp at 0xbf8c4098, esi at 0xbf8c4090, edi at 0xbf8c4094, eip at 0xbf8c409c
(gdb) info frame 6
Stack frame at 0xbf8c4520:
eip = 0xb7585f66; saved eip 0xb74ff6d7
called by frame at 0xbf8c4540, caller of frame at 0xbf8c40a0
Arglist at 0xbf8c4518, args:
Locals at 0xbf8c4518, Previous frame's sp is 0xbf8c4520
Saved registers:
ebx at 0xbf8c450c, ebp at 0xbf8c4518, esi at 0xbf8c4510, edi at 0xbf8c4514, eip at 0xbf8c451c
(gdb) info frame 7
Stack frame at 0xbf8c4540:
eip = 0xb74ff6d7; saved eip 0xb748a096
called by frame at 0xbf8c49b0, caller of frame at 0xbf8c4520
Arglist at 0xbf8c4538, args:
Locals at 0xbf8c4538, Previous frame's sp is 0xbf8c4540
Saved registers:
ebp at 0xbf8c4538, eip at 0xbf8c453c
(gdb) info frame 8
Stack frame at 0xbf8c49b0:
eip = 0xb748a096; saved eip 0xb74d0a80
called by frame at 0xbf8c50d0, caller of frame at 0xbf8c4540
Arglist at 0xbf8c49a8, args:
Locals at 0xbf8c49a8, Previous frame's sp is 0xbf8c49b0
Saved registers:
ebx at 0xbf8c499c, ebp at 0xbf8c49a8, esi at 0xbf8c49a0, edi at 0xbf8c49a4, eip at 0xbf8c49ac
(gdb) info frame 9
Stack frame at 0xbf8c50d0:
eip = 0xb74d0a80; saved eip 0xb74e8ace
called by frame at 0xbf8c51a0, caller of frame at 0xbf8c49b0
Arglist at 0xbf8c50c8, args:
Locals at 0xbf8c50c8, Previous frame's sp is 0xbf8c50d0
Saved registers:
ebx at 0xbf8c50bc, ebp at 0xbf8c50c8, esi at 0xbf8c50c0, edi at 0xbf8c50c4, eip at 0xbf8c50cc
(gdb) info frame 10
Stack frame at 0xbf8c51a0:
eip = 0xb74e8ace; saved eip 0xb74f12e3
called by frame at 0xbf8c5210, caller of frame at 0xbf8c50d0
Arglist at 0xbf8c5198, args:
Locals at 0xbf8c5198, Previous frame's sp is 0xbf8c51a0
Saved registers:
ebx at 0xbf8c518c, ebp at 0xbf8c5198, esi at 0xbf8c5190, edi at 0xbf8c5194, eip at 0xbf8c519c
(gdb) info frame 11
Stack frame at 0xbf8c5210:
eip = 0xb74f12e3; saved eip 0xb74f267b
called by frame at 0xbf8c5280, caller of frame at 0xbf8c51a0
Arglist at 0xbf8c5208, args:
Locals at 0xbf8c5208, Previous frame's sp is 0xbf8c5210
Saved registers:
ebx at 0xbf8c51fc, ebp at 0xbf8c5208, esi at 0xbf8c5200, edi at 0xbf8c5204, eip at 0xbf8c520c
(gdb) info frame 12
Stack frame at 0xbf8c5280:
eip = 0xb74f267b; saved eip 0xb74eae0c
called by frame at 0xbf8c5560, caller of frame at 0xbf8c5210
Arglist at 0xbf8c5278, args:
Locals at 0xbf8c5278, Previous frame's sp is 0xbf8c5280
Saved registers:
ebx at 0xbf8c526c, ebp at 0xbf8c5278, esi at 0xbf8c5270, edi at 0xbf8c5274, eip at 0xbf8c527c
(gdb) info frame 13
Stack frame at 0xbf8c5560:
eip = 0xb74eae0c; saved eip 0xb74f1463
called by frame at 0xbf8c55d0, caller of frame at 0xbf8c5280
Arglist at 0xbf8c5558, args:
Locals at 0xbf8c5558, Previous frame's sp is 0xbf8c5560
Saved registers:
ebx at 0xbf8c554c, ebp at 0xbf8c5558, esi at 0xbf8c5550, edi at 0xbf8c5554, eip at 0xbf8c555c
(gdb) info frame 14
Stack frame at 0xbf8c55d0:
eip = 0xb74f1463; saved eip 0xb74f267b
called by frame at 0xbf8c5640, caller of frame at 0xbf8c5560
Arglist at 0xbf8c55c8, args:
Locals at 0xbf8c55c8, Previous frame's sp is 0xbf8c55d0
Saved registers:
ebx at 0xbf8c55bc, ebp at 0xbf8c55c8, esi at 0xbf8c55c0, edi at 0xbf8c55c4, eip at 0xbf8c55cc
(gdb) info frame 15
Stack frame at 0xbf8c5640:
eip = 0xb74f267b; saved eip 0xb74a1247
called by frame at 0xbf8c5b10, caller of frame at 0xbf8c55d0
Arglist at 0xbf8c5638, args:
Locals at 0xbf8c5638, Previous frame's sp is 0xbf8c5640
Saved registers:
ebx at 0xbf8c562c, ebp at 0xbf8c5638, esi at 0xbf8c5630, edi at 0xbf8c5634, eip at 0xbf8c563c
(gdb) info frame 16
Stack frame at 0xbf8c5b10:
eip = 0xb74a1247; saved eip 0xb74a58b1
called by frame at 0xbf8c5ba0, caller of frame at 0xbf8c5640
Arglist at 0xbf8c5b08, args:
Locals at 0xbf8c5b08, Previous frame's sp is 0xbf8c5b10
Saved registers:
ebx at 0xbf8c5afc, ebp at 0xbf8c5b08, esi at 0xbf8c5b00, edi at 0xbf8c5b04, eip at 0xbf8c5b0c
(gdb) info frame 17
Stack frame at 0xbf8c5ba0:
eip = 0xb74a58b1; saved eip 0xb75bf4ba
called by frame at 0xbf8c5bf0, caller of frame at 0xbf8c5b10
Arglist at 0xbf8c5b98, args:
Locals at 0xbf8c5b98, Previous frame's sp is 0xbf8c5ba0
Saved registers:
ebx at 0xbf8c5b8c, ebp at 0xbf8c5b98, esi at 0xbf8c5b90, edi at 0xbf8c5b94, eip at 0xbf8c5b9c
(gdb) info frame 18
Stack frame at 0xbf8c5bf0:
eip = 0xb75bf4ba in TimeLib::SystemAlarm(); saved eip 0xb75895c2
called by frame at 0xbf8c5dd0, caller of frame at 0xbf8c5ba0
Arglist at 0xbf8c5be8, args:
Locals at 0xbf8c5be8, Previous frame's sp is 0xbf8c5bf0
Saved registers:
ebx at 0xbf8c5bdc, ebp at 0xbf8c5be8, esi at 0xbf8c5be0, edi at 0xbf8c5be4, eip at 0xbf8c5bec
(gdb) info frame 19
Stack frame at 0xbf8c5dd0:
eip = 0xb75895c2 in SocketLib::WaitForSocketIO(long, unsigned char); saved eip 0x804ae34
called by frame at 0xbf8c5f80, caller of frame at 0xbf8c5bf0
Arglist at 0xbf8c5dc8, args:
Locals at 0xbf8c5dc8, Previous frame's sp is 0xbf8c5dd0
Saved registers:
ebx at 0xbf8c5dbc, ebp at 0xbf8c5dc8, esi at 0xbf8c5dc0, edi at 0xbf8c5dc4, eip at 0xbf8c5dcc
(gdb) info frame 20
Stack frame at 0xbf8c5f80:
eip = 0x804ae34; saved eip 0xb6ddae46
called by frame at 0xbf8c6000, caller of frame at 0xbf8c5dd0
Arglist at 0xbf8c5f78, args:
Locals at 0xbf8c5f78, Previous frame's sp is 0xbf8c5f80
Saved registers:
ebx at 0xbf8c5f64, ebp at 0xbf8c5f78, esi at 0xbf8c5f68, edi at 0xbf8c5f6c, eip at 0xbf8c5f7c
(gdb) info frame 21
Stack frame at 0xbf8c6000:
eip = 0xb6ddae46 in __libc_start_main; saved eip 0x804a731
called by frame at 0x0, caller of frame at 0xbf8c5f80
Arglist at 0xbf8c5ff8, args:
Locals at 0xbf8c5ff8, Previous frame's sp is 0xbf8c6000
Saved registers:
ebx at 0xbf8c5fec, ebp at 0xbf8c5ff8, esi at 0xbf8c5ff0, edi at 0xbf8c5ff4, eip at 0xbf8c5ffc
(gdb) info frame 22
Stack frame at 0x0:
eip = 0x804a731; saved eip 0x804a731
Outermost frame: outermost
caller of frame at 0xbf8c6000
Arglist at unknown address.
Locals at unknown address, Previous frame's sp in esp


There are 22 frames in this one. I ran info frame on them all since there were a lot of libbyond.so frames.

Edit: Looks like a lead.
Wed Jun 15 22:01:49 2016
proc name: Stat (/mob/Stat)
source file: PM.dm,149
usr: (src)
src: Zeex00 (/mob)
src.loc: null
call stack:
Zeex00 (/mob): Stat()

Backtrace for BYOND 510.1345 on Linux:
Generated at Wed Jun 15 22:01:49 2016

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804bb24]
libbyond.so 0x32d630, 0x32d752
[0xb77da000, 0xb77da600], [0xb77da000, 0xb77da600]
libbyond.so 0x32d630, 0x32d752
libbyond.so [0xb7291000, 0x0], 0x258479
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804c253]
[0xb77da000, 0xb77da500], [0xb77da000, 0xb77da500]
libbyond.so [0xb7291000, 0x0], 0x32d9ba
libbyond.so [0xb7291000, 0x0], 0x2f4e3d
libbyond.so [0xb7291000, 0x0], 0x2f4f66
libbyond.so [0xb7291000, 0x0], 0x26e6d7
libbyond.so [0xb7291000, 0x0], 0x1f9096
libbyond.so [0xb7291000, 0x0], 0x23fa80
libbyond.so [0xb7291000, 0x0], 0x257ace
libbyond.so [0xb7291000, 0x0], 0x2602e3
libbyond.so [0xb7291000, 0x0], 0x26167b
libbyond.so [0xb7291000, 0x0], 0x259e0c
libbyond.so [0xb7291000, 0x0], 0x260463
libbyond.so [0xb7291000, 0x0], 0x26167b
libbyond.so [0xb7291000, 0x0], 0x210247
libbyond.so [0xb7291000, 0x0], 0x2148b1
libbyond.so 0x32e3a0, 0x32e4ba
libbyond.so 0x2f83c0, 0x2f85c2
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804ae34]
libc.so.6 0x16d60, 0x16e46 (__libc_start_main)
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a731]

Recent proc calls:
/mob/Stat
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage
/proc/getServer_CPU
/proc/getCPU_Usage

To help the BYOND developers debug this, please send the above trace as part
of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1


proc
StatLoop() // @ Lummox -- Ran from new. Not from Stat(). We used to do it from Stat() but that was inefficient since it was running for each player.
set background = 1

statLoopTicks--
if(statLoopTicks <= 0)
statLoopTicks = 360
var/list/curDate = currentDate()
var/curDateText="[curDate[1]],[curDate[2]],[curDate[3]]"
if((!lastExpirationCheck) || curDateText!=lastExpirationCheck)
lastExpirationCheck = curDateText
for(var/k in subDatums)//the keys
set background = 10
var/subHandler/s=subDatums[k]
if(istype(s,/subHandler))
s.checkExp()
else
subDatums-=k

if(SERVERS_VIEWED)
spawn()
displayServers()

if(playersC > 0)
timestamp()
//spawn(50)
//displayServers()
spawn(10)
StatLoop()

mob/Stat()
winset(src,"main.label2","text='[timeOfDay] EST'")
if(statpanel("Players"))
stat("World CPU: [world.cpu]%")
stat("<hr>")
stat(players)
stat("<font size = 1><b>[playersCount] player\s online.</b></font>")
stat("<font size = 1><b>[adminsCount] player\s admining.</b></font>")
stat("<font size = 1><b>[moderatorsCount] player\s moderating.</b></font>")
stat("<font size = 1><b>[awayCount] player\s away.</b></font>")
if(src.client)
if(!src.AFK)
spawn()
src.labelSet()
src.isTyping()
if(src.AFK==0&&src.client.inactivity>=3000)
spawn()
src.autoAFK()
Who()
if(src.AFK==1&&src.client.inactivity<3000&&src.autoAFK==1)
spawn()
src.autoAFK()
Who()

if(statpanel("Servers"))
stat("<b>Total Servers: [serverCount]</b>")
spawn()
if(!gettingCPUUsage&&(!src.AFK))
SERVERS_VIEWED = 1
stat(SERVERS)
It's the restart function. I noticed it myself. I was up and i logged on to reboot my server (Just to make sure everything was in Tip Top Shape!) and all of a sudden the entire server 1 crashed after I had attempted a reboot. Might want to look into that.
Does this happen frequently when you try to reboot? Because if it just happens once then it's not enough of a lead. It might happen once in a blue moon.

Edit:
It also uses shell @Lummox.
mob/permitted/verb
Restart(serverChild/s in world)
set category=null
if(cukey==ckey(s.Key))
switch(alert("Are you sure you wish to restart [s]?","Restart:","No","Yes"))
if("Yes")
if(cukey==ckey(s.Key))
if(s.PID)
src<<"Rebooting [s], this could take a moment."
shell("kill -10 [s.PID]")
else
src<<"An error occured."
CRASH("[src] tried to shutdown [s] while [s] did not have a PID.")
else
src<<"This server belongs to [s.Key]."
Well, We could test it again right now.
http://www.byond.com/forum/?post=1496464:
I guess we can assume that it was fixed earlier than 509.1306? The way I solved the issue was that I just threw the if(shell()) in a spawn(). That was my hackish way of fixing the issue.

The above is in relation to the displayServers() and getServer_CPU().

We're planning on doing another debugging session like the one that took place above. Only this time it'll be more informative as to what's going on in shell().