ID:133278
 
I know all talks of improving the networking on BYOND have been shot down in the past, but with the ever growing community size on BYOND it can't be put it off forever. And improving networking capabilities would help bring in more players in the process.

I really don't know anything about how the internal networking is setup for BYOND, so if you (Lummox, Tom, or anyone else who has actual facts) could post some information about it here, maybe we could come up with viable solutions for improvements as a community.


I ran some simple tests today and also analyzed the data from my server in order to see what a BYOND server was capable of.

The server I use has a 100mbps connection meaning it can send and receive at a maximum of 10,000 kbps. My computer has a 10mbps connection meaning 1,000 kbps download speed. However, when downloading the resources from a game; the speed only averaged 133 kbps, is there some built-in restriction on byond? or just a general limit to how fast it can transfer data?

Again, when looking at the the total use of daily bandwidth on the server it only sends, on average, 231 kb per second of a maximum 10,000. I'm not sure if this means that BYOND just doesn't need to send 10,000 kbps of data per second or if it is reaching some sort of internal restriction. (20 GB per day bandwidth usage, my math might be wrong here =P)

From what I've seen on the server; the overall system usage (according to the task manager) never exceeds 25% and usually stays around 10-15%, while the networking (again according to task manager) only expends 3% of its maximum. The in-game world.cpu of HU at the time of this writing was reporting an average of 68 players with 11% cpu usage. So the lag seen in-game can't possibly be caused by system exhaustion or overuse; if anything it's capabilities are being under-used.

Also, I run multiple BYOND servers on my server. 5 or more of them at all times. Logging in to the main server while it has 90+ players will present laggy gameplay, however logging in to one of the smaller servers at the same time, which has say only 25 players, will be as lag free as a BYOND game can get. Meaning that each of the servers are reaching some sort of personal max without slowing down the others that are being hosted from the same machine.

Also, BYOND noticeably lags regardless of player count, or processor usage. Even the simplest demo, say a simple map, basic animated walking player sprite, and nothing else; when connected to a remote server will visibly lag. When I say "remote server" I mean if you host on one machine and play on a different one; hosting on daemon/seeker but connecting to it from the same computer will still present smooth gameplay.
This can make it hard to judge some aspects of gameplay when testing them; unless you go through the hassle of uploading the game files to a server and testing on the hosted game.
Recently I set up a fully functional pixel-based movement and collision system, which ran fine, and looked fine when played locally, but when connecting to a server there were horribly visible jumps between movement and missed animation frames that made the system practically useless.
Though this isn't as apparent in games that use the built-in systems, you can still see a noticeable difference in the gameplay between playing a game locally and playing on a server, even if you are the only person in that server.
It seems like the lag in BYOND games will pick up mildly around 30 players, then more-so at 50, and by 100 the lag becomes a major hindrance to gameplay. I'm sure this varies by computer specs and networking speed on the machine, this is just in my experience.

I only post that previous section because I recently messed around with hosting a private WoW server, which, though it never got over 7 players online at one time, it didn't display any more or less lag if it was hosted locally or if I was connecting to the remotely hosted server. And after contacting the hosts of 2 popular private WoW servers, I confirmed that they host them on a single machine, and yet they can get 6000 players in a single server with as much lag as a BYOND server would have with 60.


As I said above, I have no real insight into how BYOND manages its networking internally, so this suggestion might be a mute point. But if the server could do some pre-processing, determining which values have been modified since the last send, it could cut down on the amount of information that it had to send to each player in each transmission, which would in turn reduce the amount of information that each client would have to process and then update. Since the z and icon variables change a lot less often than say the x and y do, and in the case of many NPCs never change at all, running some per-player pre-processing on the server to determine whats new could cut down on what information needs to be sent. Though this would require some more processing on the server's side... but that's kind of the whole point of a server, to process the inner workings of the game so the players' computers don't have to.

As mentioned above, if you could at least post some details about how the current systems works; maybe members of the community could come up with viable suggestions for improvements? If servers could hold hundreds, or even thousands of players, we could fill them up; we just need the opportunity =)
Falacy wrote:
The server I use has a 100mbps connection meaning it can send and receive at a maximum of 10,000 kbps. My computer has a 10mbps connection meaning 1,000 kbps download speed. However, when downloading the resources from a game; the speed only averaged 133 kbps, is there some built-in restriction on byond? or just a general limit to how fast it can transfer data?

Again, when looking at the the total use of daily bandwidth on the server it only sends, on average, 231 kb per second of a maximum 10,000. I'm not sure if this means that BYOND just doesn't need to send 10,000 kbps of data per second or if it is reaching some sort of internal restriction. (20 GB per day bandwidth usage, my math might be wrong here =P)

I think before I get into the math, the 100 megabit connection rating you're citing is likely the Ethernet speed, which is (supposedly) how fast data flows in your network, such as between two PCs, or a router and a PC.

As far as your math -- connections are always rated in bits. When an ISP claims they can give you 512 kilobits, your physical download speed is 64 KB/s. Let's see what speed your connection is at:

1.) Megabit-rated connections have to be multiplied by 1024, since there are 1024 bytes in a kilobyte, and 1024 kilobytes in a megabyte, etc. -- 10 megabits*1024=10240 kilobits.
2.) Since there are eight bits in a byte, you divide the above figure by eight to get what would be reported by, say, an IE file download box -- 10240 kilobits/8=1280 kilobytes.

Now given that 1280 kilobytes per second is still an impressive download speed, that's what you're paying for with your "10 megabit connection". Sounds a lot more impressive as advertised, doesn't it?

With that out of the way, there's also something else to be known: your connection is simply *capable* of 1280 kilobytes per second download speed. Not only are you likely working against network congestion and line resistance, you can only download as quickly as the server can send it out. 133 KB/s is likely a 2 megabit connection, so that server can only send data around 128 KB/s, so that's how fast you download. There's precisely zero you can do about it. Similarly, if your PC was hosting and a 1 megabit-connected player dropped by, they could only download the resources from you at around 128 KB/s.

This is why I'm happy with my 512 kilobit DSL.

The in-game world.cpu of HU at the time of this writing was reporting an average of 68 players with 11% cpu usage. So the lag seen in-game can't possibly be caused by system exhaustion or overuse; if anything it's capabilities are being under-used.

world.cpu is just a measure of how much of a tick the server requires to process the commands issued in the previous tick. 11% world.cpu means the server can process all of its' commands and still has the capacity to process around ten times as much in the same amount of time. Similarly, 120% world.cpu means it needs 1/5 more time (or 20% of a tick extra) per tick to process its command queue, which is obviously bad since it's unable to complete its workload before the tick has passed. Your 11% world.cpu is really a matter of workload shortage than network underutilization.

I only post that previous section because I recently messed around with hosting a private WoW server, which, though it never got over 7 players online at one time, it didn't display any more or less lag if it was hosted locally or if I was connecting to the remotely hosted server. And after contacting the hosts of 2 popular private WoW servers, I confirmed that they host them on a single machine, and yet they can get 6000 players in a single server with as much lag as a BYOND server would have with 60.

From what I understand, BYOND servers hold all of the cards and every client has to ask it what to do continuously, whereas with games like Warcraft you've downloaded a gigantic program with a ton of the instruction built into it already. I think the primary advantage is the use of client-side processing, since it isn't the Warcraft server PC that's making your health overlay.
In response to Mobius Evalon (#1)
You're making a lot of assumptions here. You've also swapped the upload and download relationship: Rarely does the upload ever get near half of the download.

My current connection is 20mbit down 2mbit up.

The bottom line seems to be that he has a download capability of 1000kbps and is only downloading with 133kbps on average. This either means the server isn't capable of uploading faster or there is something with BYOND stopping things from going any faster.

An easy way to test, Falacy, is to download something via FTP from your server and see if the speed is much better. If so, then you can isolate this to BYOND.
In response to Alathon (#2)
[Edit: Never mind. Sorry.]
In response to Alathon (#2)
Alathon wrote:
You're making a lot of assumptions here. You've also swapped the upload and download relationship: Rarely does the upload ever get near half of the download.

Err... damnit. I'm trashed on cold meds =(

I was just trying to point out that the common conception that a 10 megabit connection shows up as 10 MB/s on a file download is wrong. Disregarding step C, his download speed is still just over one megabyte per second, and what private party could possibly afford a 10 megabit symmetrical, anyway?
In response to Mobius Evalon (#4)
Mobius Evalon wrote:
I was just trying to point out that the common conception that a 10 megabit connection shows up as 10 MB/s on a file download is wrong. Disregarding step C, his download speed is still just over one megabyte per second, and what private party could possibly afford a 10 megabit symmetrical, anyway?

20mbit symmetrical is within what I can afford and I'm on student pay (It'd cost me $10 more a month, compared to the 20/2 I have now). I suppose it depends on country, really ;)

And yes, his download speed is just over one megabyte per second. Thats almost ten times what he was downloading with.
If there are internal bandwidth restrictions, my assumption would be that they were implemented in the days of Windows 95/98 and were intended to prevent a BYOND game from eating someone's entire dial-up connection.

Of course, that would just be a guess, on the possibly false premises that there are any such restrictions.
Falacy wrote:
I know all talks of improving the networking on BYOND have been shot down in the past, but with the ever growing community size on BYOND it can't be put it off forever. And improving networking capabilities would help bring in more players in the process.

Actually, I think BYOND is developing (slowly) in that direction anyway, given 4.0's interface.
The more of the game that is handled client sided, the "faster" the games are going to get.
Makes me wonder/wish for client sided code to be introduced with the considered/planned layers for the client interface.


Falacy wrote:
is there some built-in restriction on byond? or just a general limit to how fast it can transfer data?

My guess would be that .configure delay and world-tick_lag have a say in that, though, as said, it's a pure guess.


Falacy wrote:
they can get 6000 players in a single server with as much lag as a BYOND server would have with 60.

6000 on a single server sounds a bit much.
Even if WoW has a highly specific optimized network protocol, what's the specification of these servers?
In response to Schnitzelnagler (#7)
Mobius wrote:
I think before I get into the math, the 100 megabit connection rating you're citing is likely the Ethernet speed, which is (supposedly) how fast data flows in your network, such as between two PCs, or a router and a PC.

I wouldn't have bothered pointing out both the listed speed and the actual download speed if I thought you would still be confused -_-
My Computer: http://www.speedtest.net/result/279364804.png
My Server: http://www.speedtest.net/result/279363689.png

world.cpu is just a measure of how much of a tick the server requires to process the commands issued in the previous tick. 11% world.cpu means the server can process all of its' commands and still has the capacity to process around ten times as much in the same amount of time.

That's my point, the cpu on both the server and the games is reporting 10x less than max, and yet the games still lag.

I'd respond to some of your other comments but either you didn't read my post in full, didn't understand it, or just take me for a moron that cited completely wrong/made up facts about the server and/or my computer specs.


Kuraudo wrote:
If there are internal bandwidth restrictions, my assumption would be that they were implemented in the days of Windows 95/98 and were intended to prevent a BYOND game from eating someone's entire dial-up connection.

This is possible, the server is running windows 2003 server edition so it is a slightly older operating system.


Schnitzelnagler wrote:
6000 on a single server sounds a bit much.
Even if WoW has a highly specific optimized network protocol, what's the specification of these servers?

CPU: Intel Dual Quad Core Xeon
RAM: 4 GB
1000 mbit Connection
In response to Falacy (#8)
Does your server have a raid setup? If not, it might be hard drive read and write speed bottle-necking your system. You could check your disk usage in windows.
In response to Soldierman (#9)
Soldierman wrote:
Does your server have a raid setup? If not, it might be hard drive read and write speed bottle-necking your system.

Hmmmmm, no RAID, it has a single 250 GB SATA hard drive with a 7200 RPM read/write speed
In response to Falacy (#10)
Falacy wrote:
Soldierman wrote:
Does your server have a raid setup? If not, it might be hard drive read and write speed bottle-necking your system.

Hmmmmm, no RAID, it has a single 250 GB SATA hard drive with a 7200 RPM read/write speed

Eh. Even without sequential read/write (like big files), bad 7200RPM HDD's average at least 40mbps. This is not a HDD issue.
In response to Alathon (#2)
Alathon wrote:
An easy way to test, Falacy, is to download something via FTP from your server and see if the speed is much better. If so, then you can isolate this to BYOND.

When I download something from the web-server I have running it downloads at a max of 250kbps, however I use WAMP as my web/php/mysql/apache-server all in one, and my friend who helped me set it up told me that the speed restrictions are built into the WAMP software.

I can upload a 6 MB file to my angelfire.com site in under 10 seconds, and I assume the speedtest.net results have to use some sort of file transfer method, so the max speeds (or something close to them) do seem to be possible if software isn't limiting them.
In response to Falacy (#12)
Apache has no inherent bandwidth limit on serving up files, unless you specifically put it in config. I happily serve up files at 50MB/s via LAN here.
I was having similar issues a while back while I was doing hosting. 2 Games running off of the same server, once one of the games would hit around 70 players it would become almost unplayable but the other game on the server would run completely fine.

Probably whats causing this is BYOND isn't quite built for handling 75+ players doing what people want the games to be doing. Games that are currently running with large numbers of players per area are specifically tailored to having that many players. Look at almost any normal game. Most of the time you won't see more than 50 players in a given area before the server is either full, or a second instance of the zone is opened up. The few games I can think of that went above 50...

Battlefield games (64 players): Servers would get laggy as hell and 64 players per server weren't an officially supported amount of players.

Planetside (~300 players): Game was tailored to this, majority of things were handled client-side to keep the loads on the servers to a minimum. It worked, but there were many issues with it (eventually they reduced the number per zone to help with this).

EVE Online (1000+ players per zone, 40,000+ server-wide): Players cram into the systems and using dedicated servers and every trick in the book these systems still end up laggy.

Now take BYOND, which is not particularly tailored to any specific setup and it comes as no suprise that the engine starts to have some issues when you introduce a decent amount of people.

Probably the best thing that could be done would be to make it easier to seamlessly transfer players between worlds and not have the players have to confirm moving constantly (if they haven't turned it off which most won't have). Given the experience with the two games I was hosting, having the game with the 75+ players split up into 2-3 linked worlds may have fixed the issues.
In response to Falacy (#8)
Falacy wrote:
Schnitzelnagler wrote:
6000 on a single server sounds a bit much.
Even if WoW has a highly specific optimized network protocol, what's the specification of these servers?

CPU: Intel Dual Quad Core Xeon
RAM: 4 GB
1000 mbit Connection

The reason why WoW servers can support thousands of players and your server struggles to support over 100 is simple.
A "server" in WoW is actually a server farm. It is not one physical machine, it is a lot, I wouldn't be suprized if it totalled in the hundreds. They also probably use some god like internet connection that is faster than anything you could ever get. And WoW is probably optimized in such a way that certain things are done client side to minimize the stress on the network.

I'm not saying BYOND couldn't do with optimizing in this area (I don't know, I've never had problems with it). But trying to compare it to a commercial MMORPG is craziness.
There is a very good reason they pay millions and millions of dollars a year just to keep a game up and running. And that is simply so the games servers can support such a large number of players.
I think everyone who responded to this post so far missed the entire purpose of his post. It was not to talk about his connection speed. Or any of the other stuff but to try and figure out if it would be possible to increase the speed of the networking in byond. It goes without saying that byond wont be able to handle the 1000s+ players like all the commercial MMORPG but having a limit of around 50 people seems a little low.

Now I agree with falacy and that maybe if we (the community) knew a little more about how it works we would either realize theres nothing that can be done or has suggestions that could be implemented.

Now one thing that I think would be a nice feature and allow customization of games to allow more players on one world would be a seemless transfers between servers with customizable loading screens. Alathon has a system right now but its not seemless cause you still have the switching between servers.

But again the big thing that would either shut up all the request (me included) or allow for help in solving them is if we knew a little about how it works.
In response to NightJumper88 (#16)
There isn't an inherent limit in it, aside from the fact the server needs to track and process some more user actions from the thin-client. Spike seems to suggest idle-RPG comfortably handled 100+ soon after release on a single instance, I'm sure Naruto GOA can provide us with 60 - 70 (struggling CPU at that point), same goes for NBOLTS. In both cases there, we are also talking people who (in the grand scheme of things) are casual programmers. I'm sure they do stand reasonably in the BYOND field for programming knack, but they just don't clock the regular hours and years of experience that potential other BYOND programmers would. I suppose what I'm saying is (and I mean it in the nicest possible way), I suspect there is significant room for improvement in the code-base of these high-user games and that should be explored before we go tweaking BYOND itself.
The problem, I suspect, is due to BYOND being single-threaded. When you transfer files to players, they are being broken into chunks and sent piecewise while processing the remainder of your gameflow logic. The benefit is that the game doesn't grind to a halt when any single operation (such as file transfer) takes a long time. The downside is that even in a fairly idle state individual operations will take longer because of the extra overhead to do this "chunking" (which occurs within server ticks).

When you transfer a file via FTP, that process is doing all it can to use 100% of the cpu. The OS will internally load-balance to allow other processes to function, but the individual process will take as many resources as it can. It can do this because it only has one task to accomplish, unlike BYOND which has to worry about multiple players and game logic.

We will have to run some tests, similar to yours, to see the extent of this lag, since we may be able to improve the behavior (I admit that your stats don't speak very well of the system). At the least, a single-player game with little going on shouldn't be taking so much time (vs FTP) to transfer data.

One thing we could (and should) do is at least offload the file transfer to a separate thread. This would particularly benefit servers with multiple cores since oftentimes a secondary cpu is doing nothing while the first is cranking away at 100%. We'll do some research on that to see what it takes-- I suspect it's not a drastic change since the system basically works as a simulated multi-process already.

From the developer's perspective, the best thing to do to accomodate large volumes of players is, as others have mentioned, using link() and multiple servers. In recent updates we should have removed any of the old prompts that bugged players when they were moving between servers, so the experience is seamless. Correct me if I'm wrong about that and we will fix it.
In response to Tom (#18)
The Magic Man wrote:
The reason why WoW servers can support thousands of players and your server struggles to support over 100 is simple.
A "server" in WoW is actually a server farm. It is not one physical machine, it is a lot, I wouldn't be suprized if it totalled in the hundreds. They also probably use some god like internet connection that is faster than anything you could ever get.

If you had actually read my posts you would see that I am not referring to official retail servers of WoW, but rather Privately hosted ones. Burning WoW for example hosted 4 realms (basically 4 servers) off of 2 physical machines with the specs I mentioned in my previous post. I've never seen their servers with less than 500 people on them, and at peak hours they have between 3,000 and 6,000 online. Not to mention, the 3 retail WoW servers that I've played on were all completely deserted compared to the private ones, so I see no reason why they would need a server farm to run a single server.


Stephen001 wrote:
Spike seems to suggest idle-RPG comfortably handled 100+ soon after release on a single instance

Not only is idle RPG a text based game, but the point of the game is to be idle, if there was ever any visible lag in that game, regardless of player count, it must have had horribly horribly inefficient programming

I'm sure Naruto GOA can provide us with 60 - 70 (struggling CPU at that point)

The point I'm trying to get across here is that the CPU ISN'T struggling, and yet there is still lag. I can't speak for GOA, but I know HU and BE don't tax the CPU

I suspect there is significant room for improvement in the code-base of these high-user games and that should be explored before we go tweaking BYOND itself.

I spend a decent portion of time making sure all of my current games run as efficiently as possible.
http://www.angelfire.com/hero/straygames/ByondBugs/HUCPU.png


Tom wrote:
One thing we could (and should) do is at least offload the file transfer to a separate thread. This would particularly benefit servers with multiple cores since oftentimes a secondary cpu is doing nothing while the first is cranking away at 100%. We'll do some research on that to see what it takes-- I suspect it's not a drastic change since the system basically works as a simulated multi-process already.

Not sure if it helps any, but here is a screen shot from my server, at the time hosting 6 BYOND servers on daemon with over 200 players between them.
http://www.angelfire.com/hero/straygames/ByondBugs/ ServerTasks.png
Page: 1 2