ID:136255
 
I'm curious about something. On one end of the spectrum, we have everyone complaining about the 65,535 limit for various internal IDs and internal arrays in the game. On the other end of the spectrum, we have the problem of bandwidth that would result from increasing the bytes required from 2 to 4, resulting in exponentially more transmission.

What if a compromise is reached by having the client-server communication handler to transmit shorts until the object count breaches 65,500, after which point it transmits longs? (Technically, it wouldn't have to resort to a long until 65,534, but it's always good to play it safe.)

The IDs would be internally stored as longs, but would be converted to shorts to send to the client provided they were less than the magical 65,500. Likewise, the server would receive the shorts and internally convert them to longs to compare against the master object index.

When the number finally breaches 65,500, the client would only need to receive some sort of message telling it to expect long identifiers instead of short identifiers, and then everything would work peachy keen on both the client's end and the server's end. When the object count falls back below 65,500, the client receives a message telling it to expect short identifiers instead.


In other words, games that need the extra amount of objects will have it at the necessary cost in bandwidth, while games that do not will not have such a penalty. (Unfortunately, all games would see doubled memory usage, but given the amounts of RAM that are typical these days, that's not really a problem.)


The one problem I can see with this system is if the packet containing the message telling the client to expect long identifiers were to get dropped, and the server then send long identifiers to the client when it is still expecting short ones. Visible objects would look seriously strange.
Spuzzum wrote:
I'm curious about something. On one end of the spectrum, we have everyone complaining about the 65,535 limit for various internal IDs and internal arrays in the game. On the other end of the spectrum, we have the problem of bandwidth that would result from increasing the bytes required from 2 to 4, resulting in exponentially more transmission.

I don't think this would be as serious a concern if the RLE encoding was replaced with gzip encoding. I didn't realize until recently that RLE was as compressed as the data got. No wonder screen objects are so slow--they're sending entire strings uncompressed!

What if a compromise is reached by having the client-server communication handler to transmit shorts until the object count breaches 65,500, after which point it transmits longs? (Technically, it wouldn't have to resort to a long until 65,534, but it's always good to play it safe.)

The IDs would be internally stored as longs, but would be converted to shorts to send to the client provided they were less than the magical 65,500. Likewise, the server would receive the shorts and internally convert them to longs to compare against the master object index.

When the number finally breaches 65,500, the client would only need to receive some sort of message telling it to expect long identifiers instead of short identifiers, and then everything would work peachy keen on both the client's end and the server's end. When the object count falls back below 65,500, the client receives a message telling it to expect short identifiers instead.
...
The one problem I can see with this system is if the packet containing the message telling the client to expect long identifiers were to get dropped, and the server then send long identifiers to the client when it is still expecting short ones. Visible objects would look seriously strange.

Doing this with on/off packets doesn't seem like a great idea. (Actually you don't have to worry about the packet being dropped because this is TCP/IP, and the packet would have to precede the other data.) It makes more sense to send out a single-bit flag along with the data.

However probably the larger issue is that the entire packet could probably be reworked. I really have no idea how it's constructed now but I bet I could develop a very bandwidth-friendly alternative if I knew how it was supposed to work.

Many of the ordinary numbers handled by BYOND, particularly by the client, do not have to be 4 bytes; visibility settings could be compared with just one byte, for example. Object references could be kept to 2 bytes, or expanded to 3 or 4 when necessary. And it wouldn't especially surprise me if vars are sent with strings for their names, whereas it would make more sense to send a hash or numerical ID and allow the client to ask "what is that?" in response for a list of the actual strings.

Lummox JR
In response to Lummox JR
Lummox JR wrote:
Spuzzum wrote:
I'm curious about something. On one end of the spectrum, we have everyone complaining about the 65,535 limit for various internal IDs and internal arrays in the game. On the other end of the spectrum, we have the problem of bandwidth that would result from increasing the bytes required from 2 to 4, resulting in exponentially more transmission.

I don't think this would be as serious a concern if the RLE encoding was replaced with gzip encoding. I didn't realize until recently that RLE was as compressed as the data got. No wonder screen objects are so slow--they're sending entire strings uncompressed!

Hmm... see also my other thread, then.


Doing this with on/off packets doesn't seem like a great idea. (Actually you don't have to worry about the packet being dropped because this is TCP/IP, and the packet would have to precede the other data.) It makes more sense to send out a single-bit flag along with the data.

However probably the larger issue is that the entire packet could probably be reworked. I really have no idea how it's constructed now but I bet I could develop a very bandwidth-friendly alternative if I knew how it was supposed to work.

Many of the ordinary numbers handled by BYOND, particularly by the client, do not have to be 4 bytes; visibility settings could be compared with just one byte, for example. Object references could be kept to 2 bytes, or expanded to 3 or 4 when necessary. And it wouldn't especially surprise me if vars are sent with strings for their names, whereas it would make more sense to send a hash or numerical ID and allow the client to ask "what is that?" in response for a list of the actual strings.

Expanding it to 3 or 4 would take a two-bit operation. Teehee.
In response to Lummox JR
Lummox JR wrote:
Spuzzum wrote:
I'm curious about something. On one end of the spectrum, we have everyone complaining about the 65,535 limit for various internal IDs and internal arrays in the game. On the other end of the spectrum, we have the problem of bandwidth that would result from increasing the bytes required from 2 to 4, resulting in exponentially more transmission.

I don't think this would be as serious a concern if the RLE encoding was replaced with gzip encoding. I didn't realize until recently that RLE was as compressed as the data got. No wonder screen objects are so slow--they're sending entire strings uncompressed!

This piqued my curiosity so I performed a little test. I let Una run in automatic mode for a couple hours yesterday and captured all of the network traffic between the client and server. I then took each individual packet carrying actual data and compressed it using gzip. I had a feeling that because the vast majority of client-server messages are very small, gzip would be entirely wasted on them, and would likely add bytes, not remove them.

I was right. Individually compressing over 44,000 packets resulted in an average increase of 31% in size. Of course, compressing the whole thing at once reduced the data to 1/4 of its original size, but that's hardly optimal! If you wait to accumulate enough data that compression is useful, then you've introduced lag by waiting.

At best, BYOND could either test each packet and send it either compressed on uncompressed depending on which is smaller, or try to do something like compressing only text strings before encoding them and sending them on their way. I still remain skeptical as to the real world benefit of this, but it would certainly help in extreme cases.

All that said, I could still be mistaken on my impression about BYOND's current usage of compression. Dan's probably the one person who really knows for sure.

As for the original question, I also have my doubts as to whether bandwidth is an issue regarding 2-byte integers vs. 4-byte. This is, of course, pure speculation, but I wonder just how many objects you would need on the screen at once to cause noticeable extra lag due to the extra 2 bytes per data item.

I would guess that the biggest issue is simply backwards compatibility in the network protocol, as Dan said way back in [link]. Of course, nearly two years later that change is probably long overdue. Unfortunately again, Dan is the only one who is intimately familiar with that code and knows all of the pitfalls to watch out for when making such a change. I hear he's still quite busy these days...