I'm trying to figure out a simple way to handle reliability for UDP messages. I figured I would just send each one with a sequencing ID and by comparing the ID to the one previously received, a loss can be detected. I would normally just use integers however the idea that it would just keep incrementing indefinitely did not sit right with me.
I could use base64, but that would only make it a bit more readable but doesn't really solve anything.
I also considered prefixing a date stamp however that would be kind of sloppy when dealing messages received around midnight.
I feel like there must a better solution that someone could suggest, even if that is to just stick with integers.
My preference for this particular job is to use an incrementing (at least 64-bit) integer sequence seeded with a high-resolution timestamp. In this way, even if there is a loss of state at the sending end, when the sequence is re-seeded from the time, it will in all likelihood simply jump forward. Any Year 10K bugs that this might introduce are left as an exercise for Lazarus Long. :-)
Keep in mind that sequence-gap detection is essentially an optimization. The transmitting end needs to retransmit until an ack is received, with a nack (for a gap or corrupted datagram) simply eliciting earlier retransmission. (ZMODEM is a rare exception to this rule, with its default mode of operation being the use of a single ack at the end of the stream and all other retransmissions governed by nacks; as a file transfer protocol, however, it is essentially one giant multipart datagram.)
Use TCP? This is why TCP is different to UDP
I don't mean to be sarcastic, but this is why TCP is there.
Related
I am learning server development with IO Completion Ports. My book, "Network Programming for Microsoft Windows - Second Edition", states the following:
With every overlapped send or receive operation, it is probable that
the data buffers submitted will be locked. When memory is locked, it
cannot be paged out of physical memory. The operating system imposes a
limit on the amount of memory that may be locked. When this limit is
reached, overlapped operations will fail with the WSAENOBUFS error. If
a server posts many overlapped receives on each connection, this limit
will be reached as the number of connections grow. If a server
anticipates handling a very high number of concurrent clients, the
server can post a single zero byte receive on each connection. Because
there is no buffer associated with the receive operation, no memory
needs to be locked. With this approach, the per-socket receive buffer
should be left intact because once the zero-byte receive operation
completes, the server can simply perform a non-blocking receive to
retrieve all the data buffered in the socket's receive buffer. There
is no more data pending when the non-blocking receive fails with
WSAEWOULDBLOCK.
Now, I'm trying to understand this paragraph; I think I've got it but want to make sure please.
I understand about memory being locked if I post make multiple WSARecv() calls with large buffers. But I am not entirely sure how a zero byte buffer prevents this.
I am thinking it is this (and would like confirmation please):
If I have n connections, and I post 50 WSARecv() calls with a 1KB buffer on each connection, that is n * 50KB total memory locked. All of that memory is locked, regardless of whether or not it is actually being used (i.e. whether or not anything is being copied into it from the TCP buffers). Hence if I keep adding more connections, I will keep locking more memory that may or may ever be used. Thus I can run out, with WSAENOBUFS error.
If I however post a zero byte receive on each connection, a completion packet will be generated on that connection only when there is data available for reading. (That is my first assumption, is that correct?)
Now, when I know there is some data, I can then post a WSARecv() with a buffer of 1KB (or however much) - or indeed loop repeatedly reading it all as suggested in my book - knowing that it will be filled immediately hence not remain unused and locked (second assumption, is that correct?)
Question 1
Thus, if my two assumptions are correct, then I have understood my book :) This means then that my server could, in theory, post a zero byte receive when a new connection is established, then when a completion packet is generated, read all of the data until there is no more, then post another zero byte receive - is that correct?
Question 2
However, isn't there still a risk that if I receive completion packets for lots of my zero byte receive posts at once, and I then go onto make multiple WSARecv() calls, that I will still end up with some failing with WSAENOBUFS?
Hopefully someone can clarify these two assumptions and two questions for me.
OK I've done research into this along with experimentation and have found the following:
Assumptions:
1) Assumption 1 is correct.
2) I believe assumption 2 is correct.
Questions
1) I have tested this and this seems to work.
2) This I guess remains a possibility but much less likely than if I posted receives with a none-zero buffer.
Note that we can still raise the WSAENOBUF error when sending too fast; more details here.
I was reading the article on gafferon about network programming, and it explains the advantages and disadvantages of UDP vs TCP.
However, It feels like that only works for a game that requires constant streams of input. I'm developing a card game, and I was wondering what would be the most effective way to create the multiplayer features, UDP or TCP. I feel like UDP is probably still the best pick just for its speed, but that raises further questions.
If I use UDP, on the low chance that a packet gets loss, how do I know it got lost and how do I recover from the loss?
How to detect the loss of a message?
The receiver needs to send something back to the original sender that tells him it received the message. If the original sender gets no acknowledgement during a defined timeframe, it will resend the message.
Another method is to send each message multiple times per default and let the receiver ignore duplicates.
The rules for this can get arbitrarily complex and in the end you'll reimplement parts of TCP. You might also want to ensure ordering of the messages.
UDP is good for frequent updates where it's not that bad to lose some messages (e.g. delivering a video stream or keeping player positions in sync in a first person shooter). A card game is slower-paced but requires reliable, ordered messages. If you don't plan to host game sessions with multiple thousands of players, use TCP. It's fast enough and much easier to work with.
It only works with constant streams of input?
TCP works fine even if you only send one message every minute. The term "stream oriented" basically means that the receiver doesn't know where a message ends. You'll have to provide that information in your protocol by prepending the message length for example.
If you choose UDP you'd be better off using a library on top of it.
Let's say I have some record like {my_table, Id, Value}.
I constantly overwrite the value so that it holds consecutive integers like 1, 2, 3, 4, 5 etc.
In a distributed environment, is it guaranteed that my event listeners will receive all of the values? (I don't care about ordering)
I haven't verified this by reading that part of the source yet, but it appears that sending a message out is part of the update process, so messages should always come out, even on very fast changes. (The alternative would be for Mnesia to either queue messages or queue changes and run them in batches. I'm almost positive this is not what happens -- it would be too hard to predict the variability of advantageous moments to start batching jobs or queueing messages. Sending messages is generally much cheaper than making a change in the db.)
Since Erlang guarantees delivery of messages to a live destination this is as close to a promise that every Mnesia change will eventually be seen as you're likely to get. The order of messages couldn't be guaranteed on the receiving end (as it appears you expect), and of course a network failure could make a set of messages get missed (rendering the destination something other than live from the perspective of the sender).
Short question but for me its difficult to understand.
Why exactly does ePoll scale better than Poll?
While Damon's reason is correct for the unusual case where you never block on a socket, in typical real-world programs, the reason is completely different. A typical program looks like this:
1) Do all the work we can do now.
2) Check if any network connections need service, blocking if there's nothing to do.
3) Service any network connections discovered.
4) Go to step 1.
Typically, because you just did all the work you can do, when you come around to step 2, there is no work for you to do. So you'll have to wait a bit. Now, imagine there are 800 sockets you are interested in. The kernel has to put on the wait queue for each of those 800 sockets. And, a split-second later when data arrives on one of those 800 sockets, the kernel has to remove you from those 800 wait queues. Placing a task on a wait queue requires creating a 'thunk' to link that task to that wait queue. No good optimizations are possible because the kernel has no idea which 800 sockets you'll be waiting for.
With epoll, the epoll socket itself has a wait queue, and the process is only put on that one single wait queue. A thunk is needed to link each of the 800 connections to the epoll wait queue, but that thunk is persistent. You create it by adding a socket to an epoll set, and it remains there until you remove the socket from the set.
When there's activity on the socket, the kernel handles it in the task that detects the activity. When you wait, the kernel already knows if there's a detected event and the kernel only has to put you on that one wait queue. When you wake, it only has to remove you from that one queue.
So it's not so much the copying that's the killer with select or poll, it's the fact that the kernel has to manipulate a massive number of wait queues on each blocking operation.
The poll system call needs to copy your list of file descriptors to the kernel each time. This happens only once with epoll_ctl, but not every time you call epoll_wait.
Also, epoll_wait is O(1) in respect of the number of descriptors watched1, which means it does not matter whether you wait on one descriptor or on 5,000 or 50,000 descriptors. poll, while being more efficient than select, still has to walk over the list every time (i.e. it is O(N) in respect of number of descriptors).
And lastly, epoll can in addition to the "normal" mode work in "edge triggered" mode, which means the kernel does not need keep track of how much data you've read after you've been signalled readiness. This mode is more difficult to grasp, but somewhat more efficient.
1As correctly pointed out by David Schwartz, epoll_wait is of course still O(N) in respect of events that occur. There is hardly a way it could be any different, with any interface. If N events happen on a descriptor that is watched, then the application needs to get N notifications, and needs to do N "things" in order to react on what's happening.
This is again slightly, but not fundamentally different in edge triggered mode, where you actually get M events with M <= N. In edge triggered mode, when the same event (say, POLLIN) happens several times, you will probably get fewer notifications, possibly only a single one. However, this doesn't change much about the big-O notation as such.
However, epoll_wait is irrespective of the number of descriptors watched. Under the assumption that it is used in the intended, "normal" way (that is, many descriptors, few events), this is what really matters, and here it is indeed O(1).
As an analogy, you can think of a hash table. A hash table accesses its content in O(1), but one could argue that calculating the hash is actually O(N) in respect of the key length. This is technically absolutely correct, and there probably exist cases where this is a problem, however, for most people, this just doesn't matter.
When I was writing a simple server for a simple client <> server multiplayer game, I thought of the following text-based protocol using a translation library. Basically, each command had a certain meaning, eg:
1 = character starts turning right
2 = character starts turning left
3 = character stops turning
4 = character starts moving forward
5 = character stops moving
6 = character teleports to x, y
So, the client would simply broadcast the following to inform that the player is now moving forward and turning right:
4
1
Or, to teleport to 100x200:
6#100#200
Where # is the parameter delimiter.
The socket connection would be connected to the player identifier, so that no identifier has to be broadcasted with the protocol to know what player the message belongs to.
Of course all data would be validated server side, but that is a different subject.
Now, this seems pretty efficient to me, only 2 bytes to inform the server that I am moving forward and turning right.
However, most "professional" code snippets I saw seemed to be sending objects or xml commands. This seems to require a lot more server resources to me, doesn't it?
Is my unexperienced logic of why my text based protocol would be efficient flawed? Or what is the recommended protocol for real-time action multiplayer games?
I want to setup a protocol that is as efficient as possible, because I do not want to use multiple clusters/servers to cover excessive amounts of bandwidth for my 2D multiplayer game, and to safe synchronization problems and hassle.
However, most "professional" code
snippets I saw seemed to be sending
objects or xml commands. This seems to
require a lot more server resources to
me, doesn't it?
Is my unexperienced logic of why my
text based protocol would be efficient
flawed? Or what is the recommended
protocol for real-time action
multiplayer games?
Plain text is more expensive to send than a binary format containing the same information. For example, if you only send 1 byte, you can only send 10 different commands, digits 0 to 9. A binary format can send as many different commands as there are different values you can fit into a byte, ie. 256.
As such, although you are thinking of objects as being large, in actual fact they are almost always smaller than the plain text representation of that same object. Often they are as small as is possible without compression (and you can always add compression anyway).
The benefits of a plain text format are that they are easy to debug and understand. Unfortunately you lose those benefits if you put your own encoding in there (eg. reducing commands down to single digits instead of readable names). The downside is that the format is bigger, and that you have to write your own parser. XML formats eliminate the second problem, but they can't compete with a binary format for pure efficiency.
You are probably overthinking this issue at this stage, however. If you're only sending information about events such as the commands you mention above, bandwidth will not be a concern. It's broadcasting information about the game state that can get expensive - but even that can be mitigated by being careful who you send it to, and how frequently. I would recommend working with whatever format is easiest for now, as this will be the least of your problems. Just make sure that your code is always in a state where you can change the message writing and reading routines later if you need.
You need to be aware of the latency involved in sending your data. "Start turning"/"stop turning" will be less effective if the time between the receipt of those packets is different than the time between sending them.
I can't speak for all games, but when I've worked on this sort of code we'd send orientation and position information across the wire. That way the receiver could do smoothing and extrapolation (figure out where the object should be "now" based on data that I have that is already known to be old). Different games will want to send different data, but generally speaking you will need to figure out how to make the receiver's display of the data match the sender's, so you'll need to send data that is resilient in the face of networking problems.
Also, many games use UDP for this sort of data transfer instead of TCP. UDP is unreliable, so you may not get all of your packets. That means that "stop moving now" or "start moving now" may not be received in pairs. When coding on top of UDP then it's even more important to send "this is the state right now" every so often so that clients get ample opportunity to sync up.
The common way is to use a binary format, not text, not xml. So with only one byte you can represent one of 256 different commands.
Also use UDP and not TCP. The game will be a lot more responsive with UDP in case of packet loss. In case of packet loss you can still extrapolate the movements. With each packet send a packet number so that the server knows when the command was sent.
I highly recommend that you download the Quake source code where you can learn more about network programming in modern multiplayer games. It's really easy to read and understand.
edit:
I almost forgot..
Google's Protocol Buffers can be of great help when sending complex data structures.
I thought I would give my two cents and provide a practical application to what is being referred to as Binary Serialization. The concept is actually incredibly simple, yet only seems complicated on the outside.
You can actually send XMLs and have a server that processes the data within the XML to different functions within the server itself. You can also just send the server a single number that is stored within the server as a variable. After that, it can process the rest of the data and choose the correct course of actions.
As an example, some rough code:
private const MOVE_RIGHT:int = 0;
private const MOVE_LEFT:int = 1;
private const MOVE_UP:int = 2;
private const MOVE_DOWN:int = 3;
function processData(e:event.data)
{
switch (e)
{
case MOVE_RIGHT:
//move the clients player to the right
case MOVE_LEFT:
//move the clients player to the left
case MOVE_UP:
//move the clients player to the up
case MOVE_DOWN:
//move the clients player to the down
}
}
This would be a very simple example, and would need to be modified but as you can see you merely just store the variables encoded with whole numbers that you transmit in strings of numbers. You can parse these and create headers of information to organize them into different sections of data that needs to be transmitted.
Also, it is better to do a UDP setup for games because just missing a packet should NOT halter the gaming experience, but instead should be able to handle it client-side AND server-side.