Erlang decode binary data from packet - erlang
I get an UDP packet, like so:
<<83,65,77,80,188,64,171,138,30,120,105,0,0,0,10,0,4,0,0,0,84,101,115,116,15,0,0,0,82,101,122,111,110,101,32,82,111,108,101,80,108,97,121,11,0,0,0,83,97,110,32,65,110,100,114,101,97,115>>
How can I decode packet if I know that I can remove first 11 bytes, and the 12-13 byte contains amount of players online on the server (Byte width is 2), how can I get this amount?
UPD
Maybe I send incorrect packet...
SAMP Query
So, I send:
<<$S,$A,$M,$P,188,64,172,136,7808:16,$i>>
For server 188.64.172.136:7808, and I get
<<83,65,77,80,188,64,172,136,30,128,105,0,0,0,10,0,4,0,0,0,84,101,115,116,15,0,0,0,82,101,122,111,110,101,32,82,111,108,101,80,108,97,121,11,0,0,0,83,97,110,32,65,110,100,114,101,97,115>>
You can use the bit syntax and clever pattern matching to get the result:
<<_:11/bytes, NumberOfPlayers:16/integer-big, _/binary>> = <<83,65,77,80,188,64,171,138,30,120,105,0,0,0,10,0,4,0,0,0,84,101,115,116,15,0,0,0,82,101,122,111,110,101,32,82,111,108,101,80,108,97,121,11,0,0,0,83,97,110,32,65,110,100,114,101,97,115>>,
NumberOfPlayers.
If your packet binary is stored in P, you can do something like (assuming big endian):
<<NumberOfPlayersOnline:16/big>> = binary:part(P,11,2).
The result is in NumberOfPlayers.
Related
JSON data exceeds aws kinesis firehose put_record limit, is there a work around?
I'm fetching streaming data from an API and then sending the raw data to S3 bucket via Kinesis Firehose. Occasionally, the data size exceeds the limit I can send through firehose, so I get the following error botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the PutRecord operation: 1 va lidation error detected: Value at 'record.data' failed to satisfy constraint: Member must have length less than or equal to 1024000 What is the best way to work around this so I end up with something resembling the original structure? I was thinking some sore of buffering/chunking or should I just write to file and push directly into S3?
I figured it out, found the following statement in the API docs:
Kinesis Data Firehose buffers records before delivering them to the destination. To disambiguate the data blobs at the destination, a common solution is to use delimiters in the data, such as a newline (\n) or some other character unique within the data. This allows the consumer application to parse individual data items when reading the data from the destination.
So I realized I can send json string in chunks of <= 1000 kB and then send the last chunk ending with '\n' to close the buffer to ensure the original complete data structure is intact.
I then implemented the function below that checks the size of the json string and if within the size limits processes the whole data. If not, then send in chunks via put_record_batch().
def send_to_firehose(json_data: str, data_name: str, verbose=False):
if len(json_data) > 1024000:
# send json_data in chunks of 1000000 bytes or less
start = 0
end = 1000000
chunk_batch = list()
while True:
chunk_batch.append({'Data': json_data[start:end]})
start = end
end += 1000000
if end >= len(json_data):
end = len(json_data) + 1
chunk_batch.append({'Data': json_data[start:end] + '\n'})
firehose_batch(
client=AWS_FIREHOSE_CLIENT, data_name=data_name,
records=chunk_batch, verbose=verbose
)
break
else:
record = {'Data': json_data + '\n'}
firehose_put(
client=AWS_FIREHOSE_CLIENT, data_name=data_name,
record=record, verbose=verbose
)
CocoaAsyncSocket processing hex values
I'm using CocoaAsyncSocket to process a fixed length header message. My node implementation is as follows var output = "\xA5\xA5" + jsonStr; socket.write(output); I intend to read the header first (\xA5\xA5) and then process the body of the message. The problem that I have is that I've noticed that A5 takes 4 bytes to be represented in iOS. So when I read 2 bytes I receive c2a5. CocoaAsync reading code socket?.readDataToLength(2, withTimeout: -1, tag: 1) What am I doing wrong here ?. I would like to keep a fixed length header on my server side. Any help is much appreciated.
Erlang get string from UDP packet
I have this udp packet: P = <<83,65,77,80,188,64,172,85,30,144,105,0,0,0,50,0,7,0,0,0,115,97,109,112,45,114,112,11,0,0,0,149,78,87,149,82,80,149,118,50,46,50,11,0,0,0,83,97,110,32,65,110,100,114,101,97,115>> 14-15 byte is the players var (Byte width - 2) 15-18 byte of it is the length of the server hostname (Byte width - 4) 19 + strlen is the hostname of server (Byte width - strlen) I get players var so: <<_:11/bytes, Players:16/integer-big, Max:16/integer-big, _/binary>> = P. It's 50. How can I get the hostname?
You can improve the expression to obtain the correct values. Note that server length, as you put it, is 32 bits, and, by the look of it, it seems that it is little endian, not big endian (note how the name is 7 bytes, in this case "samp-rp", and the coding of these bytes is <<7,0,0,0>>, which indicates little endian (maybe your players are also little endian). Also, your numbers seem a little bit off. The expression would then be: <<_:14/bytes, Players:16/integer-little, HNameLength:32/integer-little, HostNameBinary:HNameLength/binary, _/binary>> = P. Then, the host name can be converted to a string from the binary with binary_to_list: HostName = binary_to_list(HostNameBinary).
How to manual set minimal value for dynamic buffer in Netty 3.2.6? For example 2048 bytes
I need to receive full packet from other IP (navigation device) by TCP/IP.
The device has to send 966 bytes periodically (over one minute), for example.
In my case first received buffer has length 256 bytes (first piece of packet), the second is 710 bytes (last piece of packet), the third is full packet (966 bytes).
How to manual set minimal value for first received buffer length?
This is piece of my code:
Executor bossExecutors = Executors.newCachedThreadPool();
Executor workerExecutors = Executors.newCachedThreadPool();
NioServerSocketChannelFactory channelsFactory =
new NioServerSocketChannelFactory(bossExecutors, workerExecutors);
ServerBootstrap bootstrap = new ServerBootstrap(channelsFactory);
ChannelPipelineFactory pipelineFactory = new NettyServerPipelineFactory(this.HWController);
bootstrap.setPipelineFactory(pipelineFactory);
bootstrap.setOption("child.tcpNoDelay", true);
bootstrap.setOption("child.keepAlive", true);
bootstrap.setOption("child.receiveBufferSizePredictorFactory",
new FixedReceiveBufferSizePredictorFactory(2048)
);
bootstrap.bind(new InetSocketAddress(this.port));
No matter what receiveBufferSizePredictorFactory you specify, you will see a message is split into multiple MessageEvents. It's because TCP/IP is not a message-oriented protocol but a stream-oriented one. Please read the user guide that explains how to write a proper decoder that deals with this common issue.
TCP/IP Client / Server commands data
I have a Client/Server architecture (C# .Net 4.0) that send's command packets of data as byte arrays. There is a variable number of parameters in any command, and each paramater is of variable length. Because of this I use delimiters for the end of a parameter and the command as a whole. The operand is always 2 bytes and both types of delimiter are 1 byte. The last parameter_delmiter is redundant as command_delmiter provides the same functionality. The command structure is as follow: FIELD SIZE(BYTES) operand 2 parameter1 x parameter_delmiter 1 parameter2 x parameter_delmiter 1 parameterN x ............. ............. command_delmiter 1 Parameters are sourced from many different types, ie, ints, strings etc all encoded into byte arrays. The problem I have is that sometimes parameters when encoded into byte arrays contain bytes that are the same value as a delimiter. For example command_delmiter=255.. and a paramater may have that byte inside of it. There is 3 ways I can think of fixing this: 1) Encode the parameters differently so that they can never be the same value as a delimiter (255 and 254) Modulus?. This will mean that paramaters will become larger, ie Int16 will be more than 2 bytes etc. 2) Do not use delimiters at all, use count and length values at the start of the command structure. 3) Use something else. To my knowledge, the way TCP/IP buffers work is that SOME SORT of delimiter has to be used to seperate 'commands' or 'bundles of data' as a buffer may contain multiple commands, or a command may span multiple buffers.. So this BinaryReader / Writer seems like an obvious candidate, the only issue is that the byte array may contain multiple commands ( with parameters inside). So the byte array would still have to be chopped up in order to feel into the BinaryReader. Suggestions? Thanks.
The standard way to do this is to have the length of the message in the (fixed) first few bytes of a message. So you could have the first 4 bytes to denote the length of a message, read those many bytes for the content of the message. The next 4 bytes would be the length of the next message. A length of 0 could indicate end of messages. Or you could use a header with a message count. Also, remember TCP is a byte stream, so don't expect a complete message to be available every time you read data from a socket. You could receive an arbitrary number of bytes at ever read.