I'm using mbedtls to run SSL over TCP. I'm using my own custom I/O functions for the interface between mbedtls and TCP, set using the mbedtls_ssl_set_bio call.
I've noticed that Application records are generated one-to-one with calls to mbedtls_ssl_write. For example, if I write data in 52-byte chunks, the client sends 52-byte records. If I write data in 520-byte chunks, then the client sends 520-byte records. The latter case is preferable; in the former case, a significant amount of bandwidth is lost to TLS headers.
My question is: is it possible to configure mbedtls to "coalesce" multiple small writes into a single record? For example, TCP does this (e.g., if you write data in 52-byte chunks to a TCP connection in quick succession, they do eventually get coalesced into MSS-sized segments). I'm wondering if it's possible to do something similar for TLS.
With TCP, when the write function returns, the data has written to the socket and is out of control of the application. The network stack can still coalesce packets if it wants. Either the sender, or a router, can merge successive TCP packets on a connection if the second packet arrives before the previous one can be sent.
With TLS, when the mbedtls_ssl_write function returns (or similar function in any TLS stack), the data has been encrypted-and-authenticated into a TLS record and the record has been sent over the socket (or whatever the underlying I/O function does). The TCP stack can still coalesce the TCP packets, but it can't coalesce TLS records because it doesn't have the necessary cryptographic keys.
If you want to coalesce data on a TLS connection, either you need to buffer it before you pass it to the TLS stack, or the TLS stack needs to buffer it itself. If you want the TLS stack to buffer it, you need an API with at least two functions: one to take some data as input and not try to send it immediately, and another one to say “send the data now!”. The first function would not really do anything except write the data into some temporary buffer, unless the buffer is full in which case it would need to flush it first. And that's something you can do yourself: just memcpy the data to a temporary buffer, and call mbedtls_ssl_write on that temporary buffer when you're ready to send. So there's not really an incentive for a TLS stack to provide this interface.
Related
Is it possible to modify users' HTTP request to
www.example.com/options
instead of
www.example.com/options_and_params
My scenario is that about 30000 users connect to my company's network backbone and I want to add one or more server (with the code I'm current working on) between the backbone switches and Radware LoadProof to achieve this work.
After googling all the night, I have no lead but some more questions:
I don't need to intercept every packet through the network. With some helps like iptables, I can filter out the package I want. I have done it before using iptables. However, packet is not equal to HTTP stream. Do I need to do HTTP re-construct?
If I successfully find a way to modify HTTP request URL content, I still should put it back to network stream. As I know TCP packets have a checksum and after I modify the content it must be wrong. How do I calculate a new checksum and put the packet back to network?
It's my first time to do network programming or packet processing develop. Any suggestion is appreciate.
This depends on whether you are doing HTTP/1.0 or HTTP/1.1 and whether its an initial request you need to modify or all requests in a single HTTP 1.1 session.
If you have the packet and can modify it before it is sent on and you are trying to modify just the request then given the length of a typical packet and the location of the URL in the HTTP request stream (very near the beginning) and the fact that it will be the first thing sent in the TCP stream I think you can fairly safely assume that it will be present in the first N bytes of the first packet sent and therefore won't be split over multiple packets.
However, if this is an HTTP/1.1 stream then multiple requests will be being sent via the same TCP connection in which case in future requests the URL may well be split over two TCP packets.
If you can maybe force HTTP/1.0 or possibly if you modify the initial or all requests to be HTTP/1.0 then you can be pretty sure that the first packet will correspond to the first packet of the TCP stream and that you are very unlikely to see the URL split over multiple packets, meaning no reconstruction and the ability to just do a replace.
However this will come at a cost of new TCP connections which is pretty inefficient.
If you don't and you leave it as HTTP/1.1 then the URL could be at any random point in any future request and therefore split over multiple TCP packets (two realistically given the size of the URL).
If I got your question right, then this could be probably done with some fast reverse-proxy like nginx.
First of all my intend is to create 2 programs server and client to send big byte array. I just started to test with UDPclient and udpserver components. I am not if it is right way.
I found program example but it was built with an old Delphi version, when I compile I am getting an error.
HostName.Caption := UDPServer.LocalName;
HostAddress.Caption := GStack.LocalAddress;
Port.Caption := IntToStr(UDPServer.DefaultPort);
BufferSize.Caption := IntToStr(UDPServer.BufferSize);
UDPServer.Active := True;
it seems udpserver.localname was in old versions. I need to find what property to use in here . How to get and write Host name in udpserver components. udpclient has Host property and workes fine, I could compile client program
Can anyone help me what to put instead of "localname" property
And if anyone can advice me another way to send big byte arrays. the size of it will be about 120000 . I will send it in every minute
Thank you
It's best to use TCP as it can detect data-corruption in your transmission and request the corrupted packets again. The detection happens by checking packets against a 16-bit checksum, this will only detect errors, but not allow the system to correct it, which is where requesting the data again comes in.
It's probably most convenient to use the Indy socket library that comes with Delphi to create TCP sockets. The TIdTCPServer and TIdTCPClient components should do the job nicely.
You can find some demo's on using the Indy library here:
http://www.indyproject.org/sockets/demos/index.en.aspx
http://www.atozed.com/indy/Demos/10/index.en.aspx
http://www.ciuly.com/delphi/indy/
GStack.HostName is the replacement for the old LocalName property.
UDP is a message-based transport. 120000 is too many bytes to fit in a single UDP message. You will have to break it up into small chunks. To send large amounts of data using UDP, consider using a UDP-based transfer protocol, such as TFTP. Indy has TIdTrivialFTP and TIdTrivialFTP components for that purpose.
Otherwise, switch to TCP, which is a stream-based transport. Then you can send as much data as you want.
It depends on the data you are trying to send over network. If you want to send a file over network, you should use TCP protocol because it guarantees that the data received at server-side will be the same sent on the client-side. If the data you want to send is lossy like Voice (I mean it's not important we have delays or losses in the middle and in data) you can use UDP protocol. It'll be very faster because it has no overhead for checking the data against corruption. In UDP you may loose your packets or datagrams so UDP is called a connection-less protocol because there is no connection in fact. I think all known programming languages support TCP and UDP connections.
A very simple question I can't seem to find a definitive answer for.
I have a classic TCP Indy server. I send data in chunks. Each data packet is send in 1 or more chunks. When it arrives the OnExecute is triggered one or more times for each packet. More then one client can send data at any given time. How do I know which client / packet am I receiving data for in the OnExecute? I know its a trivial question probably but I want to have a definitive answer.
If you can design the protocol, it could be done like this:
the client starts with a initial command which includes total size and the chunk size
the server OnExecute creates a temporary output file stream and stores file information in the context
the client sends the chunks
the server OnExecute reads chunks (using Indy TCP server blocking read methods with the known chunk length) and appends them to the output stream
The Indy TCP context class can be extended to add custom information for the client connection.
How does a server knows what kind of packets his receiving ? (strcut,array...) ?
PS: i know dumb question
It doesn't - its just a bunch of bits/bytes. The application, using application level protocols, decides how to interpret those bits/bytes.
Just like memory is a bunch of bits/bytes - a pointer to a struct can be forced to point anywhere and the struct can used to read the memory, but the data may be nonsensical. Your application logic ensures (hopefully) that the struct pointer is only applied to memory that contains valid struct data. Similarly, your network application must decide how to interpret what the bits in the packets or TCP stream mean.
An application may use a well-known protocol to decide how to interpret those bytes. For example, the HTTP protocol indicates what the client should transmit and the server knows to interpret that data from the client according to how the HTTP spec. Regardless of what the client sends (e.g. if a gaming client accidentally sent a binary stream to the HTTP server), the HTTP server will nevertheless try to interpret the bits as a HTTP client request.
TCP servers receive a stream of bytes. Any interpretation beyond this is up to the application logic.
I have created a raw socket which takes all IPv4 packets from data link layer (with data link layer header removed). And for reading the packets I use recvfrom.
My doubt is:
Suppose due to some scheduling done by OS, my process was asleep for 1 sec. When it woke up,it did recvfrom (with number of bytes to be received say 1000) on this raw socket (with the intention of receiving only one IPv4 packet and say the size of this packet is 380 bytes). And suppose many network applications were also running simultaneously during this time, so all IPv4 packets must have been queued in the receive buffer of this socket. So now recvfrom will return all 1000 bytes (with other IPv4 packets from 381th byte onwards) bcoz it has enough data in its buffer to return. Although my program was meant to understand only one IPv4 packet
So how to prevent this thing? Should I read byte by byte and parse each byte but it is very inefficient.
IIRC, recvfrom() will only return one packet at a time, even if there are more in the queue.
Raw sockets operate at the packet layer, there is no concept of data streams.
You might be interested in recvmmsg() if you want to read multiple packets in one system call. Recent Linux kernels only, there is no equivalent send side implementation.