I am reading a continuous data stream from an api and at times the program will freeze on the following line and eventually time out.
Private BUFFER_SIZE As Integer = 8100
...
Dim bufferread(81000) As Byte
numbytesread = responseStream.Read(bufferread, 0, BUFFER_SIZE)
I could reduce the buffersize I guess and also write more frequently to my files but I also want to make sure I create good files where data is not snipped off and I reach a delimitter which indicates end of a post. Any ideas on why this freezes up.
Related
I want read on stdout to be non-blocking. I was using the newlib-lpc library in WINarm to do this. But even though it is set as non-Blocking, the code stops at read every time and waits for a character to be received.
Here is my read line:
read(fileno(stdout), &inchar, 1);
Here is the code to set stdout to non-blocking using newlib-lpc
blocking=BLOCKING_IO_NO;
ioctl( fileno(stdout), BLOCKING_SETUP, &blocking);
When that didn't work, I decided to forget using the library and do it myself with this line
fcntl(fileno(stdout), F_SETFL,fcntl(fileno(stdout), F_GETFL) |O_NONBLOCK);
But that didn't work either.
Could anyone give me some advice?
Thanks!
edit:
Someone asked if I really meant stdout. Yes, I am resurrecting some old code, and the stdout has been set as the socket for UART communication between my board and the computer.
sp.baud = 115200uL;
sp.length = UART_WORD_LEN_8;
sp.parity = UART_PARITY_NONE;
sp.stop = UART_STOP_BITS_1;
ioctl( fileno(stdout), UART_SETUP, &sp);
irq.FIQ = 0;
irq.pri = (INT_PRIORITY)10;
ioctl( fileno(stdout), INTERRUPT_SETUP, &irq);
edit2: They used stdout this way so that printf works. When it tries to print to stdout, it gets redirected to the UART serial stream.
edit3: The error I get when calling
ioctl( fileno(stdout), BLOCKING_SETUP, &blocking);
is
errno88 function not implemented
I generate a very large .csv file from a database using the method outlined in
https://stackoverflow.com/a/13456219/141172
It works fine, up to a point. When the exported file is too large, I get an OutOfMemoryException.
If I turn off output buffering by modifying that code like this:
protected override void WriteFile(System.Web.HttpResponseBase response)
{
response.BufferOutput = false; // <--- Added this
this.Content(response.OutputStream);
}
the file download completes. However, it is several orders of magnitude slower than when output buffering was enabled (measured for the same file with buffering true/false, on localhost).
I understand that is slower, but why would it slow to a relative crawl? Is there anything I can do to improve processing speed?
UPDATE
It would also be an option to use File(Stream stream, String contentType) as suggested in the comments. However, I'm not sure how to create stream. The data is dynamically assembled based on a DB query, and a MemoryStream would run out of contiguous physical memory. Suggestions are welcome.
UPDATE 2
It was suggested in the comments that alternately reading from the database and writing to the stream is causing a degradation. I modified the code to perform the stream writing in a separate thread (using the producer/consumer pattern). There is no appreciable difference in performance.
I don't know what ASP.NET and IIS are doing exactly with output streaming but maybe too small chunks are being uses. Hook in a BufferedStream with a very big buffer, like 4MB.
According to your comments it worked. Now, tune down the buffer size to save memory and have a smaller working set. Good for cache.
As a subjective comment I'm disappointed that this is even necessary. IIS should use the right buffers automatically which is extremely easy with TCP connections.
EDIT FROM OP
Here is the code derived from this answer
public ActionResult Export()
{
// Domain specific stuff here
return new FileGeneratingResult("MyFile.txt", "text/text",
stream => this.StreamExport(stream), false);
}
private void StreamExport(Stream stream)
{
using (BufferedStream bs = new BufferedStream(stream, 256*1024))
using (StreamWriter sw = new StreamWriter(bs))
foreach (var stuff in MyData())
{
sw.Write(stuff);
}
}
In Eric's latest update, he mentioned using another thread. I too had this problem for implementing database exports. Here is some example code for the solution I used:
Handling with temporary file stream
Im having a problem while reading a stream from a Client. Im using Delphi XE3 with Indy 10.
This Client, send me a tagged text data stream, that is in XML format. My problem start when i read data with a TIdTCPServer. With this code:
if Acontext.Connection.IoHandler.InputbufferisEmpty then
begin
Acontext.Connection.IoHandler.CheckforDataonSource(1000);
Acontext.Connection.IoHandler.CheckForDisconnect;
if Acontext.Connection.IoHandler.InputbufferisEmpty then Exit;
end;
Acontext.Connection.Iohandler.ReadBytes(Abuffer,
Acontext.connection.ioHandler.Inputbuffer.size);
data := BytesToString(Abuffer, Tencoding.bigendianUnicode);
Data is encoded in BigEndianUnicode. I noticed that when im recieving data, the server read and show me the data in 2 'blocks' diferents. If the server recieve the same data all the time, when i parse with BytesToString the data get "cut" at same char everytime.
Here is a result of what im getting:
<rootNode>
<Node1></Node1>
<Node2></Node2>
<
Node3></Node3>
<Node4></Node4>
The problem get bigger, when i assign that WideString to an XMLDocument. I get an Invalid at the top level of the document error.
Sometimes, i run the server and get all the data correct, without this error, but when i close the aplication and run it again, the server recieve the data corrupted.
Sorry, for my english. I need some help with this.
You are only reading whatever raw bytes happen to be in the IOHandler.InputBuffer at the moment you access it, so you are likely not reading the entire XML, or worse you are reading more data that follows the XML. If the client is sensible, it should be sending either the XML length before the XML data, or a unique terminator after the XML data. Either of which would allow you to read the entire XML by itself in full without having to resort to calling CheckforDataOnSource() at all (which you should be avoiding in most situations), eg:
len := AContext.Connection.IOHandler.ReadLongWord; // or however the XML length is sent
data := AContext.Connection.IOHandler.ReadString(len, IndyUTF16BigEndianEncoding);
Or:
data := AContext.Connection.IOHandler.ReadLn('the terminator here', IndyUTF16BigEndianEncoding);
I posted this to the Squeak Beginners list too - I'll be sure to make sure any answers from there get here :)
I'm using Squeak 4.2 and working on the smalltalk end of a named pipe connection, which sends a message to the named pipe server with:
msg := 'Here''s Johnny!!!!'.
pipe nextPutAll: msg; flush.
It should then receive an acknowledgement, which will be a 32-byte md5 hash of the received message (which the smalltalk app can then verify). It's possible the named pipe server may have gone away or otherwise been unable to deal with the request, and so I'd like to set a timeout on reading the acknowledgement. I've tried using this:
ack := [ pipe next: 32 ] valueWithin: (Duration seconds: 3) onTimeout: [ 'timeout'. ].
and then made the pipe server pause artificially to test the code. But the smalltalk thread blocks on the read and doesn't carry on (even after the timeout), although if I then get the pipe server to send the correct response (after a 5 second delay, for example), the value of 'ack' is 'timeout'. Obviously the timeout did what it's supposed to do, but couldn't 'unblock' the blocking read on the pipe.
Is there a way to accomplish this even with a blocking FileStream read? I'd rather avoid a busy wait on there being 32 characters available if at all possible.
This one may come in handy but not on Windows I am afraid.
http://www.samadhiweb.com/blog/2013.07.27.unixdomainsockets.html
i am making a program that sends data between a server program and its clients. They all use the server and client socket components found in Delphi! I have looked on the Internet and cannot find a way on how to measure how much data has been transferred through a socket!
Any help, especially some code (pascal/Delphi), would be very much appreciated!
here you have a full example http://delphi.about.com/od/fullcodeprojects/l/aa112903a.htm
If you use win/*nix API sockets (not a special libraries) you have to count returns from recv and send functions.
total_data = 0;
...
...
get_data = recv(...)
total_data = total_data + get_data
...
send_data = send(...)
total_data = total_data + send_data
Magenta Systems has a free set of components that can monitor network traffic using either raw sockets or WinPcap.
Update:
From your comment to RBA's response: The Magenta components let you identify and differentiate between different IP addresses and services (ports).
I keep stats on all my TCP traffic to the byte. Every time I send data I update the stats and every time I receieve data I update the stats. This is accurate to one byte and requires only a few lines of code. And it does not rely on any particular TCP components. Why is doing it this way so hard?