I am unsure of what is the answer to this question:
If I wrap a BufferedInputStream around an InputStream coming from for example a http call -does the BufferedInputStream automatically and immediately fills up the buffer with data from the InputStream (at "wrapping"-time?), or do I have to call read() on the BufferedInputStream first, before the buffer gets filled up? To put it short: When is the buffer first filled?
I am asking because if I have a http call, I'd like to be able to quickly store the content of small text files in RAM (where they are safer) than relying on the connection to still be up when reading actually starts...
Will I get an IOException with BufferedInputStream if the underlying TCP connection is closed, even if the buffer managed to fill itself up with all data that was actually requested?
When the BufferedInputStream is created, an internal buffer array is created. As bytes from the stream are read or skipped, the internal buffer is refilled as necessary from the contained input stream, many bytes at a time. Until you read, it should be empty.
Fill method is called only on the first read or skip method call.
Code extract form BufferInputStream.
public synchronized int read() throws IOException {
if (pos >= count) {
fill();
if (pos >= count)
return -1;
}
return getBufIfOpen()[pos++] & 0xff;
}
Skip too have a similar check for fill operation.
Related
I am using native NT API in my application to access files (NtCreateFile/etc). In order to avoid dealing with STATUS_PENDING I am using FILE_SYNCHRONOUS_IO_NONALERT flag when opening related file. So, opening file looks like this:
UNICODE_STRING fname = toNtUnicode(ntpath);
OBJECT_ATTRIBUTES oa;
InitializeObjectAttributes(&oa, &fname, 0, at.handle(), NULL);
HANDLE h;
IO_STATUS_BLOCK io_status;
NTSTATUS r = NtOpenFile(&h, GENERIC_READ|SYNCHRONIZE, &oa, &io_status,
FILE_SHARE_READ, FILE_SYNCHRONOUS_IO_NONALERT|FILE_DIRECTORY_FILE);
if (r != STATUS_SUCCESS)
...; // error handling
Unfortunately, it causes kernel to serialize all operations on given handle. I.e. if I try to execute multiple reads in parallel (using multiple threads) -- only one request will be processed at any point in time.
I could get rid of serialization:
HANDLE h;
IO_STATUS_BLOCK io_status;
NTSTATUS r = NtOpenFile(&h, GENERIC_READ|SYNCHRONIZE, &oa, &io_status,
FILE_SHARE_READ, FILE_DIRECTORY_FILE);
if (r == STATUS_PENDING)
...; // what to do here???
but how exactly should I wait for completion -- WaitForSingleObject() on file handle? As far as I know it can change to signaled state due to many reasons -- is there any way to tell that my open file (or dir) operation completed?
Similarly, if I submit multiple reads (from multiple threads) -- how can I tell which one (if any) has finished?
NtOpenFile is synchronous api. it never return STATUS_PENDING to you. even if driver return STATUS_PENDING for IRP_MJ_CREATE i/o sub-system will be wait for IRP complete
https://github.com/Zer0Mem0ry/ntoskrnl/blob/master/Io/iomgr/parse.c#L1404
so you never need check for STATUS_PENDING after NtOpenFile and never need wait (and in principle we can not wait here - we yet not have file handle -so can not wait on it or bind it to say IOCP. we not pass any event or another callback mechanism for NtOpenFile)
I am trying write a real-time flv stream demuxer on windows-runtime, for win8.1 and wp8.1's MediaElement.
I've already finish the demux code, flv files can be correctly demuxed into h264 and aac tag-datas.
When I was trying to play network files and streams, I got a very strange network problem:
The same code,
run under win8.1, all good, whatever files or network streams (proves demux code is ok);
run under wp8.1 (real phone or emulator), files are good, network streams are bad - no matter how I read bytes from HttpClient Stream, the target server only gives me 65536 bytes data, and then the connection is choked, no any response and error, and even no timeout, it's just choking the thread.
Code for opening the stream:
var uri = new Uri("http://hdl.xxx.com/live/yyyy")
//uri is dymatic
var client = new HttpClient();
var stream = await client.GetStreamAsync(uri);
openStream(stream)
Code for reading the data:
public static byte[] ReadBlocks(this Stream stream, int count)
{
byte[] buffer = new byte[count];
int offset = 0;
int length;
while (offset < count)
{
//a loop statement to guarantee I can get *count* bytes
Debug.WriteLine("read " + (count - offset));
//a debug message show how many bytes do I need
length = stream.Read(buffer, offset, count - offset);
if (length == 0)
{
throw new EndOfStreamException();
}
Debug.WriteLine("got " + length);
//a debug message show how many bytes I got
offset += length;
}
return buffer;
}
For example, when I need to rea 1024 bytes from the flv stream, I run stream.ReadBlocks(1024) under wp8.1, the debug tells me like:
read 1024
got 768
read 256
and then nothing happend any more. I wrote an extra counter, the counter shows once server send a total of 65536 bytes, next Read method of stream will always be choked.
I'm sure the uri is available. I can download some stream data as a flv file by using pc web browser, and this downloaded flv file can be played under wp8.1 as well.
It looks like this problem only happens under wp8.1, android and ios are not affected.
So is it my code's problem or actually the server is not set up properly?
From last three weeks, I've tried every http method that can open a stream, but still got choked at 65536 bytes.
Could someone help me, please?
I just solved the same problem - do NOT use System.Net.HttpClient, but Windows.Web.Http.HttpClient
The one in System.Net uses header Connection: Close by default, which causes the stream to close, reading only 65 kB. It also contains a bug which prevents you to override the header to Keep-Alive (it throws some nonesense exception)
I have two NSInputStream and NSOutputStream between devices that are connected to each other via network. When I write something in output stream, the data is written until the NSStreamEventEndEncountered event occurs. I close the output stream but on the other side (input stream) the NSStreamEventEndEncountered event never occurs, until I exit the view controller of the output stream. So:
1. Why does not the NSStreamEventEndEncountered event occurs at input stream even after the the same occurred at the output stream ? (the output stream is even closed in this event)
2. It is my understanding that, once you open the NSOutputStream, you can only write data once. Opening the output stream again after NSStreamEventEndEncountered event (for example to write something new on any event) is not possible, right ????
I probably need more info about your connection and how you're sending your data but let me try and answer your questions:
1.
You're not encountering an end of your inputstream because you never started reading from it. The outputstream finished writing because it probably encountered an end, just like you said.
Imagine Jacob (your outputstream) delivering a envelope (your data) to his friend's house. Jacob puts the envelope on his friend's doormat and walks back to his own house. At this point Jacob's work is done so he tells himself that he's done (in your case the outputstream signals an NSStreamEventEndEncountered).
Jacob's friend George (your inputstream) could see the envelope or not, but nevertheless never looks what is in it. So unless George takes the envelope and looks what is inside it, he could never tell himself that he finished looking at it (in your case the inputstream never signals an NSStreamEventEndEncountered).
2.
This actually depends on how you plan to use your outputstream. If you plan to send data multiple times to the same device, why not leave the outputstream open? You can write data as long as the socket is open and there is space available. When you close the outputstream you need to reopen it however.
I have the following code in my OnExecute in C++ Builder XE:
void __fastcall Test::TestExecute( TIdContext* AContext )
{
try
{
// get the command directive
DWORD startTime = timeGetTime( );
UnicodeString DBCommand = AContext->Connection->IOHandler->ReadChar();
DWORD endTime = timeGetTime();
UnicodeString log;
log.printf( L"getting command %d ms", endTime - startTime );
Log( log );
...
The log starts at getting command 100 milliseconds and creeps to 300 where it sits for the rest of the application run. I thought that OnExecute was called once data was in the buffer, so why would it take 100 to 300 ms for the first read to succeed?
After this first read in the same OnExecute all other data is read very very quickly (millisecond to sub millisecond).
What could be going wrong?
EDIT:
at method launch: AContext->Connection->IOHandler->InputBuffer->Size is 0. After the first read returns AContext->Connection->IOHandler->InputBuffer->Size contains whats left int he buffer after the read. So this implies that OnExecute is called before any data is actually available to the caller. So the 100-300 ms is the amount of time its taking Indy to fetch the data from the socket and place it in the Buffer after it get notification that data is arriving. That seems way too long.
EDIT:
removed do{ as it was implying a loop that was not there.
The OnExecute event is not tied to the socket buffer at all. TIdTCPServer begins calling OnExecute immeidately after the OnConnect event is called, and continues calling OnExecute in an endless loop until the client disconnects (in other words, you should NOT be looping yourself inside of your OnExecute handler. Read a packet, process, exit, wait for the next event, repeat).
You are correct that the InputBuffer can grow larger than what you are asking for in code. All of the IOHandler's reading methods get their data from the InputBuffer only, not the socket directly. If the InputBuffer does not have enough bytes cached to satisfy a read request, the IOHandler will then wait for bytes to be available on the socket, and will then read all of the bytes into the InputBuffer for later use. This minimizes how often the socket needs to be accessed, and help keep the socket responsive to new data.
I am sending data to the server twice. First, I send "Hello world" and then I send "Server".
But the server received the data at 1 read. But the server have to read the data in a two-read operation.
Also, I write the data. Then read data from server and then I write the data.
In this case, the server can read the first data. But server can not read the second data.
The server uses read, write, read.
So how to overcome this issue? How do I write data to socket in BlackBerry?
What you describe is how TCP is supposed to work by default. What you are seeing is the Nagle algorithm (RFC 896) at work, reducing the number of outbound packets being sent so they are processed as efficiently as possible. You may be sending 2 packets in your code, but they are being transmitted together as 1 packet. Since TCP is a byte stream, the receiver should not be making any assumptions about how many packets it gets. You have to delimit your packet data in a higher-level protocol, and the receiver has to process data according to that protocol. It has to handle cases where multiple packets arrive in a single read, a single pakcet arriving in multiple reads, and everything in between, only processing packet data when they have been received in full, caching whatever is left over for subsequent reads to process when needed.
Hard to say without a little more detail, but it sounds like you're using 1-directional communication in the first case - i.e. the client writes, then writes again. There are any number of reasons that the server would receive the 2 writes as 1 read. Buffering on the client, somewhere in the wireless stack (or in the BES), buffering on the server side. All of those are legal with TCP/IP.
Without knowing anything more about your solution, have you thought about defining a small protocol - i.e. the client writes a known byte or bytes (like a 0 byte?) before sending the second write? Then the server can read, then recognize the delimiting byte, and say 'aha, this is now a different write from the client'?
As previously said this is an expected TCP behavior to save bandwidth. Note that to deliver your package TCP adds lot of data (e.g. destination port,sequence number, checksums...).
Instead of flushing the data I´ll recommend you to put more work in your protocol. For example you can define a header that contains the number of bytes to read and then the payload (the actual data).
The following code is a protocol encoded in an string with the structure [length];[data]
StringBuffer headerStr = new StringBuffer();
StringBuffer data = new StringBuffer();
//read header
char headerByte = dataInputStream.readChar();
while (headerByte != ';') {
headerStr.append(headerByte);
headerByte = dataInputStream.readChar();
}
//header has the number of character to read
int header= Integer.parseInt(headerStr.toString());
int bytesReaded = 1;
char dataByte = dataInputStream.readChar();
//we should read the number of characters indicated in the header
while (bytesReaded < header) {
data.append(dataByte);
dataByte = dataInputStream.readChar();
bytesReaded++;
}
For the first query, I guess you are using TCP. If you use UDP, then the server will read the packets in the order you want.
Can you be more clear/elaborative on the second query ?
I would try explicitly telling Connector.open to open up the stream as read_write. Then I would ensure that I flush my connections after each time I talked to the server.
SocketConnection connection = (SocketConnection) Connector.open(url, Connector.READ_WRITE);
OutputStream out = connection.openOutputStream();
// ... write to server
out.flush()
I got a solution to overcome to extract both the string
On sender device
Create a header which contains details of that data eg the data
length, datatype etc.
Add this header to the actual data and send it
On recipient device
read the header
retrieve the actual data length from the header
read the next data upto the data length as specified by the header