I want to get length of BehaviorSubject's stream, but I can't get it.
test('get stream length', ()async{
BehaviorSubject<int> subject = new BehaviorSubject(seedValue: 0);
var act = await subject.stream.length;
expect(act, 1);
});
How I can get this length?
The length of a stream can only be known after it is closed. As long as it is not closed it's always possible that another event will be added.
https://api.dartlang.org/stable/2.1.1/dart-async/Stream/length.html
length property
Future<int> length
The number of elements in this
stream.
Waits for all elements of this stream. When this stream ends, the
returned future is completed with the number of elements.
If this stream emits an error, the returned future is completed with
that error, and processing stops.
This operation listens to this stream, and a non-broadcast stream
cannot be reused after finding its length.
test('get stream length', ()async{
BehaviorSubject<int> subject = new BehaviorSubject(seedValue: 0);
var actFuture = await subject.stream.length;
await subject.close();
expect(actFuture, completion(equals(1));
});
Related
IMPORTANT EDIT
After further investigating, I found out that the packet size is in fact much larger than the stated 1024 bytes, the 1024 bytes were just the limit of the standard out I was using (android studio / flutter).
Some of the packets received are now up to ~27 000 bytes large, however that is nowhere near the actually transmitted size of over 10x that.
I am trying to send singular packets of up to 5 MB in length over a Socket connection in Dart. For this, I am using the following code:
Socket socket = await Socket.connect(globals.serverUrl, globals.serverPort);
Stream<Uint8List> stream = socket?.asBroadcastStream();
Uint8List? response = await stream?.first;
String responseString = String.fromCharCodes(response);
Note that my Server is running Java while the Client is using Dart.
After sending the data packet from the Server to the Client, it successfully receives the first 1024 bytes exactly of the packet, and the rest is nowhere to be found, even after reading stream.first multiple times, they continuously read the newly sent packet and not the remaining bytes of the old packet.
So my question is, how do I require the Socket stream to read ALL bytes of the packet until finished, and not just the first 1024?
EDIT:
The received packet on the client is parsed using:
String? decrypt(String cipherText, String keyString) {
final key = Key.fromUtf8(keyString);
final iv = IV.fromBase64(cipherText.split(":")[1]);
final encrypter = Encrypter(AES(key, mode: AESMode.cbc, padding: null));
final encrypted = Encrypted.fromBase64(cipherText.split(":")[0]);
final decrypted = encrypter.decrypt(encrypted, iv: iv);
globals.log.i("DECRYPTED: $decrypted");
return decrypted;
}
The error that I am getting stems from getting the IV, since the message is cut off at 1024 bytes and the ":" appears much later in the String.
The problem is that the Dart socket split messages bigger than 1024 bytes into multiple packets of 1024 bytes. So there's some approaches you can use to combine them together in the client:
By extending Socket class
I do not believe this is a right solution:
Hard to extend since it's a platform implementation (you can see the sdk implementation of dart:io almost any class method is external).
Hard to maintain.
Since it depends on custom platform implementations you need to do it on multiple platforms.
It's easy to create undocumented memory leaks.
Let me know if you still prefer this approach I'll do a further research.
By using Stream<T>.reduce function
The problem with this approach in your context is that Sockets do not emit a done event when a message is sent by using socket.write('Your message').
So unless you're using a socket to send a single message this function can't help you cause it will return a Future<T> that will never complete (only when the socket connection is closed).
By emitting a EOF message from the server
This is a solution I found even not so elegant, improvements are welcome.
The idea is to concatenate all client received packets into a single one and stop receiving when a pre-determined termination (EOF) string is received.
Implementation
Below is a server implementation that emits a message of 5mb followed by a message:end string every time a new client is connected.
import 'dart:io';
Future<void> main() async {
final ServerSocket serverSocket =
await ServerSocket.bind(InternetAddress.anyIPv4, 5050);
final Stream<Socket> serverStream = serverSocket.asBroadcastStream();
serverStream.listen((client) async {
print(
'New client connected: ${client.address}:${client.port} ${client.done} Remote address: ${client.remoteAddress}');
const int k1byte = 8;
const int k2bytes = k1byte * 2;
const int k1kb = k1byte * 1000;
const int k1mb = k1kb * 1000;
const int k5mb = k1mb * 5;
// Create a 5mb string that follows: '1000.....0001'
final String k1mbMessage = '1${createStringOf(k5mb - k2bytes, '0')}1';
client.write(k1mbMessage);
client.write('message:end');
});
print('Listening on: ${serverSocket.address} ${serverSocket.port}');
}
String createStringOf(int size, [String char = ' ']) {
// https://api.dart.dev/stable/2.17.3/dart-core/String-class.html it says:
// > A sequence of UTF-16 code units.
// And from https://www.ibm.com/docs/en/db2-for-zos/12?topic=unicode-utfs says:
// > UTF-16 is based on 16-bit code units. Each character is encoded as at least 2 bytes.
int dartStringEncodingSize = 2;
assert(size >= dartStringEncodingSize && size.isEven,
'''Dart char contains 2 bytes so we can only create Strings (with exact size) of even N bytes''');
assert(char.length == 1, '''[char] must be a single char String''');
int charCount = size ~/ dartStringEncodingSize;
return char * charCount;
}
And here we can see a client implementation where we use 'our own reduce' function that combine all packets while the termination string is not found.
import 'dart:io';
Future<void> main() async {
final Socket server = await Socket.connect('localhost', 5050);
final Stream<String> serverSocket =
server.asBroadcastStream().map(String.fromCharCodes); // Map to String by default
const kMessageEof = 'message:end';
String message = '';
await for (String packet in serverSocket) {
// If you are using [message] as a List of bytes (Uint8List):
// message = [...Uint8List.fromList(message), ...Uint8List(packet)]
message += packet;
// Do not compare directly packet == kMessageEof
// cause it can be 'broken' into multiple packets:
// -> 00000 (packet 1)
// -> 00000 (packet 2)
// -> 00mes (packet 3)
// -> sage: (packet 4)
// -> end (packet 5)
if (message.endsWith(kMessageEof)) {
// remove termination string
message = message.replaceRange(
message.length - kMessageEof.length,
message.length,
'',
);
}
print('Received: $message'); // Prints '1000000......0000001'
}
}
You can make it more generic if you want by using an extension:
import 'dart:io';
/// This was created since the native [reduce] says:
/// > When this stream is done, the returned future is completed with the value at that time.
///
/// The problem is that socket connections does not emits the [done] event after
/// each message but after the socket disconnection.
///
/// So here is a implementation that combines [reduce] and [takeWhile].
extension ReduceWhile<T> on Stream<T> {
Future<T> reduceWhile({
required T Function(T previous, T element) combine,
required bool Function(T) combineWhile,
T? initialValue,
}) async {
T initial = initialValue ?? await first;
await for (T element in this) {
initial = combine(initial, element);
if (!combineWhile(initial)) break;
}
return initial;
}
}
Future<void> main() async {
final Socket server = await Socket.connect('localhost', 5050);
final Stream<String> serverSocket =
server.asBroadcastStream().map(String.fromCharCodes);
const kMessageEof = 'message:end';
// Reduce with a condition [combineWhile]
String message = await serverSocket.reduceWhile(
combine: (previous, element) => '$previous$element',
combineWhile: (message) => !message.endsWith(kMessageEof),
);
// Remove termination string
message = message.replaceRange(
message.length - kMessageEof.length,
message.length,
'',
);
print('Received: $message');
}
Since the socket itself doesn't send the done event the way I found to reduce all packets into a single one was by emitting 'our own done event'.
Speech to text throws an error after some time:
Grpc.Core.RpcException: Status(StatusCode=OutOfRange, Detail="Audio
Timeout Error: Long duration elapsed without audio. Audio should be
sent close to real time.")
Another error is:
Grpc.Core.RpcException: Status(StatusCode=Cancelled, Detail="The
operation was cancelled.")
How to avoid these errors? Both the errors are at function:
if (firstMessage == false) {
// var buffer = new byte[32 * 1024];
var buffer = new byte[4096];
int bytesRead;
while ((bytesRead = await outputStream.ReadAsync(
buffer, 0, buffer.Length)) > 0) {
await streamingCall.WriteAsync(
new StreamingRecognizeRequest() {
AudioContent = Google.Protobuf.ByteString
.CopyFrom(buffer, 0, bytesRead),
});
// await Task.Delay(100);
};
}
You must be doing some speech recognition from a streaming input. This error usually occurs when there's a period of silence (e.g 10 seconds or more) in your streaming input, thus, it signals the end of the recognition and cancels the operation. To avoid, make sure to send inputs close to real time.
When should I close an ObjectOutputStream using the following code? Thank you all...
try{
ObjectOutputStream output = new ObjectOutputStream(client.getOutputStream());
AnObject[] array = new AnObject[4];
for(int i = 0; i < array.length ; i++){
array[i] = new AnObject();
output.writeObject(array[i]);
}
output.flush();
output.close();
}
catch(IOException e){
processException();
}
Whenever you're done using it. You should flush() after sending, to force through all the data added to the stream's buffer to your underlying stream (Object*Streams are wrappers to your client's OutputStream), even if the stream's buffer isn't full.
Also, no need to flush before closing, because output.close() calls flush().
It looks like you close it after sending data, so unless you have more stuff to send through your client's stream, you're closing properly.
I'm attempting to write a bit of JS that will read a file and write it out to a stream. The deal is that the file is extremely large, and so I have to read it bit by bit. It seems that I shouldn't be running out of memory, but I do. Here's the code:
var size = fs.statSync("tmpfile.tmp").size;
var fp = fs.openSync("tmpfile.tmp", "r");
for(var pos = 0; pos < size; pos += 50000){
var buf = new Buffer(50000),
len = fs.readSync(fp, buf, 0, 50000, (function(){
console.log(pos);
return pos;
})());
data_output.write(buf.toString("utf8", 0, len));
delete buf;
}
data_output.end();
For some reason it hits 264900000 and then throws FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory. I'd figure that the data_output.write() call would force it to write the data out to data_output, and then discard it from memory, but I could be wrong. Something is causing the data to stay in memory, and I've no idea what it would be. Any help would be greatly appreciated.
I had a very similar problem. I was reading in a very large csv file with 10M lines, and writing out its json equivalent. I saw in the windows task manager that my process was using > 2GB of memory. Eventually I figured out that the output stream was probably slower than the input stream, and that the outstream was buffering a huge amount of data. I was able to fix this by pausing the instream every 100 writes to the outstream, and waiting for the outstream to empty. This gives time for the outstream to catch up with the instream. I don't think it matters for the sake of this discussion, but I was using 'readline' to process the csv file one line at a time.
I also figured out along the way that if, instead of writing every line to the outstream, I concatenate 100 or so lines together, then write them together, this also improved the memory situation and made for faster operation.
In the end, I found that I could do the file transfer (csv -> json) using just 70M of memory.
Here's a code snippet for my write function:
var write_counter = 0;
var out_string = "";
function myWrite(inStream, outStream, string, finalWrite) {
out_string += string;
write_counter++;
if ((write_counter === 100) || (finalWrite)) {
// pause the instream until the outstream clears
inStream.pause();
outStream.write(out_string, function () {
inStream.resume();
});
write_counter = 0;
out_string = "";
}
}
You should be using pipes, such as:
var fp = fs.createReadStream("tmpfile.tmp");
fp.pipe(data_output);
For more information, check out: http://nodejs.org/docs/v0.5.10/api/streams.html#stream.pipe
EDIT: the problem in your implementation, btw, is that by doing it in chunks like that, the write buffer isn't going to get flushed, and you're going to read in the entire file before writing much of it back out.
According to the documentation, data_output.write(...) will return true if the string has been flushed, and false if it has not (due to the kernel buffer being full). What kind of stream is this?
Also, I'm (fairly) sure this isn't the problem, but: how come you allocate a new Buffer on each loop iteration? Wouldn't it make more sense to initialize buf before the loop?
I don't know how the synchronous file functions are implemented, but have you considered using the asynch ones? That would be more likely to allow garbage collection and i/o flushing to happen. So instead of a for loop, you would trigger the next read in the callback function of the previous read.
Something along these lines (note also that, per other comments, I'm reusing the Buffer):
var buf = new Buffer(50000),
var pos = 0, bytesRead;
function readNextChunk () {
fs.read(fp, buf, 0, 50000, pos,
function(err, bytesRead){
if (err) {
// handle error
}
else {
data_output.write(buf.toString("utf8", 0, bytesRead));
pos += bytesRead;
if (pos<size)
readNextChunk();
}
});
}
readNextChunk();
When reading the InputStream of an HttpURLConnection, is there any reason to use one of the following over the other? I've seen both used in examples.
Manual Buffer:
while ((length = inputStream.read(buffer)) > 0) {
os.write(buf, 0, ret);
}
BufferedInputStream
is = http.getInputStream();
bis = new BufferedInputStream(is);
ByteArrayBuffer baf = new ByteArrayBuffer(50);
int current = 0;
while ((current = bis.read()) != -1) {
baf.append(current);
}
EDIT I'm still new to HTTP in general but one consideration that comes to mind is that if I am using a persistent HTTP connection, I can't just read until the input stream is empty right? In that case, wouldn't I need to read the message length and just read the input stream for that length?
And similarly, if NOT using a persistent connection, is the code I included 100% good to go in terms of reading the stream properly?
I talk about a good way to do it on my blog in a post about using JSON in android. http://blog.andrewpearson.org/2010/07/android-why-to-use-json-and-how-to-use.html. I will post the relevant part of the relevant post below (the code is pretty generalizable):
InputStream in = null;
String queryResult = "";
try {
URL url = new URL(archiveQuery);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
httpConn.setAllowUserInteraction(false);
httpConn.connect();
in = httpConn.getInputStream();
BufferedInputStream bis = new BufferedInputStream(in);
ByteArrayBuffer baf = new ByteArrayBuffer(50);
int read = 0;
int bufSize = 512;
byte[] buffer = new byte[bufSize];
while(true){
read = bis.read(buffer);
if(read==-1){
break;
}
baf.append(buffer, 0, read);
}
queryResult = new String(baf.toByteArray());
} catch (MalformedURLException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
} catch (IOException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
}
}
Regarding persistent HTTP connections it is just the opposite. You should read everything from the input stream. Otherwise the Java HTTP client does not know that the HTTP request is complete and the socket connection can be reused.
See http://java.sun.com/javase/6/docs/technotes/guides/net/http-keepalive.html:
What can you do to help with Keep-Alive?
Do not abandon a connection by
ignoring the response body. Doing so
may results in idle TCP connections.
That needs to be garbage collected
when they are no longer referenced.
If getInputStream() successfully
returns, read the entire response
body.
Use former -- latter has no real benefits over first one, and is bit slower; reading things byte by byte is inefficient even if buffered (although horribly slow when not buffered). That style of reading input went out of vogue with C; although may be useful in cases where you need to find an end marker of some sort.
Only if you're using the BufferedInputStream-specific methods.