I'm having some trouble writing and reading from a socket multiple times. I'm writing a speed test and I essentially want to write an http request using a socket, which will return garbage data which I then read to test the bandwidth speed. I've made the http request successfully, but I'm unable to flip flop between writing to the socket and then reading the response. I can read the data from the socket's stream once, and then it doesn't seem to have any more data to read even after i've made another request.
The salient part of the code is in the start method where the loop runs:
class DownloadTest {
late Socket socket;
late Stream<Uint8List> dataStream;
bool graceTimeOver = false;
int totalBytesDownloaded = 0;
late DateTime startTime;
final ckSize = 10;
final graceTime = 2;
final dlTime = 5;
final client = HttpClient();
final String _serverAddress;
final void Function(double mbps) onProgress;
DownloadTest({required serverAddress, required this.onProgress})
: _serverAddress = serverAddress;
Future<void> start() async {
await resetTest();
while (true) {
writeDlRequest();
var bytesDownloaded = await downloadData();
print(graceTimeOver);
if (!graceTimeOver) {
checkGraceTime();
} else {
print(bytesDownloaded);
totalBytesDownloaded += bytesDownloaded;
if (testFinished()) {
break;
}
}
}
print('test over');
await socket.close();
}
Future<int> downloadData() async {
var bytes = 0;
await for (var data in dataStream) {
bytes += data.length;
}
return bytes;
}
Future<void> resetTest() async {
socket = await Socket.connect(_serverAddress, 80);
dataStream = socket.asBroadcastStream();
graceTimeOver = false;
startTime = DateTime.now();
totalBytesDownloaded = 0;
}
void checkGraceTime() {
if (!graceTimeOver) {
var elapsedSeconds =
startTime.difference(DateTime.now()).inMilliseconds / 1000;
if (elapsedSeconds >= graceTime) {
graceTimeOver = true;
startTime = DateTime.now();
print('grace time over');
}
}
}
void writeDlRequest() {
socket.write('GET /garbage.php?ckSize=$ckSize HTTP/1.1\r\n');
socket.write('Host: speedtest.somethingsomething.com:80\r\n');
socket.write('Connection: keep-alive\r\n');
socket.write('\r\n');
}
bool testFinished() {
var elapsedSeconds =
startTime.difference(DateTime.now()).inMilliseconds / 1000;
return elapsedSeconds >= dlTime;
}
}
I've made the socket's stream into a broadcast stream so I should be able to listen to it multiple times, but after reading from it the first time, it fails to read any more data and the loop just spins. Any ideas where i'm going wrong?
Related
How can I await a stream (or any other event queue) multiple times?
I tried Stream.first & Stream.single, both doesn't work.
What I want to do:
//next is a fake member
Future<void> xxx() async {
final eventStream = foo();
await eventStream.next; // wait for first event
//do some work
await eventStream.next; // wait for second event
//do some other differnet work
await eventStream.next; // wait for 3rd event
// another differnet work
return;
}
equvalent to:
Future<void> xxx() async {
final eventStream = foo();
int i=0;
await for (final _ in eventStream){
if(i==0);//do some work
else if(i==1);//do some other differnet work
else if(i==2){;break;}//another differnet work
++i;
}
return;
}
Try StreamQueue from package:async.
var q = StreamQueue(eventStream);
var v1 = await q.next;
var v2 = await q.next;
var v3 = await q.next;
// other work.
await q.cancel(); // If not listened to completely yet.
I end up with a custom class
class CompleterQueue<T> {
final _buf = <Completer<T>>[];
var _iWrite = 0;
var _iRead = 0;
int get length => _buf.length;
Future<T> next() {
if (_iRead == _buf.length) {
_buf.add(Completer<T>());
}
final fut = _buf[_iRead].future;
_iRead += 1;
_cleanup();
return fut;
}
void add(T val) {
if (_iWrite == _buf.length) {
final prm = Completer<T>();
_buf.add(prm);
}
_buf[_iWrite].complete(val);
_iWrite += 1;
_cleanup();
}
void _cleanup() {
final minI = _iWrite < _iRead ? _iWrite : _iRead;
if (minI > 0) {
_buf.removeRange(0, minI);
_iWrite -= minI;
_iRead -= minI;
}
}
}
We are trying web crawl and get contents from multiple pages. I am taking the advantage of async API with Promise ALL which can execute requests in parallel.
Is there a limitation on the number of contexts which can be opened parallel?
const fs = require('fs');
let browser;
const batch_size = 4; // control the number of async parallel calls
(async () => { // main function
let urls = [];
urls = fs.readFileSync('./resources/input_selenium_urls.csv').toString().split("\n");
browser = await chromium.launch();
let context_size = 0;
let processUrls = [];
let total_length = 0;
for (let i=0;i<urls.length;i++,total_length++) {
if ((context_size==batch_size)||(i==urls.length-1)){
await Promise.all(processUrls.map(x => getHTMLPageSource(x)));
context_size = 0;
processUrls = [];
} else {
processUrls.push(urls[i]);
context_size++;
}
}
await browser.close();
})();
async function getHTMLPageSource(url) {
const context = await browser.newContext();
const page = await context.newPage();
let response = {}
try {
await page.goto(url, { waitUntil: 'networkidle' });
response = {
url : url,
content: await page.title(),
error : null
}
console.log(response);
}
catch {
response = {
error : "Timeout error"
}
}
context.close;
return response;
}
Browser contexts are cheap to create, but it's not clear whether there is a hard-coded limit on them from the docs perhaps the limit might depend on the browser you chose and your OS resources. I think you might only be able to find out by creating a lot of contexts.
I'm trying to do something like this:
Future<String> getOutAndAnswer(testcase) async {
Process python = await Process.start('python', ['tasks/histogram/run.py']);
Process java = await Process.start('java', ['solutions/Histogram.java']);
String results = "";
for(int i = 0; i < testcase; i++){
final String out = await python.stdout.transform(utf8.decoder).first;
java.stdin.writeln(out);
final String answer = await java.stdout.transform(utf8.decoder).first;
python.stdin.writeln(answer);
results += "($out, $answer)";
}
return results;
}
Basically, the python program is responsible for generating the input of each test case, then the java program will take the input and return the answer, which is sent to the python program to check if it's correct or not, and so on for every test case.
But when I try to use the above code I get an error saying I've already listened to the stream once:
Exception has occurred.
StateError (Bad state: Stream has already been listened to.)
Python program:
import os
CASE_DIR = os.path.join(os.path.dirname(__file__), "cases")
test_cases = next(os.walk(CASE_DIR))[2]
print(len(test_cases))
for case in sorted(test_cases):
with open(os.path.join(CASE_DIR, case), 'r') as f:
print(f.readline(), end='', flush=True)
f.readline()
expected_output = f.readline()
user_output = input()
if expected_output != user_output:
raise ValueError("Wrong answer!")
print("EXIT", flush=True)
Java program:
public class Histogram {
public static void main(String[] args) {
Scanner scanner = new Scanner(System.in);
int t = scanner.nextInt();
for (int i = 0; i < t; i++) {
String input = scanner.nextLine();
String answer = calculateAnswer(input);
System.out.println(answer);
}
}
}
Your issue is with .first which is going to listen to the stream, get the first element, and then immediately stop listening to the stream. See the documentation here: https://api.dart.dev/stable/2.17.3/dart-async/Stream/first.html
You should instead listen once and define an onData method to perform the steps. See the documentation for .listen() here: https://api.dart.dev/stable/2.17.3/dart-async/Stream/listen.html
You could try wrapping the stdout streams in StreamIterator<String>. You will have to give it a try to verify, but I think this is what you are looking for.
Future<String> getOutAndAnswer(int testcase) async {
Process python = await Process.start('python', ['tasks/histogram/run.py']);
Process java = await Process.start('java', ['solutions/Histogram.java']);
String results = "";
StreamIterator<String> pythonIterator = StreamIterator(
python.stdout.transform(utf8.decoder).transform(LineSplitter()));
StreamIterator<String> javaIterator = StreamIterator(
java.stdout.transform(utf8.decoder).transform(LineSplitter()));
for (int i = 0; i < testcase; i++) {
if (await pythonIterator.moveNext()) {
final String out = pythonIterator.current;
if (out == 'EXIT') {
break;
}
java.stdin.writeln(out);
if (await javaIterator.moveNext()) {
final String answer = javaIterator.current;
python.stdin.writeln(answer);
results += "($out, $answer)";
}
}
}
await pythonIterator.cancel();
await javaIterator.cancel();
return results;
}
You may need to add the following imports:
import 'dart:async';
import 'dart:convert';
I'm trying to setup a StreamingRecognize, with multiple request's. Is it possible ?
The point is that i want to send audio stream from the mic with a unknown time, so i think that i must implement multiple requests. (Considering that a request session has a max_time = 65 seconds).
Anyone can help me with this ?
Thank's alot ;)
Google sample code:
static async Task<object> StreamingMicRecognizeAsync(int seconds)
{
if (NAudio.Wave.WaveIn.DeviceCount < 1)
{
Console.WriteLine("No microphone!");
return -1;
}
var speech = SpeechClient.Create();
var streamingCall = speech.StreamingRecognize();
// Write the initial request with the config.
await streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
StreamingConfig = new StreamingRecognitionConfig()
{
Config = new RecognitionConfig()
{
Encoding =
RecognitionConfig.Types.AudioEncoding.Linear16,
SampleRateHertz = 16000,
LanguageCode = "en",
},
InterimResults = true,
}
});
// Print responses as they arrive.
Task printResponses = Task.Run(async () =>
{
while (await streamingCall.ResponseStream.MoveNext(
default(CancellationToken)))
{
foreach (var result in streamingCall.ResponseStream
.Current.Results)
{
foreach (var alternative in result.Alternatives)
{
Console.WriteLine(alternative.Transcript);
}
}
}
});
// Read from the microphone and stream to API.
object writeLock = new object();
bool writeMore = true;
var waveIn = new NAudio.Wave.WaveInEvent();
waveIn.DeviceNumber = 0;
waveIn.WaveFormat = new NAudio.Wave.WaveFormat(16000, 1);
waveIn.DataAvailable +=
(object sender, NAudio.Wave.WaveInEventArgs args) =>
{
lock (writeLock)
{
if (!writeMore) return;
streamingCall.WriteAsync(
new StreamingRecognizeRequest()
{
AudioContent = Google.Protobuf.ByteString
.CopyFrom(args.Buffer, 0, args.BytesRecorded)
}).Wait();
}
};
waveIn.StartRecording();
Console.WriteLine("Speak now.");
await Task.Delay(TimeSpan.FromSeconds(seconds));
// Stop recording and shut down.
waveIn.StopRecording();
lock (writeLock) writeMore = false;
await streamingCall.WriteCompleteAsync();
await printResponses;
return 0;
}
In Cloud Speech-to-Text audio length limit for each streaming request is around 1 minute [1]. You can either use asynchronous speech recognition [2] for audio files up to 180 minutes or renew the streaming request before it reaches to the time limit for streaming speech recognition [3].
Here is a Python example how to renew streaming request and stream audio more than 1 minute [4].
I have already try to understand the API doc, the articles about them, and this post: How do you create a Stream in Dart
I'm making a simple web app using WebSocket. Actually, it's working well, but I want add a feature (enjoy learn).
This is my class (can be optimized I guess)
library Ask;
import 'dart:html';
import 'dart:async';
import 'dart:convert';
class Ask {
final String addr;
String _protocol;
String _port;
WebSocket _ws;
bool openned;
Map<int, Completer> _completer_list = {};
int _counter = 0;
static final Map<String, Ask> _cache = <String, Ask>{};
factory Ask(String addr) {
if (_cache.containsKey(addr)) {
return _cache[addr];
} else {
final ask_server = new Ask._internal(addr);
_cache[addr] = ask_server;
return ask_server;
}
}
Ask._internal(this.addr);
Future<bool> open() {
if (openned)
return true;
_completer_list[0] = new Completer();
if (window.location.protocol == 'http:') {
_port = ':8080/ws';
_protocol = 'ws://';
} else {
_port = ':8443/ws';
_protocol = 'wss://';
}
_ws = new WebSocket(_protocol + addr + _port);
_ws.onOpen.listen((e) {
_get_data();
_get_close();
openned = true;
_completer_list[0].complete(true);
});
return _completer_list[0].future;
}
Future<String> send(Map data) {
bool check = false;
int id;
_completer_list.forEach((k, v) {
if (v.isCompleted) {
id = data['ws_id'] = k;
_completer_list[k] = new Completer();
_ws.send(JSON.encode(data));
check = true;
}
});
if (!check) {
_counter++;
id = data['ws_id'] = _counter;
_completer_list[id] = new Completer();
_ws.send(JSON.encode(data));
}
return _completer_list[id].future;
}
void _get_data() {
_ws.onMessage.listen((MessageEvent data) {
var response = JSON.decode(data.data);
_completer_list[response['ws_id']].complete(response);
});
}
void _get_close() {
_ws.onClose.listen((_) {
print('Server have been lost. Try to reconnect in 3 seconds.');
new Timer(new Duration(seconds: 3), () {
_ws = new WebSocket(_protocol + addr + _port);
_get_data();
_get_close();
_ws.onOpen.listen((e) => print('Server is alive again.'));
});
});
}
}
Example of use:
void showIndex() {
Element main = querySelector('main');
Ask connect = new Ask('127.0.0.1');
Map request = {};
request['index'] = true;
connect.open().then((_) {
connect.send(request).then((data) {
main.setInnerHtml(data['response']);
});
});
}
I would replace the then by a listen who will be canceled when the message will completed. By this way, I can add a progress bar, I think...
So my question, my send function can be a stream and keep my concept of one websocket for all ? (yes, if my function is used when a request is in progress, it's sent and if she's finish before the first, I recovered her properly. Thank you ws_id).
Thank you.
I think what you need is a StreamController
https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart-async.StreamController