I have tried returning 'transfer-encoding:chunked' header to a large download but the response terminated at "31.9M" which is very close to the documented "32MB" limit.
The Unofficial FAQ states chunked encoding is now possible, but I can't seem to get it working.
Do I have to flip any flags (e.g. https/2) to enable streaming? Is it only possible in some regions? (I am using europe-west1)
The following minimal case does actually stream 45MB over Cloud Run, without any special configuration, confirmed in us-central1 and europe-west1
FROM node:14-slim
COPY index.js index.js
CMD [ "node", "index.js" ]
const http = require('http');
http.createServer((request, response) => {
response.setHeader('Transfer-Encoding', 'chunked');
// 45MB
var i = 1000000;
function nextChunk() {
return new Promise(resolve => {
if (i-- > 0) {
response.write(
'123456781234567812345678123456781234567812345678',
() => nextChunk()
);
} else {
response.end();
}
})
};
Promise.all(nextChunk())
}).listen(process.env.PORT || 8080);
Even after straming into chunks I'm always getting error after 32 MB reponse size, I can't get the response header 'Transfer-Encoding' = 'chunked' from Cloud Run service response :
I'm using NestJs, and I can get it in the response Headers when I execute the app locally.
I'm specifying it as :
#Get('streamable')
#Header('Transfer-Encoding', 'chunked')
async streamable(#Res() res) { etc.. res.write(chunk)}
Related
So, I'm using filtered stream. Unfortunately, every time I turn it on, after 5 minutes its turns off but during that time, it catches tweets.
What I'm trying to do is keep it on 24/7 so, it doesn't turn off after 5 minutes. Also, if it disconnects I want it so, it tries connecting again.
This is the code sample I'm using to help adjust my code:
https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/master/Filtered-Stream/filtered_stream.js
function streamTweets(retryAttempt) {
const stream = needle.get(streamURL, {
headers: {
Authorization: `Bearer ${TOKEN}`
},
retryAttempt: 20000
});
stream.on('data', (data) => {
try {
const json = JSON.parse(data)
console.log(json.data.text)
retryAttempt = 0;
} catch (e) {
if (data.detail === "This stream is currently at the maximum allowed connection limit.") {
console.log(data.detail)
process.exit(1)
} else {
// Keep alive signal received. Do nothing.
}
}
}).on('err', error => {
if (error.code !== 'ECONNRESET') {
console.log(error.code);
process.exit(1);
} else {
// This reconnection logic will attempt to reconnect when a disconnection is detected.
// To avoid rate limits, this logic implements exponential backoff, so the wait time
// will increase if the client cannot reconnect to the stream.
setTimeout(() => {
console.warn("A connection error occurred. Reconnecting...")
streamTweets(++retryAttempt);
}, 2 ** retryAttempt)
}
});
return stream;
}
(async() => {
let currentRules;
try {
//get all stream rules
currentRules = await getRules();
//delete all stream rules
await deleteRules(currentRules);
//Set rules based on array above
await setRules();
} catch (e) {
console.error(e);
process.exit(1);
}
streamTweets(0);
})();
Have you tried to send the keepalive in your header?
Also, I changed the authorization header to match the syntax that of your linked GitHub code source.
function streamTweets(retryAttempt) {
const stream = needle.get(streamURL, {
headers: {
"authorization": `Bearer ${token}`,
"Connection": "keep-alive"
},
retryAttempt: 20000
});
Also, according to the Twitter docs: " If you would like to close your connection, you can press Control-C in your command line tool on either Mac or Windows systems to break the connection, or you can also close the window. "
Are you sure it's not disconnecting because you closed the terminal session, or was closed by something like an ssh timeout?
I have this Ionic 5/Capacitor app, which I'm making an API call to a local server from, that server running on docker at localhost:3000. When I test from the browser, the request is made fine. From Postman it requests fine, too. In my XCode logs the emulator, I see this
[error] - ERROR {"headers":{"normalizedNames":{},"lazyUpdate":null,"headers":{}},"status":0,"statusText":"Unknown Error","url":"http://localhost:3000/pins","ok":false,"name":"HttpErrorResponse","message":"Http failure response for http://localhost:3000/pins: 0 Unknown Error","error":{"isTrusted":true}}
The really interesting part, is that I'm running Fiddler to monitor the request as it's made. Fiddler gets a 200 as well, I can even see the response data. So, Fiddler sees the proper network call, but then my Ionic app gets that error. That makes me feel like it's an Ionic/Emulator/IOS problem, but I don't have enough familiarity with Ionic to know right off the bat what it is.
Here's the code responsible for making the request:
ngOnInit() {
const request = this.http.get('http://localhost:3000/pins');
this.refresh$.subscribe(
(lastPos: { latitude?: any; longitude?: number }) => {
request.subscribe(data => {
if (data) {
this.addMarkersToMap(data, lastPos);
}
});
}
);
}
And the HTTPClient imported in the constructor is from Angular:
import { HttpClient } from '#angular/common/http';
I ended up having to use this package, doing a check on if I'm on mobile or not.
https://ionicframework.com/docs/native/http/
Try with this :
const request = this.http.get('http://localhost:3000/pins', { observe: 'response', withCredentials: true });
Solution 2 : capacitor.config.json
"server": {
"hostname": "localhost", (maybe try precising the port number too)
}
Solution 3 : On your Express server (from https://ionicframework.com/docs/troubleshooting/cors)
const express = require('express');
const cors = require('cors');
const app = express();
const allowedOrigins = [
'capacitor://localhost',
'ionic://localhost',
'http://localhost',
'http://localhost:8080',
'http://localhost:8100'
];
// Reflect the origin if it's in the allowed list or not defined (cURL, Postman, etc.)
const corsOptions = {
origin: (origin, callback) => {
if (allowedOrigins.includes(origin) || !origin) {
callback(null, true);
} else {
callback(new Error('Origin not allowed by CORS'));
}
}
}
// Enable preflight requests for all routes
app.options('*', cors(corsOptions));
app.get('/', cors(corsOptions), (req, res, next) => {
res.json({ message: 'This route is CORS-enabled for an allowed origin.' });
})
app.listen(3000, () => {
console.log('CORS-enabled web server listening on port 3000');
});
I was watching Steve Sanderson's NDC presentation on up-and-coming web features, and saw his caching example as a prime candidate for an application I am developing. I couldn't find the code, so I have typed it up off the Youtube video as well as I could.
Unfortunately it doesn't work in Chrome (which is also what he is using in the demo) It fails with Uncaught TypeError: fetch(...).then(...).timeout is not a function
at self.addEventListener.event.
I trawled through Steve's Github, and found no trace of this, nor could I find anything on the NDC Conference page
//inspiration:
// https://www.youtube.com/watch?v=MiLAE6HMr10
//self.importScripts('scripts/util.js');
console.log('Service Worker script running');
self.addEventListener('install', event => {
console.log('WORKER: installing');
const urlsToCache = ['/ServiceWorkerExperiment/', '/ServiceWorkerExperiment/scripts/page.js'];
caches.delete('mycache');
event.waitUntil(
caches.open('mycache')
.then(cache => cache.addAll(urlsToCache))
.then(_ => self.skipWaiting())
);
});
self.addEventListener('fetch', event => {
console.log(`WORKER: Intercepted request for ${event.request.url}`);
if (event.request.method !== 'GET') {
return;
}
event.respondWith(
fetch(event.request)
.then(networkResponse => {
console.log(`WORKER: Updating cached data for ${event.request.url}`);
var responseClone = networkResponse.clone();
caches.open('mycache').then(cache => cache.put(event.request, responseClone));
return networkResponse;
})
//if network fails or is too slow, return cached data
//reference for this code: https://youtu.be/MiLAE6HMr10?t=1003
.timeout(200)
.catch(_ => {
console.log(`WORKER: Serving ${event.request.url} from CACHE`);
return caches.match(event.request);
})
);
});
As far as I read the fetch() documentation, there is no timeout function, so my assumption is that the timeout function is added in the util.js which is never shown in the presentation... can anyone confirm this? and does anyone have an Idea about how this is implemented?
Future:
It's coming.
According to Jake Archibald's comment on whatwg/fetch the future syntax will be:
Using the abort syntax, you'll be able to do:
const controller = new AbortController();
const signal = controller.signal;
const fetchPromise = fetch(url, {signal});
// 5 second timeout:
const timeoutId = setTimeout(() => controller.abort(), 5000);
const response = await fetchPromise;
// …
If you only wanted to timeout the request, not the response, add:
clearTimeout(timeoutId);
// …
And from another comment:
Edge & Firefox are already implementing. Chrome will start shortly.
Now:
If you want to try the solution that works now, the most sensible way is to use this module.
It allows you to use syntax like:
return fetch('/path', {timeout: 500}).then(function() {
// successful fetch
}).catch(function(error) {
// network request failed / timeout
})
I was trying to send data to a local server using HttpClient. However, the data is never added to the request, I'm using this code:
new HttpClient().put('127.0.0.1', 4040, '/employees/1').then((request) {
request.cookies.add(new Cookie('DARTSESSID',sessionId)..path = '/');
request.headers.add(HttpHeaders.ACCEPT_ENCODING, "");
request.headers.add(HttpHeaders.CONTENT_TYPE, "text/json");
request.write('{"id": 1, "name": "luis"}');
print(request.contentLength);
return request.close();
}).then(expectAsync((HttpClientResponse response) {
expect(response.statusCode, 200);
UTF8.decodeStream(response).then(expectAsync((body) {
expect(body, equals('"employee: 1"'));
}));
}));
but that always prints that the request.contentLenght is -1. I saw those links before without luck:
https://code.google.com/p/dart/issues/detail?id=13293
dart - HttpClientRequest failing on adding data
https://code.google.com/p/dart/issues/detail?id=10026
A ContentLength of -1 does not mean that there is no data, it means that the length of the content is unknown and that a streaming content mode is used - for HTTP 1.1, this will usually mean Chunked ContentEncoding.
I've tried to insert your code in a setup including a server, but without the unittest stuff:
import 'dart:convert';
import 'dart:io';
void main() {
HttpServer.bind('127.0.0.1', 4040).then((server) {
server.listen((request) {
UTF8.decodeStream(request).then((body) {
print(body);
request.response.close();
});
});
new HttpClient().put('127.0.0.1', 4040, '/employees/1').then((request) {
request.cookies.add(new Cookie('DARTSESSID', "1")..path = '/');
request.headers.add(HttpHeaders.ACCEPT_ENCODING, "");
request.headers.add(HttpHeaders.CONTENT_TYPE, "text/json");
request.write('{"id": 1, "name": "luis"}');
print(request.contentLength);
return request.close();
}).then((HttpClientResponse response) {
UTF8.decodeStream(response).then((body) {
print(body);
});
});
});
}
When I run the code, I get
-1
{"id": 1, "name": "luis"}
as expected. Perhaps the problem you are having are on the server?
Writing to the request is an asynchronous operation. Just because the contentLength says that it still is -1 doesn't mean that the data isn't added to the request before sending it to the server.
Also: the content-length is not supposed to update whenever you add new data. It is the value that is sent to the server. -1 means that you don't know the size yet.
I'm not sure, if the library automatically updates it, if it knows the size, but it doesn't need to.
I need to send a request to the server to run a jar file with a string argument/parameter and return the results as a string.
On server side you can run a process and send result back like this :
HttpServer.bind(InternetAddress.ANY_IP_V4, 3031).then((server) {
server.listen((HttpRequest request) {
var param = request.uri.queryParameters['name'];
Process.run('java', ['-jar', 'myJar.jar', param]).then((pr) =>
request.response
..write(pr.stdout)
..close()
);
});
});