So, I'm using filtered stream. Unfortunately, every time I turn it on, after 5 minutes its turns off but during that time, it catches tweets.
What I'm trying to do is keep it on 24/7 so, it doesn't turn off after 5 minutes. Also, if it disconnects I want it so, it tries connecting again.
This is the code sample I'm using to help adjust my code:
https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/master/Filtered-Stream/filtered_stream.js
function streamTweets(retryAttempt) {
const stream = needle.get(streamURL, {
headers: {
Authorization: `Bearer ${TOKEN}`
},
retryAttempt: 20000
});
stream.on('data', (data) => {
try {
const json = JSON.parse(data)
console.log(json.data.text)
retryAttempt = 0;
} catch (e) {
if (data.detail === "This stream is currently at the maximum allowed connection limit.") {
console.log(data.detail)
process.exit(1)
} else {
// Keep alive signal received. Do nothing.
}
}
}).on('err', error => {
if (error.code !== 'ECONNRESET') {
console.log(error.code);
process.exit(1);
} else {
// This reconnection logic will attempt to reconnect when a disconnection is detected.
// To avoid rate limits, this logic implements exponential backoff, so the wait time
// will increase if the client cannot reconnect to the stream.
setTimeout(() => {
console.warn("A connection error occurred. Reconnecting...")
streamTweets(++retryAttempt);
}, 2 ** retryAttempt)
}
});
return stream;
}
(async() => {
let currentRules;
try {
//get all stream rules
currentRules = await getRules();
//delete all stream rules
await deleteRules(currentRules);
//Set rules based on array above
await setRules();
} catch (e) {
console.error(e);
process.exit(1);
}
streamTweets(0);
})();
Have you tried to send the keepalive in your header?
Also, I changed the authorization header to match the syntax that of your linked GitHub code source.
function streamTweets(retryAttempt) {
const stream = needle.get(streamURL, {
headers: {
"authorization": `Bearer ${token}`,
"Connection": "keep-alive"
},
retryAttempt: 20000
});
Also, according to the Twitter docs: " If you would like to close your connection, you can press Control-C in your command line tool on either Mac or Windows systems to break the connection, or you can also close the window. "
Are you sure it's not disconnecting because you closed the terminal session, or was closed by something like an ssh timeout?
Related
I have small app that execute requests to external service.
final app = Alfred();
app.post('/ctr', (req, res) async { // complex tender request
var data = await req.body;
await complexTendexAPIRequest(data as Map<String, dynamic>);
print('Hello World');
await res.json({'data': 'ok'});
});
Handler code:
complexTendexAPIRequest(Map<String, dynamic> data) async {
print('Request: $data');
try {
final response = await http.post(
Uri.parse(COMPLEX_URL),
headers: {'Content-Type': 'application/json', 'Authorization': 'bearer $ACCESS_TOKEN'},
body: json.encode(data)
);
if(response.statusCode == 200) {
var res = json.decode(response.body);
int latestId = res['id'];
String url = 'https://api.ru/v2/complex/status?id=$latestId';
stdout.write('Waiting for "complete" status from API: ');
Timer.periodic(Duration(seconds: 1), (timer) async {
final response = await http.get(
Uri.parse(url),
headers: {'Content-Type': 'application/json', 'Authorization': 'bearer $ACCESS_TOKEN'}
);
var data = json.decode(response.body);
if(data['status'] == 'completed') {
timer.cancel();
stdout.write('[DONE]');
stdout.write('\nFetching result: ');
String url = "https://api.ru/v2/complex/results?id=$latestId";
final response = await http.get(
Uri.parse(url),
headers: {'Content-Type': 'application/json', 'Authorization': 'bearer $ACCESS_TOKEN'}
);
stdout.write('[DONE]');
var data = prettyJson(json.decode(response.body));
await File('result.json').writeAsString(data.toString());
print("\nCreating dump of result: [DONE]");
}
});
}
else {
print('[ERROR] Wrong status code for complex request. StatusCode: ${response.statusCode}');
}
}
on SocketException catch(e) {
print('No Internet connection: $e');
} on TimeoutException catch(e) {
print('TenderAPI Timeout: $e');
} on Exception catch(e) {
print('Some unknown Exception: $e');
}
}
But output is very strange it's look like it's do not waiting complexTendexAPIRequest completion and go forward:
Waiting for "complete" status from API: Hello World
[DONE]
Fetching result: [DONE]
Creating dump of result: [DONE]
But should be:
Waiting for "complete" status from API: [DONE]
Fetching result: [DONE]
Creating dump of result: [DONE]
Hello World
I suppose that reason can be in Timer.periodic but how to fix it to get expected order and execution of:
print('Hello World');
await res.json({'data': 'ok'});
only after complexTendexAPIRequest completed.
upd: I rewrote code to while loop:
https://gist.github.com/bubnenkoff/fd6b4f0d7aeae7007680e7902fbdc1e9
it's seems that it's ok.
Alfred https://github.com/rknell/alfred
The problem is the Timer.periodic, as others have pointed out.
You do:
Timer.periodic(Duration(seconds: 1), (timer) async {
// do something ...
});
That sets up a timer, then immediately continues execution.
The timer triggers every second, calls the async callback (which returns a future that no-one ever waits for) and which does something which just might take longer than a second.
You can convert this to a normal loop, basically:
while (true) {
// do something ...
if (data['status'] == 'completed') {
// ...
break;
} else {
// You can choose your own delay here, doesn't have
// to be the same one every time.
await Future.delayed(const Duration(seconds: 1));
}
}
If you still want it to be timer driven, with fixed ticks, consider rewriting this as:
await for (var _ in Stream.periodic(const Duration(seconds: 1))) {
// do something ...
// Change `timer.cancel();` to a `break;` at the end of the block.
}
Here you create a stream which fires an event every second. Then you use an await for loop to wait for each steam event. If the thing you do inside the loop is asynchronous (does an await) then you are even ensured that the next stream event is delayed until the loop body is done, so you won't have two fetches running at the same time.
And if the code throws, the error will be caught by the surrounding try/catch, which I assume was intended, rather than being an uncaught error ending up in the Future that no-one listens to.
If you want to retain the Timer.periodic code, you can, but you need to do something extra to synchronize it with the async/await code around it (which only really understands futures and streams, not timers).
For example:
var timerDone = Completer();
Timer.periodic(const Duration(seconds: 1), (timer) async {
try {
// do something ...
// Add `timerDone.complete();` next to `timer.cancel()`.
} catch (e, s) {
timer.cancel();
timerDone.completeError(e, s);
}
});
await timerDone.future;
This code uses a Completer to complete a future and effectively bridge the gap between timers and futures that can be awaited (one of the listed uses of Completer in the documentation).
You may still risk the timer running concurrently if one step takes longer than a second.
Also, you can possibly use the retry package, if it's the same thing you want to retry until it works.
I am using Twilio Flex to support a call center. I have a TaskRouter workflow set up where Task Reservation Timeout is set to 120 seconds. In its filter, I've created two routing steps. The first one finds matching workers in the main queue and has a timeout of 120 seconds. After 120 seconds, it should move to Call Forward Queue. In the call forward queue, no workers exist (target worker expression: 1==2). I'm catching all these events with a "trEventListener" function. Once a task is moved into the Call Forward queue, I call the "callForward" function which uses twiml.dial() to connect the call to an external number. I also change this task's status to "canceled" with a custom reason so I can track it in flex insights. I am using the guide in this link to form my logic: https://support.twilio.com/hc/en-us/articles/360021082934-Implementing-Voicemail-with-Twilio-Flex-TaskRouter-and-WFO.
Call forwarding is working fine but according to Flex insights, there are some calls that get handled after 120 seconds (between 120 - 300 seconds). Ideally, these should be forwarded as well. There is also no error logged for me to track down why this is happening to only a handful of calls.
Furthermore, in some cases, when I try to change the task status to cancel with my custom reason, it spits out the following error: Cannot cancel task because it is not pending or reserved. In other cases, it works fine. It's again hard to figure out why it's selectively working and not consistent in its behavior.
Here is the function code.
trEventListener.js:
exports.handler = function(context, event, callback) {
const client = context.getTwilioClient();
let task = '';
let workspace = '';
console.log(`__[trEventStream]__: Event recieved of type: ${event.EventType}`);
// setup an empty success response
let response = new Twilio.Response();
response.setStatusCode(204);
// switch on the event type
switch(event.EventType) {
case 'task-queue.entered':
// ignore events that are not entering the 'Call Forward' TaskQueue
if (event.TaskQueueName !== 'Call Forward') {
console.log(`__[trEventStream]__: Entered ${event.TaskQueueName} queue - no forwarding required!`);
return callback(null, response);
}
console.log(`__[trEventStream]__: entered ${event.TaskQueueName} queue - forwarding call!`);
task = event.TaskSid;
workspace = event.WorkspaceSid;
const ta = JSON.parse(event.TaskAttributes);
const callSid = ta.call_sid;
let url = `https://${context.DOMAIN_NAME}/forwardCall`;
// redirect call to forwardCall function
client.calls(callSid).update({
method: 'POST',
url: encodeURI(url),
}).then(() => {
console.log(`__[trEventStream]__: [SUCCESS] ~> Task with id ${task} forwarded to external DID`);
// change task status to canceled so it doesn't appear in flex or show up as a pending task
client.taskrouter.workspaces(workspace)
.tasks(task)
.update({
assignmentStatus: 'canceled',
reason: 'Call forwarded'
})
.then(task => {
console.log(`__[trEventStream]__: [SUCCESS] ~> Task canceled`);
return callback(null, response);
}).catch(err => {
console.log(`__[trEventStream]__: [ERROR] ~> Task not marked complete: `, err);
// doesn't warrant reponse 500 since call still forwarded :)
return callback(null, response);
});
}).catch(err => {
console.log(`__[trEventStream]__: [ERROR] ~> Task failed to forward to external DID: `, err);
response.setStatusCode(500);
return callback(err, response);
});
break;
default:
return callback(null, response);
}
};
callForward.js:
exports.handler = function(context, event, callback) {
console.log(`forwarding call`);
// set-up the variables that this Function will use to forward a phone call using TwiML
// REQUIRED - you must set this
let phoneNumber = event.PhoneNumber || context.NUMBER;
// OPTIONAL
let callerId = event.CallerId || null;
// OPTIONAL
let timeout = event.Timeout || null;
// OPTIONAL
let allowedCallers = event.allowedCallers || [];
let allowedThrough = true;
if (allowedCallers.length > 0) {
if (allowedCallers.indexOf(event.From) === -1) {
allowedThrough = false;
}
}
// generate the TwiML to tell Twilio how to forward this call
let twiml = new Twilio.twiml.VoiceResponse();
let dialParams = {};
if (callerId) {
dialParams.callerId = callerId;
}
if (timeout) {
dialParams.timeout = timeout;
}
if (allowedThrough) {
twiml.dial(dialParams, phoneNumber); // making call :)
}
else {
twiml.say('Sorry, you are calling from a restricted number. Good bye.');
}
// return the TwiML
callback(null, twiml);
};
Any kind of help and/or guidance will be appreciated.
Twilio developer evangelist here.
When you redirect a call from a task, its task is cancelled with the reason "redirected" so you don't need to cancel it yourself.
Your code was failing to update the task occasionally because of a race condition between your code and the task getting cancelled by Twilio.
I need to make a POST request and send some data. I'm using the service worker sync to handle offline situation.
But is there a way to pass the POST data to the service worker, so it makes the same request again?
Cause apparently the current solution is to store requests in some client side storage and after client gets connection - get the requests info from the storage and then send them.
Any more elegant way?
PS: I thought about just making the service worker send message to the application code so it does the request again ... but unfortunately it doesn't know the exact client that registered the service worker :(
You can use fetch-sync
or i use postmessage to fix this problem, which i agree that indexedDB looks trouble.
first of all, i send the message from html.
// send message to serviceWorker
function sync (url, options) {
navigator.serviceWorker.controller.postMessage({type: 'sync', url, options})
}
i got this message in serviceworker, and then i store it.
const syncStore = {}
self.addEventListener('message', event => {
if(event.data.type === 'sync') {
// get a unique id to save the data
const id = uuid()
syncStore[id] = event.data
// register a sync and pass the id as tag for it to get the data
self.registration.sync.register(id)
}
console.log(event.data)
})
in the sync event, i got the data and fetch
self.addEventListener('sync', event => {
// get the data by tag
const {url, options} = syncStore[event.tag]
event.waitUntil(fetch(url, options))
})
it works well in my test, what's more you can delete the memory store after the fetch
what's more, you may want to send back the result to the page. i will do this in the same way by postmessage.
as now i have to communicate between each other, i will change the fucnction sync into this way
// use messagechannel to communicate
sendMessageToSw (msg) {
return new Promise((resolve, reject) => {
// Create a Message Channel
const msg_chan = new MessageChannel()
// Handler for recieving message reply from service worker
msg_chan.port1.onmessage = event => {
if(event.data.error) {
reject(event.data.error)
} else {
resolve(event.data)
}
}
navigator.serviceWorker.controller.postMessage(msg, [msg_chan.port2])
})
}
// send message to serviceWorker
// you can see that i add a parse argument
// this is use to tell the serviceworker how to parse our data
function sync (url, options, parse) {
return sendMessageToSw({type: 'sync', url, options, parse})
}
i also have to change the message event, so that i can pass the port to sync event
self.addEventListener('message', event => {
if(isObject(event.data)) {
if(event.data.type === 'sync') {
// in this way, you can decide your tag
const id = event.data.id || uuid()
// pass the port into the memory stor
syncStore[id] = Object.assign({port: event.ports[0]}, event.data)
self.registration.sync.register(id)
}
}
})
up to now, we can handle the sync event
self.addEventListener('sync', event => {
const {url, options, port, parse} = syncStore[event.tag] || {}
// delete the memory
delete syncStore[event.tag]
event.waitUntil(fetch(url, options)
.then(response => {
// clone response because it will fail to parse if it parse again
const copy = response.clone()
if(response.ok) {
// parse it as you like
copy[parse]()
.then(data => {
// when success postmessage back
port.postMessage(data)
})
} else {
port.postMessage({error: response.status})
}
})
.catch(error => {
port.postMessage({error: error.message})
})
)
})
At the end. you cannot use postmessage to send response directly.Because it's illegal.So you need to parse it, such as text, json, blob, etc. i think that's enough.
As you have mention that, you may want to open the window.
i advice that you can use serviceworker to send a notification.
self.addEventListener('push', function (event) {
const title = 'i am a fucking test'
const options = {
body: 'Yay it works.',
}
event.waitUntil(self.registration.showNotification(title, options))
})
self.addEventListener('notificationclick', function (event) {
event.notification.close()
event.waitUntil(
clients.openWindow('https://yoursite.com')
)
})
when the client click we can open the window.
To comunicate with the serviceworker I use a trick:
in the fetch eventlistener I put this:
self.addEventListener('fetch', event => {
if (event.request.url.includes("sw_messages.js")) {
var zib = "some data";
event.respondWith(new Response("window.msg=" + JSON.stringify(zib) + ";", {
headers: {
'Content-Type': 'application/javascript'
}
}));
}
return;
});
then, in the main html I just add:
<script src="sw_messages.js"></script>
as the page loads, global variable msg will contain (in this example) "some data".
I have an app with a basic 'shell' of HTML, CSS and JS. The main content of the page is loaded via multiple ajax calls to an API that is at another URL to the one my app is running on. I have set up a service-worker to cache the main 'shell' of the application:
var urlsToCache = [
'/',
'styles/main.css',
'scripts/app.js',
'scripts/apiService.js',
'third_party/handlebars.min.js',
'third_party/handlebars-intl.min.js'
];
and to respond with the cached version when requested. The problem I am having is that the response of my ajax calls are also being cached. I'm pretty sure that I need to add some code to the fetch event of the service-worker that always get them from the network rather than looking in the cache.
Here is my fetch event:
self.addEventListener('fetch', function (event) {
// ignore anything other than GET requests
var request = event.request;
if (request.method !== 'GET') {
event.respondWith(fetch(request));
return;
}
// handle other requests
event.respondWith(
caches.open(CACHE_NAME).then(function (cache) {
return cache.match(event.request).then(function (response) {
return response || fetch(event.request).then(function (response) {
cache.put(event.request, response.clone());
return response;
});
});
})
);
});
I'm not sure how I can ignore the requests to the API. I've tried doing this:
if (request.url.indexOf(myAPIUrl !== -1) {
event.respondWith(fetch(request));
return;
}
but according to the network tab in Chrome Dev Tools, all of these responses are still coming from the service-worker.
You do not have to use event.respondWith(fetch(request)) to handle requests that you want to ignore. If you return without calling event.respondWith browser will fetch the resource for you.
You can do something like:
if (request.method !== 'GET') { return; }
if (request.url.indexOf(myAPIUrl) !== -1) { return; }
\\ handle all other requests
event.respondWith(/* return promise here */);
IOW as long as you can determine synchronously that you don't want to handle the request you can just return from the handler and let the default request processing to take over. Check out this example.
What I'm trying to do (though I fully suspect there's a better way to do it) is to send HTTP requests to a range of hosts on my network. I can hit every host by calling WinJS.xhr in a loop. However, it takes too long to complete the range.
Inspecting in Fiddler shows that a dozen or so requests are sent at a time, wait to time out, and then move on to the next dozen or so. So I figured I'd try to reduce the timeout for each request. For my needs, if the host doesn't respond in 500 ms, it's not going to respond.
Following the documentation, I tried wrapping the call to WinJS.xhr in a call to WinJS.Promise.timeout with a small enough setting, but there was no change. Changing the promise timeout didn't really affect the actual request.
A little more searching led me to a suggestion whereby I could modify the XMLHttpRequest object that WinJS.xhr uses and set the timeout on that. This worked like a charm in terms of blasting out requests at a faster rate. However, there seems to be a side-effect.
Watching the requests in Fiddler, about a dozen or so fire off very quickly and then the whole thing ends. The "next dozen or so" never come. Sometimes (based on the semi-randomness of asynchronous calls) the first dozen or so that shows up in fiddler includes 9-10 from the low and of the range and 2-3 from the top end of the range, or close to it.
Is there something else I can try, or some other way to accomplish the end goal here? (Within the scope of this question the end goal is to send a large number of requests in a reasonable amount of time, but any suggestions on a better overall way to scan for a particular service on a network is also welcome.)
Can you write out the code you're using for timeout, i wrote something like this but it wasn't working, so I'm curious as to how you're doing it:
var timeoutFired = function () {
console.log("derp");
};
var options = {
url: "http://somesite.com",
responseType: "document",
customRequestInitializer: function (req) {
req.timeout = 1;
req.ontimeout = timeoutFired;
//do something with the XmlHttpRequest object req
}
};
WinJS.xhr(options).
....
Here are some alternatives that you may find helpful, not sure how/why timeout wasn't working but I tried to write out a custom timeout function:
(function (global) {
var options = {
url: "http://something.com",
responseType: "document",
};
var request = WinJS.xhr(options).then(
function (xmlHttpRequest) {
console.log("completed");
},
function (xmlHttpRequest) {
//error or cancel() will throw err
console.log("error"+ xmlHttpRequest.message);
},
function (xmlHttpRequest) {
console.log("progress")
});
function waitTime() {
return new WinJS.Promise(
function (complete, error, progress) {
var seconds = 0;
var interval = window.setInterval(
function () {
seconds++;
progress(seconds);
//prob should be called milliseconds
if (seconds > 5) {
window.clearInterval(interval);
complete();
}
}, 100);
});
};
waitTime().done(
function () {
console.log("complete");
request.cancel();
},
function () {
console.log("error")
},
function (seconds) {
console.log("progress:" + seconds)
});
});
Another cool little trick is using promise.any (vs .join) which fires off when one OR the other finishes first, so taking that into account you can write something like this:
(function (global) {
var options = {
url: "http://url.com",
responseType: "document",
};
var request = {
runRequest: function () {
return WinJS.xhr(options).then(
function (xmlHttpRequest) {
console.log("completed");
},
function (xmlHttpRequest) {
//error or cancel() will throw err
console.log("error" + xmlHttpRequest.message);
},
function (xmlHttpRequest) {
console.log("progress")
});
}
};
WinJS.Promise.any([WinJS.Promise.timeout(500), request.runRequest()]).done(
function () {
console.log("any complete");
});
})();