Twitter API v2 Streaming with Retrofit or OkHttp - twitter

I am trying integrate the new Twitter API specifically the streaming tweets part in my android app, I am using Retrofit for my http calls.
When I try to make the call to get the streaming tweets it just hangs and does not return anything.
this is my retrofit call
#Streaming
#GET("tweets/search/stream")
suspend fun getFilteredStream(#Header("Authorization") token:String)
I then tried making a call with just OkHttp as shown in the documentation I get a successful response but I dont know how to stream the data.
I can make the call successfully via a curl call and see the data no problem.
How do I stream the data via retrofit or OkHttp
Update:
With OkHttp I was able to get data by doing this
val client: OkHttpClient = OkHttpClient().newBuilder()
.build()
val request: Request = Request.Builder()
.url("https://api.twitter.com/2/tweets/search/stream")
.method("GET", null)
.build()
val response: Response = client.newCall(request).execute()
val source = response.body?.source()
val buffer = Buffer()
while(!source!!.exhausted()){
response.body?.source()?.read(buffer, 8192)
val data = buffer.readString(Charset.defaultCharset())
}
data holds the string data representation of multiple tweet objects but how do I read one tweet at a time, or parse the response like this?

From the docs, I think you'll need to combine the two examples you have.
https://github.com/square/retrofit/blob/108fe23964b986107aed352ba467cd2007d15208/retrofit/src/main/java/retrofit2/http/Streaming.java
Treat the response body on methods returning {#link ResponseBody ResponseBody} as is, i.e. without converting the body to {#code byte[]}.
And example calling code
https://github.com/square/retrofit/blob/108fe23964b986107aed352ba467cd2007d15208/retrofit/src/test/java/retrofit2/CallTest.java#L599
I suspect, dependending on the JSON API you use you may be able to use a Streaming API from the response body. But if not you could split on either just newline or newline followed by { on next line and parse individually. Sorry I can't help here.
See https://en.wikipedia.org/wiki/JSON_streaming#Concatenated_JSON

Related

How to parse attachment values with strongGrid inbound webhook

Hello there I have setup successfully inbound webhook with strongGrid in net core 3.1.
The endpoint gets called and I want to parse value inside the attachment which is csv file.
The code I am using is following
var parser = new WebhookParser();
var inboundEmail = await parser.ParseInboundEmailWebhookAsync(Request.Body).ConfigureAwait(false);
await _emailSender.SendEmailAsyncWithSendGrid("info#mydomain.com", "ParseWebhook1", inboundEmail.Attachments.First().Data.ToString());
Please note I am sending an email as I don t know how to debug webhook with sendgrid as I am not aware of any cli.
but this line apparently is not what I am looking for
inboundEmail.Attachments.First().Data.ToString()
I am getting this on my email
Id = a3e6a543-2aee-4ffe-a36a-a53k95921998, Tag = HttpMultipartParser.MultipartFormDataParser.ParseStreamAsync, Length = 530 bytes
the csv I need to parse has 3 fields Sku productname and quantity I'd like to get sku values.
Any help would be appreciated.
The .Data property contains a Stream and invoking ToString on a stream object does not return its content. The proper way to read the content of a stream in C# is something like this:
var streamReader = new StreamReader(inboundEmail.Attachments.First().Data);
var attachmentContent = await streamReader.ReadToEndAsync().ConfigureAwait(false);
As far as parsing the CSV, there are literally thousands of projects on GitHub and hundreds on NuGet with the keyword 'CSV'. I'm sure one of them will fit your needs.

Cannot build up a string using StringBuffer and a server response

I'm trying to fetch data from this link via Dart. Since I'm making use of dart:io library's HttpClientResponse based instance to listen to the data obtained from the above link, therefore I thought that an instance of StringBuffer would be the best option to capture the received data. It would help me to build the response string in an incremental fashion. But it seems like I'm not making proper use of StringBuffer, because in the end the response string (stored in receivedBuffer) remains empty.
Code:
import 'dart:io';
import 'dart:convert';
void main() async
{
StringBuffer receivedBuffer = new StringBuffer("");
String url = "https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty";
HttpClient client = new HttpClient();
HttpClientRequest request = await client.getUrl(Uri.parse(url));
HttpClientResponse response = await request.close();
print("[info] Fetch successful. Proceeding to transform received data ...");
response.transform(utf8.decoder).listen((contents) => receivedBuffer.write(contents));
print("[info] Done. Contents:\n");
print(receivedBuffer);
}
Output :
[info] Fetch successful. Proceeding to transform received data ...
[info] Done. Contents:
Also, instead of receivedBuffer.write(contents), if I were to write print(contents), then all the
required data is printed as one would expect it to. But while trying to write the contents to recievedBuffer it seems like receivedBuffer wasn't even updated once.
I read this article, and tried to incorporate the answer present over there in my code. To be precise, I made use of Completer instance to take care of my issue, but it didn't help.
What's the issue in the above provided code?
You are not waiting for the stream to complete.
In the listen call, you set up a receiver for stream events, but you don't wait for the events to arrive.
You can either add an onDone parameter to the listen call and do something there, but more likely you will want to just wait for it here, and then I recommend:
await response.transform(utf8.decoder).forEach(receivedBuffer.write);
Using forEach is usually what you want when you are calling listen, but not remembering the returned subscription. Alternatively use an await for:
await for (var content in response.transform(utf8.decoder)) {
receivedBuffer.write(content);
}
(which corresponds to a forEach call in most ways).

Order of object properties is changed while posting data from ionic native http client

I am working on ionic app which will be used on android and iOS platforms. App uses one endpoint to post user data to the backend. Data posted on a backend is as follows:
{
"name": "Citizen Foo",
"emailAddress": "citizen.foo#gmail.com",
"role": "citizen"
}
For security purpose every request which is being sent is validated. In order to do this client sends authorization header with every request. Backend creates one for every request and matches it with one sent by client and then only responds otherwise throw an exception. For creating the authorization header, data sent over post request is also a part of the logic. I have simplified this logic becuase the actual problem is different but this part is important to understand the problem.
Following is the sample code on client side:
var sRequestBody = JSON.stringify(data);
var requestBodyBytes = this.getByteArray(unescape(encodeURIComponent(sRequestBody)));
var authHeader = md5.base64(requestBodyBytes);
var headers = {};
headers['Authorization'] = authHeader;
this.nativeHttp.setDataSerializer('json');
this.nativeHttp.clearCookies();
this.nativeHttp.setSSLCertMode('nocheck');
return Observable.fromPromise(this.nativeHttp.post(url, data, headers));
Then, backend which is asp.net web api, calculates authorization in the same way and matches it with one send by client and gives the response if it matches.
var rawContent = await content.ReadAsByteArrayAsync();
var stringContent = content.ReadAsStringAsync().Result;
var hash = md5.ComputeHash(rawContent);
var authorization = Convert.ToBase64String(hash);
if (authHeader != authorization)
throw;
When the above call is made from android,
value of dataString on client is
"{"name":"Citizen Foo","emailAddress":"citizen.foo#gmail.com","role":"citizen"}"
value of stringContent on server is
"{""name"":""Citizen Foo"",""emailAddress"":""citizen.foo#gmail.com"",""role"":""citizen""}"
and it allows request coming from android app.
When the same is run on iOS and the post user call is made,
Value of dataString on client is
"{"name":"Citizen Foo","emailAddress":"citizen.foo#gmail.com","role":"citizen"}"
Value of stringContent on server is
"{""name"":""Citizen Foo"",""role"":""citizen"",""emailAddress"":""citizen.foo#gmail.com""}"
and it does not allow requst coming from iOS app.
And the only reason it is happening is because of the way user object is serialized/received at the backend when request is made from iOS. Order of properties while sending is name,emailAddress,role. While it is received with order name,role,emailAddress. Thus, the authorization value calculated on server side is different than authHeader coming from client and the call is terminated.
stringContent was added on server side just to debug and understand what is being received at server. Order of the object properties on client is different than object properties received on server side. Is there a way to maintain the order of object properties when the request is made from iOS platform? Direction in any way to solve this problem is appreciated.
One solution that worked for us is this:
this.httpClient.setDataSerializer('utf8');
response = await this.httpClient.post(url, JSON.stringify(postBody), {'Content-Type': 'application/json'});
So key points here are:
utf8 as data serializer
body has to be stringified
header 'Content-Type': 'application/json' has to be set

Accessing Twitter with Akka Camel to return JSON

I was using an HTTP POST method using the URL
"https://stream.twitter.com/1.1/statuses/filter.json" and in the body I was posting the key/value I wanted to get tweets from - for example "track=london". This was working fine.
Now I am trying to switch to AKKA-CAMEL and I am using their twitter consumer. I am using an endpoint URL of:
def endpointUri: String = s"twitter:////search?type=direct&keywords=${Settings.queryList()}&consumerKey=${tweeterCredentials.consumerKey}&consumerSecret=${tweeterCredentials.consumerSecret}&accessToken=${tweeterCredentials.accessToken}&accessTokenSecret=${tweeterCredentials.accessTokenSecret}"
I get a response from twitter but it is not in JSON and it is not the same information about the tweet as before. It just return the tweet text but before I was getting the whole metadata which I need to analyze.
Does somebody knows how to configure Camel URI to return JSON and the whole metadata as before?
Thanks
I got this to work by using the following syntax:
def endpointUri: String = s"twitter://streaming/filter?type=event&keywords=${Settings.queryList()}&consumerKey=${tweeterCredentials.consumerKey}&consumerSecret=${tweeterCredentials.consumerSecret}&accessToken=${tweeterCredentials.accessToken}&accessTokenSecret=${tweeterCredentials.accessTokenSecret}"
Where: Settings.queryList return a comma separated list of keyworkds. The object tweeterCredentials holds the keys from Tweeter to access the site.
Also it is necessary to set autoAck like this in Camel:
override def autoAck = true
This prevents a timeout exception.

ios swift 2.1 - unable to send Patch request with body

I'm trying to write a http rest client for my webservice and i need to send some PATCH requestes with data in the body.
I'm using the JUST library for sending requests ( https://github.com/JustHTTP/Just )
My express application just doesn't see the request.
Here's some code (i'm testing in playground, and everything went fine with other kind of requests like put, post...)
headers = ["accept":"application/json","content-type":"application/json","authorization":"key"] //key is ok
var data = ["id":3, "quantity":6]
var r = Just.patch("http://api.marketcloud.it/v0/carts/1233", headers:headers, data:data) //1233 is a cart Id
print(r)
print(r.json)
The method Just.patch returns an HTTPResult Object.
this says 'OPTIONS http://api.marketcloud.it/v0/carts/13234 200'
Also this object should contain a json, but it's 'nil'.
On the server-side, my express applications doesn't receive the request (it just logs an 'OPTION', but nothing else).
Could this be a playground-related problem? Or a just-related one?
Thanks for any suggestion
I managed to contact the library's author via twitter and he fixed the bug and answered me in less than 24h!
Here's the new release of the library.
https://github.com/JustHTTP/Just/releases

Resources