How to write twitter tweets to file using Spark Streaming Java API - twitter

I have the following code segment and I am trying to collect some tweets from the timeline of say "stefanjdecker". The actual credentials of mine have been changed to make myself anonymous.
SparkConf conf = new SparkConf().setMaster("local").setAppName("SparkTwitterHelloWorldExample");
JavaStreamingContext jssc = new JavaStreamingContext(conf, new Duration(600));
String consumerKey = "key";
String accessTokenSecret = "secret";
String consumerSecret = "consumer";
String accessToken = "token";
System.setProperty("twitter4j.oauth.consumerKey", consumerKey);
System.setProperty("twitter4j.oauth.consumerSecret", consumerSecret);
System.setProperty("twitter4j.oauth.accessToken", accessToken);
System.setProperty("twitter4j.oauth.accessTokenSecret", accessTokenSecret);
String[] filters = new String[] {"stefanjdecker"};
JavaReceiverInputDStream<Status> twitterStream = TwitterUtils.createStream(jssc,filters);
twitterStream.dstream().saveAsTextFiles("/Output/twitter","txt");
The above code runs successfully. However, tweets are not being written to the path I mentioned. Please someone help me to sort this out.

Related

Google OAuth2 Java code asking permissions every time

I am using the following with a google-client-secret.json file and trying to run this as just a java application in eclipse. I want to store the permissions so once I accept the permissions it doesn't ask again. Right now it is prompting everytime. After that everything works as expected and writes to my google sheets.
public static Credential authorizeSHEETS() throws IOException, GeneralSecurityException {
File fileIn = new File("src/jg/sos/orders/google-sheets-client-secret.json");
// InputStream in = GoogleAuthorizeUtil.class.getResourceAsStream("src/jg/sos/orders/google-sheets-client-secret.json");
InputStream in = new FileInputStream(fileIn);
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JacksonFactory.getDefaultInstance(), new InputStreamReader(in));
List<String> scopes = Arrays.asList(SheetsScopes.SPREADSHEETS);
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), clientSecrets, scopes).setDataStoreFactory(new MemoryDataStoreFactory())
.setAccessType("offline").setApprovalPrompt("auto").build();
Credential credential = new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user");
return credential;
}
Any ideas on how to only have this prompt me the first time for permissions, then the next time I run this it will not?
Thanks for the help!
JJ
So found an answer to this in case anyone comes across. I used a service account instead, and downloaded the json file for it and placed in my project.
Then I just referenced it instead, and saved the token using DataStoreFactory as below:
public static Credential authorizeSHEETS() throws IOException, GeneralSecurityException {
File fileIn = new File("src/jg/sos/orders/google-sheets-client-secret.json");
// InputStream in = GoogleAuthorizeUtil.class.getResourceAsStream("src/jg/sos/orders/google-sheets-client-secret.json");
InputStream in = new FileInputStream(fileIn);
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JacksonFactory.getDefaultInstance(), new InputStreamReader(in));
List<String> scopes = Arrays.asList(SheetsScopes.SPREADSHEETS);
FileDataStoreFactory dataStoreFactory = new FileDataStoreFactory(new File("src/jg/sos/orders"));
GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(GoogleNetHttpTransport.newTrustedTransport(), JacksonFactory.getDefaultInstance(), clientSecrets, scopes)
.setDataStoreFactory(dataStoreFactory)
.setAccessType("offline").setApprovalPrompt("auto").build();
Credential credential = new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user");
System.out.println("token" + credential.getAccessToken());
return credential;
}

Twitterizer Problems

I am trying to figure out if twitterizer is still relevant. The library looks really old. And the current project i am working on uses it. I am using the following code...
private static OAuthTokens OAuthTokens()
{
var accessToken = "<insert>";
var accessTokenSecret = "<insert>";
var consumerKey = "<insert>";
var consumerSecret = "<insert>";
return new OAuthTokens
{
ConsumerKey = consumerKey,
ConsumerSecret = consumerSecret,
AccessToken = accessToken,
AccessTokenSecret = accessTokenSecret
};
}
if i do the following call
var responseFollowersIds = TwitterFriendship.FollowersIds(credentials, new UsersIdsOptions()
{
UseSSL = true,
APIBaseAddress = "http://api.twitter.com/1.1/",
ScreenName = screenName
});
I can see that i have followers and that the request being made looks like this
https://api.twitter.com/1.1/followers/ids.json?screen_name=screenname&cursor=-1
notice that i set the APIBaseAddress to the latest version, but if i try and get the followers i use the following
var responseFollowers = TwitterFriendship.Followers(credentials, new FollowersOptions
{
ScreenName = screenName,
UseSSL = true,
APIBaseAddress = "http://api.twitter.com/1.1/",
});
But then i get an error
{"errors":[{"message":"Sorry, that page does not exist","code":34}]}
upon further investigation i realized that TwitterFriendship.Followers request url is wrong and that is buried deep in the twitterizer code.
The request made is
https://api.twitter.com/1.1/statuses/followers.json?cursor=-1&screen_name=screenname
and should be
https://api.twitter.com/1.1/followers/list.json
according to the new twitter documentation.
Is there an easy way to fix it?
A little bit late, but just in case anyone faces this problem again. You can easily fix it by modifiying the code of Twitterizer at \Methods\User\FollowersCommand.cs, look for the constructor and change the URL of the method:
public FollowersCommand(OAuthTokens tokens, FollowersOptions options)
: base(HTTPVerb.GET, "followers/list.json", tokens, options)

JIRA rest api to fetch the activity stream

I am trying to get activity stream of my jira instance using the below api and it is not working , can anybody point me in the right direction ?
You should check this page out: https://developer.atlassian.com/docs/atlassian-platform-common-components/activity-streams/consuming-an-activity-streams-feed
The Atom feed of the activity stream works well only if you also log in in your feed reader.
Here is an example of consuming the activity stream through the Jira API using Basic Authentication. This is in C#, but the basic pattern can be applied anywhere:
string myJiraUsername = "username";
string myJiraPassword = "password"; //or API token
string authenticationHeaderValue = Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes(myJiraUsername + ":" + myJiraPassword));
System.Net.Http.HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", authenticationHeaderValue);
Task<HttpResponseMessage> task = client.GetAsync("https://mycompany.atlassian.net/activity");
task.Wait();
HttpResponseMessage response = task.Result;
string resultOfApiCall = "";
if (response.IsSuccessStatusCode)
{
resultOfApiCall = response.Content.ReadAsStringAsync().Result;
Console.WriteLine("This was returned by your API request:\n" + resultOfApiCall);
}

Spreadsheets API Can not update a read-only feed

I'm trying to add new spreadsheet if it not exists with GData Spreadsheet API for .NET but it gives me following exception:
Can not update a read-only feed
Here's my code:
var service = new SpreadsheetsService("<my-app>");
service.setUserCredentials("<login>", "<password>");
// Instantiate a SpreadsheetQuery object to retrieve spreadsheets.
SpreadsheetQuery query = new SpreadsheetQuery();
var title = "test";
query.Title = title;
// Make a request to the API and get all spreadsheets.
SpreadsheetFeed feed = service.Query(query);
if (!feed.Entries.Any())
{
var worksheet = new WorksheetEntry(20, 20, title);
service.Insert(feed, worksheet);
}
Through Fiddler I see that I'm doing request to:
GET /feeds/spreadsheets/private/full?title=test
and it goes fine, but I don't see any requests for updating data. I suppose that I should change somehow SpreadsheetQuery to make it capable to write data, but I can't find how.
It's me being inattentive because google documentation on Spreadsheet API says:
It is possible to create a new spreadsheet by uploading a spreadsheet
file via the Google Drive API. The Spreadsheets API does not currently
provide a way to delete a spreadsheet, but this is also provided in
the Google Drive API. For testing purposes, you may create a
spreadsheet manually or upload one.
So I basically installed GoogleDrive API with Nuget. And then added following method for adding file:
private static void AddFile(string title)
{
var clientID = "put here a clientID";
var clientSecret = "put here a clientSecret";
UserCredential credential = GoogleWebAuthorizationBroker.AuthorizeAsync(
new ClientSecrets
{
ClientId = clientID,
ClientSecret = clientSecret,
},
new[] { DriveService.Scope.Drive },
"here goes your account",
CancellationToken.None).Result;
// Create the service.
var service = new DriveService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = "Drive API Sample",
});
var body = new Google.Apis.Drive.v2.Data.File();
body.Title = title;
//body.Description = "A test document";
body.MimeType = "application/vnd.google-apps.spreadsheet";
service.Files.Insert(body).Execute();
}
When I run code above at the first time - I received an exception that said
Could not load file or assembly
'Microsoft.Threading.Tasks.Extensions.Desktop, Version=1.0.16.0
I run these in Package Manager Console:
Uninstall-Package Microsoft.Bcl.Async -Force
Install-Package Microsoft.Bcl.Async
and it worked. Hope it would help somebody who will stumble over the same issue.

BillQuery for getting bills giving Unauthorized error in Intuit

I'm getting "Unauthorized" error while trying to use filter in Intuit:
Exception Details: Intuit.Ipp.Exception.InvalidTokenException: Unauthorized
The code below is used to setup the Service Context:
string AppToken = ConfigurationManager.AppSettings["applicationToken"].ToString(CultureInfo.InvariantCulture);
String realmId = HttpContext.Current.Session["realm"].ToString();
String accessToken = HttpContext.Current.Session["accessToken"].ToString();
String accessTokenSecret = HttpContext.Current.Session["accessTokenSecret"].ToString();
String consumerKey = ConfigurationManager.AppSettings["consumerKey"].ToString(CultureInfo.InvariantCulture);
String consumerSecret = ConfigurationManager.AppSettings["consumerSecret"].ToString(CultureInfo.InvariantCulture);
IntuitServicesType intuitServiceType = (IntuitServicesType)HttpContext.Current.Session["intuitServiceType"];
OAuthRequestValidator oauthValidator = new OAuthRequestValidator(accessToken, accessTokenSecret, consumerKey, consumerSecret);
context1 = new ServiceContext(oauthValidator, AppToken, realmId, IntuitServicesType.QBO);
Query for retrieving the last modified bills is as below:
List<Bill> CustomerBills = billQry.ExecuteQuery<Bill>(context1).ToList<Bill>();
Please let me know, which parameter value I'm passing incorrectly.
The following code .NET DevKit code sends a malformed request body and results in a OAuth signature error. This is a bug in the DevKit.
BillQuery billQry = new BillQuery();
billQry.LastUpdatedTime = DateTime.Now.AddDays(-20);
billQry.SpecifyOperatorOption(FilterProperty.LastUpdatedTime, FilterOperatorType.AFTER);
billQry.LastUpdatedTime = DateTime.Now;
billQry.SpecifyOperatorOption(FilterProperty.LastUpdatedTime, FilterOperatorType.BEFORE);
billQry.PageNumber = 1;
billQry.ResultsPerPage = 15;
billQry.SpecifySortOption(SortProperty.LastUpdatedTime, SortOrderOption.HighToLow);
List<Intuit.Ipp.Data.Qbo.Bill>CustomerBills =billQry.ExecuteQuery<Intuit.Ipp.Data.Qbo.Bill>(context).ToList();
The workaround is to modify the sample code below to make the request and deserialize the response into Bill objects.
Sample Code on Pastebin

Resources