how to download file on website behind oauth - oauth-2.0

I'm trying to access a file on a site that I want to pull every day as it updates.
https://www.website.com/path/file.json
If I go through the UI, I enter my username and password to the oath system which then lets me through.
Trying to use python to emulate that:
using client_id /secret I get back a bearer token: abcdxyz
headers = {'Authorization':'Bearer abcdxyz'}
requests.get('https://www.website.com/path/file.json,headers = headers)
cookie= {'Cookie': 'abcdxyz'}
requests.post('https://www.website.com/path/file.json, cookies=cookies)
Im pretty sure im doing it the wrong way but I don't know how to look up this problem.

Related

Cannot repeat steps on how to get OAuth2.0 Access Token on Postman

On Postman, I can get new access token for OAuth2.0 by providing callback URL, auth URL and client ID.
I want to break this task down on JMeter because I cannot find this function there. From my understanding, it is divided into authenticate -> authorise -> call back.
Authenticate
URL = https://xxxxx/login
Result = Authorising URL
Authorise
URL = https://xxxxx/oauth/authorize?client_id=mmm&redirect_uri=https://yyyyy/auth/callback&response_type=code
Result = code (e.g. zzz)
Call back
URL = https://yyyyy/auth/callback?code=zzz
Result = token
As I used HTTP(S) Test Script Recorder on JMeter, I got the three actions mentioned above. When I reran them, it told me this error on Authenticate part: <oauth><error_description>Full authentication is required to access this resource</error_description><error>unauthorized</error></oauth>.
To make sure that it was not about the program I use, I did it on Postman and found this error as well.
I wonder how I can break OAuth2.0 Get New Access Token feature into basic API settings in order to get access token on Postman or JMeter.
Dont' compare these tools:
Postman is an Electron application, it's basically a heavily customised Chromium web browser + NodeJS
According to JMeter main page
JMeter is not a browser, it works at protocol level. As far as web-services and remote services are concerned, JMeter looks like a browser (or rather, multiple browsers); however JMeter does not perform all the actions supported by browsers. In particular, JMeter does not execute the Javascript found in HTML pages. Nor does it render the HTML pages as a browser does (it's possible to view the response as HTML etc., but the timings are not included in any samples, and only one sample in one thread is ever displayed at a time).
If you can obtain the token using Postman you can just add HTTP Header Manager to your JMeter Test plan and configure it to send Authorization header with the value of Bearer YOUR_TOKEN_FROM_POSTMAN and JMeter should let you in.
After testing, I found that Postman's OAuth 2.0 Get New Access Token popped up a login page of targeted URL where I needed to fill in the username and the password so that the token could be obtained.
As I tried breaking down APIs required for this login, it required GET of that https://yyyyy and POST of that https://xxxxx/login. Click Send for POST, with username and password contained in form-data, then click Send for GET. The GET response would contain such the token.
However, just putting the aforementioned GET and POST APIs into one thread group did not work on JMeter. As I used HTTP(S) Test Script Recorder to no avail, I went with BlazeMeter and realised that it was using Transaction Controller containing 1) GET of https://yyyyy 2) POST of https://xxxxx/login. With these two arranged top-down, the job would be successful. The token was contained in the response of 2).
For now, this has been my discovery which answers my question.
Try BlazeMeter.

Confused connecting to API in Postman

I am doing a Backend rails thinkster tutorial that involves creating a clone of medium. I am at a step that requires testing the routes in Postman. Thinkster provides an API that has built in requests and pre-made connections.
But the tutorial goes from step "download Postman" to step "send request." There's no explanation about how to initially start using Postman.
So the first test is to test if you can create a new user with a Register request. I am guessing this is in the Auth folder of the API as there is a Post request called "Register".
I am not seeing an area though that suggests I can make a user. All of the items in my collection are have a request Url that starts with {{apiUrl}}. Example: login's POST request is {{apiUrl}}/login. And if I hover over {{apiUrl}}, it says "unresolved variable: variable is not defined in current environment."
Could anyone help me get going with these tests? Below is a gif of my Postman setup as well as the list of tests the tutorial wants run. I am sure I am not providing something that I may need to in order to get help on this. Please let me know if you are not being presented with all info needed! Thanks.
Test out Postman authentication functionality using Postman
You should be able to:
Create an account using the Register request in Postman
Test the Login endpoint using Postman
Try registering another user with the same email or username, you
should get an error back from the backend
Test the Current User endpoint using Postman
Try logging in to the user you created with an invalid password, you
should get an error back from the backend
Try updating the email, username, bio, or image for the user
My friend I will try to answer at least the question about {{apiUrl}}
This are variables that can be global or environment, obviously global apply to all projects and environment, you will need to select the environment in which they apply.
Now to make it work, do the following.
Copy the url to your API server.
Replace it with the magic variable between {{}}
Go to the gear where you manage this global or environment variables «lest make a local one»
Click on add
first name your environment however you like, the go to the grid and make a variable placing the name inside the key and your servir url on the value side. Then save.
Don't forget to select the environment where you are working.
Thats it now your {{variable}} should work. Now you can use this variables all over postman, they are very helpful. If you are using a JWT token, you will be able to assign it to a variable and the use it on your postman api.

How to manage API side authorization for Google?

I'm responsible for the API side of our product. We have several different clients, from browsers to iPads to Chromebooks. Right now, all our authentication is done directly from the client to our API, with username & password.
I've inherited some code that does authentication using OAuth, with the usual username/password setup. So inside my OwinAuthConfig class, I have:
var oAuthAuthorizationOptions = new OAuthAuthorizationServerOptions
{
TokenEndpointPath = new PathString("/Authenticate"),
Provider = new MyAuthorizationProvider(),
AccessTokenExpireTimeSpan = TimeSpan.FromDays(14),
AllowInsecureHttp = true
};
app.UseOAuthAuthorizationServer(oAuthAuthorizationOptions);
Then, through some dark magic, this connects up with my MyAuthorizationProvider class (which inherits OAuthAuthorizationServerProvider), and on login, this invokes the method:
public override Task GrantResourceOwnerCredentials(OAuthGrantResourceOwnerCredentialsContext context)
{ ... }
where context contains the important stuff (Username and Password) which I can then use to authenticate the user, build his claims, create an AuthenticationTicket and this information then magically gets returned to the client with the access token etc.
All well and good.
Now I have a new requirement - to allow 3rd party authentication from Google. In this case, the client app (iOS/Android/whatever) does the authentication with Google, and they should just pass the token (and any other required info) to me on the API side. On my side I then need to re-authenticate the Google token, and get all the user info from Google (email, name, etc.), from which I should then again link that to our User table, build up the claims etc. and return a new token to the client, which will be used in all subsequent calls.
Being kinda new to the whole OWIN pipeline thing, I'm not sure exactly how to go about this. I could write a new GoogleAuthController, that just acts like any other controller, and have an API that accepts the Google token, and returns the new token and other info in the same format that the username/password authentication API does it. But 2 things are nagging at me:
I have this awkward feeling like this is the noobie way of doing things, reinventing the wheel, and really there's a super-cool magical way of hooking things together that I should rather be using; and
In MyAuthorizationProvider.GrantResourceOwnerCredentials(), I've got access to an OAuthGrantResourceOwnerCredentialsContext object, which allows me to validate my new AuthenticationTicket. If I'm doing this inside a plain vanilla controller, I have no idea how I would mark that ticket as validated.
Any clues, please?
EDIT I've seen the Google auth flow as described here. I'm still confused by how best to manage the process from the API side. The client will be obtaining the authorization code, and then calling the API with that auth code. I get that then I've got to take that auth code and convert it to a token by calling the Google API. (Or maybe that should be the client's responsibility?) Either way, I then need to use that token to go back to the Google API and get the user's name, email and avatar image, then I need to match up that email with my own database to identify the user and build up their claims. Then I need to return a new token that the client can use to connect to me going forward.
Let me be more specific about my questions, before my question is closed as "too broad":
When the client has completed authentication with the Google API, it gets back a "code". That code still needs to be converted into a token. Whose responsibility should that be - the client or the API? (I'm leaning towards making it the client's responsibility, if just for the reason of distributing the workload better.)
Whether the client is passing through a code or a token, I need to be able to receive it in the API. Should I just use a plain vanilla Controller to receive it, with an endpoint returning an object of type AuthenticationProperties, or is there some special OWIN way of doing this?
If I'm using a plain vanilla Controller, how do I validate my token? In other words, how do I get access to the OWIN context so that I can mark the AuthenticationTicket as validated?
How do I write an automated test that simulates the client side of the process? AFAICT, the authentication wants to have a user physically click on the "Allow" button to grant my app access to their identity stuff, before it will generate the auth code. In an automated test, I would want to pass username/password etc. all from code. How do you do that?
So I found a solution of my own. It's only slightly kludgy, doesn't require referencing any Google OWIN libraries, and best of all, reuses the code from my username/password authentication.
Firstly, I get the app to call the same Authenticate endpoint as I do for username/password, only with dummy credentials, and add in a "GoogleToken" header, containing the token.
In my authentication code, I check for the GoogleToken header, and if it exists, follow that code path to validate on the Google servers, get an email address, and link to my own User table. Then the rest of the process for building claims and returning a new API token follows the original path.
start here : https://developers.google.com/identity/protocols/OAuth2#basicsteps
This explains how oAuth2 works. So you receive a Google token, now you call Google and request the user's details. you will receive their email which is enough to authenticate them. You could store the token as they are valid for a while and you can keep reusing it for whatever you need until it expires or it is invalidated.
Check this discussion on the same subject :
How can I verify a Google authentication API access token?
if you need more info on how OAuth2 works I can point you to one of my own articles : https://eidand.com/2015/03/28/authorization-system-with-owin-web-api-json-web-tokens/
There's a lot to take in, but it sounds like you need to understand how these things work together. Hope this helps.
Update:
I don't have full access to your setup, but I hope that the following code might help you with using Google as ID provider. Please add the following code to your startup.auth.cs file.
var googleAuthOptions = new GoogleOAuth2AuthenticationOptions
{
ClientId = "ef4ob24ttbgmt2o8eikgg.apps.googleusercontent.com",
ClientSecret = "DAK0qzDasdfasasdfsadwerhNjb-",
Scope = { "openid", "profile", "email" },
Provider = new GoogleOAuth2AuthenticationProvider
{
OnAuthenticated = async ctx =>
{
//You can get the claims like this and add them to authentication
var tokenClaim = new Claim("GoogleAccessToken", ctx.AccessToken);
var emailClaim = new Claim("email", ctx.Email);
var claimsIdentity = new ClaimsIdentity();
claimsIdentity.AddClaim(tokenClaim);
claimsIdentity.AddClaim(emailClaim);
HttpContext.Current
.GetOwinContext()
.Authentication
.SignIn(claimsIdentity);
await Task.CompletedTask;
}
},
AuthenticationType = "Google"
};
app.UseGoogleAuthentication(googleAuthOptions);
This allows the Google to act as ID Provider and the OnAuthenticated gets called when the authentication is successful. You can get the claims out of it and use them to signin. Please let me know if this worked, if not give me more details about your setup (what kind of framework, client setup and may be more details about your setup in startup file).
Thank you.
Please see this link for details on how we can use Google as ID Provider. I am sure you might have looked at this link, but in case you missed it. If none of these links work for you please include specific details on where you are deviating from what is mentioned in the links.
I assume you have a different requirement than what is specified in those links. Hence, I will try to answer your questions individually. Please let me know if you have any further questions.
When the client has completed authentication with the Google API, it gets back a "code". That code still needs to be converted into a token. Whose responsibility should that be - the client or the API? (I'm leaning towards making it the client's responsibility, if just for the reason of distributing the workload better.)
Exchanging the code for access token is definitely the responsibility of the API as the token exchange involves sending the ClientId and Client Secret along with the code. Client secret is supposed to be saved on the server side (API) but not on the client
Whether the client is passing through a code or a token, I need to be able to receive it in the API. Should I just use a plain vanilla Controller to receive it, with an endpoint returning an object of type AuthenticationProperties, or is there some special OWIN way of doing this?
This should work seamlessly if you are using the Google provider as mentioned in the above links. If not, the endpoint should be an anonymous endpoint accepting the code and making a request to Google (may be by using HttpClient) to get the access token along with the profile object for user related information.
If I'm using a plain vanilla Controller, how do I validate my token? In other words, how do I get access to the OWIN context so that I can mark the AuthenticationTicket as validated?
You have to implement OnGrantAuthorizationCode as part of your MyAuthorizationProvider class. This gives access to the context to set validated to true.
How do I write an automated test that simulates the client side of the process? AFAICT, the authentication wants to have a user physically click on the "Allow" button to grant my app access to their identity stuff, before it will generate the auth code. In an automated test, I would want to pass username/password etc. all from code. How do you do that?
This can be achieved partially, but, with that partial test you can be sure of good test coverage against your code. So, you have to mock the call to the Google API and assume that you have retrieved a valid response (hard code the response you received from a valid manual test). Now test your code on how it behaves with the valid response. Mock the Google API cal for an invalid response and do the same. This is how we are testing our API now. This assumes that Google API is working fine and tests my code for both valid/ in-valid responses.
Thank you,
Soma.
Having gone through something like this recently, I'll try to answer at least some of your questions:
The client should be getting a token from Google, which you can pass unaltered through to the API:
function onSignIn(googleUser) {
var profile = googleUser.getBasicProfile();
var idToken = googleUser.getAuthResponse().id_token;
}
A plain vanilla Controller should do it. The client can subsequently post an object in there, containing at least that token plus the client id (might be useful to know where the request comes from) and even the providerUserId;
Unfortunately I'm not that familiar with the Owin stack
Fully end-to-end integration testing might be tricky, although you might achieve something through tools like Selenium, or some mocking tool. The API however should be testable just by posting some fake data to that vanilla controller, although you might have to rely on some sort of mock implementation when you get to validating that token through Google (although you could also validate it manually on the server, provided you get the Google public api key).

Permanent access to youtube api

I'm using YoutubeAPI v3.0 to automatically upload videos to my own channel. However the script still needs manual intervention during Oath2.0 authorization. How to make it completely automatic?
1) Access the API using username and password
2) Or find a way to create permanent OAuth2.0 authentication
P/S: I use this script to upload
https://developers.google.com/youtube/v3/guides/uploading_a_video
The only thing I can think of is web scraping. Basically, programmatically open the web page and get its HTML. Then find the authorization code, and store it as a string. I don't know if your scripting language of choice can do it, but Python has Beautiful Soup (links at the bottom). The problem, of course, is accessing the contents of a page like that which is pretty clearly designed to be reached by a logged in user from a web browser. I've never done that, but there's some concept of a "login handshake" where you post the data to the server that's needed as you access the page. I've a few links at the bottom.
Anyway, to give you a better idea of what I mean in pseudo-code (for those who may be confused), it'd be something like:
webURL = 'http://any-url.net";
webPageObject = openPage(webURL);
pageHTML = webPageObject.getHTML();
theHTMLTag = searchForTagById(pageHTML, "<p id='oAuthMessage'>");
//And from there, figure out where the string containing the code is.
//Probably just by getting a substring from the end of the text in the <p>
//backward until you reach the length of the oAuth code.
You'll have to look at the page source to know which tags to look for specifically, but this can all just be done programmatically/automatically, as you wanted.
Links:
Login handshake - Scraping from a website that requires a login?
Beautiful Soup - http://www.crummy.com/software/BeautifulSoup/
google.gov/webScraping - https://www.google.com/search?ie=UTF-8&oe=utf-8&q=how+to+web+scrape+logged+in+page
You can use get Google OAUTH2 for devices in order to have fully automatic token renewal process.
So all you need now is:
Request a device code and confirmation code
Enter confirmation code to confirm your application have access for specific account
Generate new or renew existing ACCESS_TOKEN for your device code
Upload Video using your device code and valid ACCESS_TOKEN
Here is documentation for it.
And here is some examples.

HTTP Get Request

I've been trying to get Rails to play with the new Facebook Graph API. After I get the authorization "code", I need to send another request which returns the access token in JSON form.
It seems to work fine, however I want to fetch the access token JSON without redirecting the user. I'm attempting to use Net::HTTP.get, but I'm not sure how to use it to get a request body, or even if it's the right thing to use to begin with.
Can anyone give an example of performing an HTTP GET?
I've figured out how to do this, the problem was mainly with the fact that I needed an HTTPS connection.
Adapted from http://snippets.dzone.com/posts/show/788:
path = '/oauth/access_token?...'
http = Net::HTTP.new('graph.facebook.com', 443)
http.use_ssl = true
res = http.get(path, nil)
#access_token = res.body
Anyone specifically trying to use the Graph API, note that the value stored in #access_token is in the form of a params string, e.g. "access_token=xxxx&expires=1234".
I got around needing to parse this by just redirecting to another page and using that as the URL params, but there's probably a better way to do this.
SOA#1
However please note that it means that server have to be log onto facebook - while if browser is redirecting it is user who have to be log into server. Hence did your server set the permission?
You can pretend that you are the user. Bad Horrible idea (you have to store passwords in cleartext on you server).
You can use OAuth. Hence you should use OAuth gem instead of Net::HTTP. You will not avoid the redirection - it is part of authorisation process and user must say that he allows to access data (imagine what would be if anyone could access anyone data on facebook). Turorial on writing OAuth clients in rails.

Resources