I need to generate an access_token with some specific scopes ['https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/cloud-platform'] from Python inside Google Colab.
By default, Google Colab provides a function inside colabtools to authenticate the user.
from google.colab import auth
auth.authenticate_user()
print('Authenticated')
But the credential file generated contains a list of fixed scopes, not including the ones I need.
import os
with open(os.environ['GOOGLE_APPLICATION_CREDENTIALS']) as f:
print(f.read())
...
"scopes": [
"https://www.googleapis.com/auth/drive",
"https://www.googleapis.com/auth/appengine.admin",
"https://www.googleapis.com/auth/accounts.reauth",
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/compute",
"https://www.googleapis.com/auth/userinfo.email"
],
...
I have tried to generate a valid Credentials token with the google-auth library enter code hereusing the google.auth.Credentials class and also the google.oauth2.Credentials class. I can generate valid instances of the classes, but when I check the token is None.
Is there a way to generate valid access_token for Google API with custom scopes from Python?
Thanks!
If you are using a VM with Google Colab, you can set the scopes for the service account on that vm with the following command (example):
gcloud compute instances set-service-account example-instance \
--service-account my-sa-123#my-project-123.iam.gserviceaccount.com \
--scopes compute-rw,storage-ro
Then, when authenticating from within Python, the generated credentials will contain the correct scopes.
Related
I have a task using the GCSToGoogleSheetsOperator in Airflow where Im trying to add data to a sheet.
I have added the service credential email to the sheet I want to edit with editor privileges, and received this error:
googleapiclient.errors.HttpError:
<HttpError 403 when requesting
https://sheets.googleapis.com/v4/spreadsheets/<SHEET_ID>/values/Sheet1?valueInputOption=RAW&includeValuesInResponse=false&responseValueRenderOption=FORMATTED_VALUE&responseDateTimeRenderOption=SERIAL_NUMBER&alt=json
returned "Request had insufficient authentication scopes.".
Details: "[{
'#type': 'type.googleapis.com/google.rpc.ErrorInfo',
'reason': 'ACCESS_TOKEN_SCOPE_INSUFFICIENT',
'domain': 'googleapis.com',
'metadata': {
'service': 'sheets.googleapis.com',
'method': 'google.apps.sheets.v4.SpreadsheetsService.UpdateValues'}
}]>
I cant update the sheet, but the GCS and BigQuery operators work fine.
My connection configuration looks like the following:
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fopt%2Fairflow%2Fcredentials%2Fgoogle_credential.json
I tried following the instructions to add the scope https://www.googleapis.com/auth/spreadsheets.
Which URL encoded looks like:
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fopt%2Fairflow%2Fcredentials%2Fgoogle_credential.json&extra__google_cloud_platform__scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets
Now, operators which previously worked error out like this:
google.api_core.exceptions.Forbidden: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/my-project/jobs?prettyPrint=false: Request had insufficient authentication scopes.
And the GCSToGoogleSheetsOperator operator still error out like this:
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/download/storage/v1/b/my-bucket/o/folder%2Fobject.csv?alt=media: Insufficient Permission: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
How can I set the permissions correctly to use both BigQuery, GCS and Sheets operators?
Adding a scope seems to ignore the IAM roles, so its either one or the other.
The service account had roles needed to access GCS and BigQuery, but by adding the scope https://www.googleapis.com/auth/spreadsheets, the service would ignore the privileges granted by the roles and look only at the ones specified by the scopes.
So, to recover it, you must add both the spreadsheet and cloud-platform scopes (or more strict scopes). cloud-platform will provide access to GCS and BigQuery and spreadsheets to Google Sheets API.
If you set your connection using environment variables, you have to URL encode the arguments, so to create a GOOGLE_CLOUD connection, you will have to do something like this, which is not encoded...
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=/abs/path_to_file/credential.json&extra__google_cloud_platform__scope=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/spreadsheets
To encode, which is the version you have to use, replace /, , and ::
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fabs%2Fpath_to_file%2Fcredentials%2Fgoshare-driver-c08e0904285b.json&extra__google_cloud_platform__scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets
enter image description here
The authorization URL as an output of the example python program from here: https://developers.google.com/youtube/v3/docs/channels/list?apix=true
leads me to a webpage. It says: 'This App is not Verified by Google'. Help needed.
Welcome, Mappy, you need to create credentials in Google Cloud Platform, you get a token as client there for pre-production tasks and limited consent screen too. When you'll ready, your app needs to be verified by Google (Auth Consent Screen and OAuth Scopes for Google).
# -*- coding: utf-8 -*-
# Sample Python code for youtube.channels.list
# See instructions for running these code samples locally:
# https://developers.google.com/explorer-help/guides/code_samples#python
import os
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
scopes = ["https://www.googleapis.com/auth/youtube.readonly"]
def main():
# Disable OAuthlib's HTTPS verification when running locally.
# *DO NOT* leave this option enabled in production.
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
api_service_name = "youtube"
api_version = "v3"
client_secrets_file = "YOUR_CLIENT_SECRET_FILE.json"
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
credentials = flow.run_console()
youtube = googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
request = youtube.channels().list(
)
response = request.execute()
print(response)
if __name__ == "__main__":
main()
A minimum test in terminal works to me.
curl \
'https://www.googleapis.com/youtube/v3/channels?part=contentDetails&forUsername=Me%20dicen%20Dai&key=[YOUR_API_KEY]' \
--header 'Accept: application/json' \
--compressed
In the Oauth Consent Screen configuration of google add the scope and it will show you an Orange triangle if verification is needed for your selected scope.
The cause of this error is because many of Google scopes need app verification before u can use it with external users.
Also, this url will tell you whether the requested scope might fall into the Sensitive or Restricted category
https://support.google.com/cloud/answer/9110914?hl=en
My app is hosted on Heroku, so I'm trying to figure out how to use the JSON Google Cloud provides (to authenticate) as an environment variable, but so far I can't get authenticated.
I've searched Google and Stack Overflow and the best leads I found were:
Google Vision API authentication on heroku
How to upload a json file with secret keys to Heroku
Both say they were able to get it to work, but they don't provide code that I've been able to get work. Can someone please help me? I know it's probably something stupid.
I'm currently just trying to test the service in my product model leveraging this sample code from Google. Mine looks like this:
def self.google_vision_labels
# Imports the Google Cloud client library
require "google/cloud/vision"
# Your Google Cloud Platform project ID
project_id = "foo"
# Instantiates a client
vision = Google::Cloud::Vision.new project: project_id
# The name of the image file to annotate
file_name = "http://images5.fanpop.com/image/photos/27800000/FOOTBALL-god-sport-27863176-2272-1704.jpg"
# Performs label detection on the image file
labels = vision.image(file_name).labels
puts "Labels:"
labels.each do |label|
puts label.description
end
end
I keep receiving this error,
RuntimeError: Could not load the default credentials. Browse to
https://developers.google.com/accounts/docs/application-default-credentials for more information
Based on what I've read, I tried placing the JSON contents in secrets.yml (I'm using the Figaro gem) and then referring to it in a Google.yml file based on the answer in this SO question.
In application.yml, I put (I overwrote some contents in this post for security):
GOOGLE_APPLICATION_CREDENTIALS: {
"type": "service_account",
"project_id": "my_project",
"private_key_id": "2662293c6fca2f0ba784dca1b900acf51c59ee73",
"private_key": "-----BEGIN PRIVATE KEY-----\n #keycontents \n-----END PRIVATE KEY-----\n",
"client_email": "foo-labels#foo.iam.gserviceaccount.com",
"client_id": "100",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url":
"https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url":
"https://www.googleapis.com/robot/v1/metadata/x509/get-product-labels%40foo.iam.gserviceaccount.com"
}
and in config/google.yml, I put:
GOOGLE_APPLICATION_CREDENTIALS = ENV["GOOGLE_APPLICATION_CREDENTIALS"]
also, tried:
GOOGLE_APPLICATION_CREDENTIALS: ENV["GOOGLE_APPLICATION_CREDENTIALS"]
I have also tried changing these variable names in both files instead of GOOGLE_APPLICATION_CREDENTIALS with GOOGLE_CLOUD_KEYFILE_JSON and VISION_KEYFILE_JSON based on this Google page.
Can someone please, please help me understand what I'm doing wrong in referencing/creating the environmental variable with the JSON credentials? Thank you!
It's really annoying that Google decides to buck defacto credential standards by storing secrets via a file instead of a series of environment variables.
That said, my solution to this problem is to create a single .env variable GOOGLE_API_CREDS.
I paste the raw JSON blob into the .env then remove all newlines. Then in the application code I use JSON.parse(ENV.fetch('GOOGLE_API_CREDS') to convert the JSON blob into a real hash:
The .env file:
GOOGLE_API_CREDS={"type": "service_account","project_id": "your_app_name", ... }
Then in the application code (Google OCR client as an example):
Google::Cloud::Vision::ImageAnnotator.new(credentials: JSON.parse(ENV.fetch('GOOGLE_API_CREDS'))
Cheers
Building on Dylan's answer, I found that I needed to use an extra line to configure the credentials as follows:
Google::Cloud::Language.configure {|gcl| gcl.credentials = JSON.parse(ENV['GOOGLE_APP_CREDS'])}
because the .new(credentials: ...) method was not working for Google::Cloud::Language
had to look in the (sparse) ruby reference section of Google Cloud Language.
And yeah... storing secrets in a file is quite annoying, indeed.
I had the same problem with Google Cloud Speech, using the "Getting Started" doc from Google.
The above answers helped a great deal, coupled with updating my Google Speech Gem to V1 (https://googleapis.dev/ruby/google-cloud-speech-v1/latest/Google/Cloud/Speech/V1/Speech/Client.html)
I simply use a StringIO object so that Psych thinks that it's an actual file that I read:
google:
service: GCS
project: ''
bucket: ''
credentials: <%= StringIO.new(ENV['GOOGLE_CREDENTIALS']) %>
I am working on google Adwords API to upgrade our code for migration from v201302 to v201309.
Any one can suggest me, what code we should use in place of following code ( as ClientLoginTokens now deprecated).
String clientLoginToken = new ClientLoginTokens.Builder()
.forApi(ClientLoginTokens.Api.ADWORDS)
.withEmailAndPassword(configurations.get("email"), configurations.get("password"))
.build()
.requestToken();
Here are the steps that I took to get OAuth2 working. YMMV of course...
Step 1 - Register Application with Google Console API
Log into Google using your email and password above
Head to the Google API Console. You probably get redirected to Google Cloud Console
Under 'APIs & Auth' click on 'Consent screen'. Fill in at least 'Product Name' and 'Email'.
Under 'APIs & Auth' click on 'Registered apps'.
Click 'Register App'. Fill in details ensuring that you select 'Native' as platform.
Under 'OAuth 2.0 Client ID' make a note of the CLIENT ID and CLIENT SECRET values.
Step 2 - Generate Refresh Token
Next step is to generate a refresh token. This is a generate-once-use-multiple-times token that allows your application to obtain new access tokens:
Download GetRefreshToken.java.
Create an aps.properties file to be referenced by the GoogleClientSecretsBuilder()
.forApi(Api.ADWORDS) call. This ads.properties file should contain two lines:
api.adwords.clientId=client-id-from-step1.6
api.adwords.clientSecret=client-secret-from-step1.6
Using web browser log into the Google AdWords MCC.
Run GetRefreshToken.java and follow instructions i.e. copy browser URL into browser, enter code returned into console etc. etc.
You should now have a refreshToken. Copy this refresh token into your ads.properties files like this:
api.adwords.refreshToken=your-refresh-token
PS GetRefreshToken.java has a couple of dependencies. If you are using Maven then here they are (adjust versions accordingly!):
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-oauth2</artifactId>
<version>v2-rev50-1.17.0-rc</version>
</dependency>
<dependency>
<groupId>com.google.api-ads</groupId>
<artifactId>adwords-axis</artifactId>
<version>1.20.0</version>
</dependency>
Step 3 - Generate Credential
With your refreshToken, clientId & clientSecret in your ads.properties you can now generate a Credential like this:
Credential oAuth2Credential = new OfflineCredentials.Builder()
.forApi(Api.ADWORDS)
.fromFile()
.build()
.generateCredential();
Step 4 - Get AdWords Session
The final step (hats off to you if you have got this far!) is to create an AdWords Session using the oAuth2Credential instance of Credential that you created in Step 3. You can do this by adding two more things into your ads.properties file:
api.adwords.developerToken=developer-token-from-mcc
api.adwords.clientCustomerId=client-id-of-adwords-account-that-you-want-to-access
Then get an AdWords session up using like this:
AdWordsSession awSession =
new AdWordsSession.Builder()
.fromFile()
.withOAuth2Credential(oAuth2Credential)
.build();
Step 5 - Grab a coffee and reflect on how easy it is to access the Google AdWords API using OAuth2
This step is entirely optional.
You can not transform your old process identical as before. There are some examples in the Migration Guide from Google. See the Authentication/OAuth 2.0 section:
If you are coming from using ClientLogin, we've added a few features to make it extremely easy to switch over.
Once you've generated a refresh token using the GetRefreshToken.java example of your examples download, and you've copied it into your ads.properties file, you'll be able to create a refreshable token with the OfflineCredentials utility.
Credential oAuth2Credential = new OfflineCredentials.Builder()
.forApi(Api.DFP)
.fromFile()
.build()
.generateCredential();
Once authorized, you can set the Credential object into the builder or session:
DfpSession session = new DfpSession.Builder()
.fromFile()
.withOAuth2Credential(oAuth2Credential)
.build();
OAuth2 will now be used when making API calls.
You might change Api.DFP to Api.ADWORDS. OAuth 2.0 at Google is fully covered in the Using OAuth 2.0 for Login article.
Trying to use Oauth 2.0 server to server authentication (using a service account) to upload a file to google drive. Have used their sample code as a reference, the resulting script is something like this:
import httplib2
import pprint
import sys
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
from apiclient.http import MediaFileUpload
def main(argv):
# Load the key in PKCS 12 format that you downloaded from the Google API
# Console when you created your Service account.
f = open('key.p12', 'rb')
key = f.read()
f.close()
# Check https://developers.google.com/drive/scopes for all available scopes
OAUTH_SCOPE = 'https://www.googleapis.com/auth/drive'
# Path to the file to upload
FILENAME = 'testfile.txt'
# Create an httplib2.Http object to handle our HTTP requests and authorize it
# with the Credentials. Note that the first parameter, service_account_name,
# is the Email address created for the Service account. It must be the email
# address associated with the key that was created.
credentials = SignedJwtAssertionCredentials(
'xxxxx-xxxxxxx#developer.gserviceaccount.com',
key,
OAUTH_SCOPE)
http = httplib2.Http()
http = credentials.authorize(http)
drive_service = build('drive', 'v2', http=http)
# Insert a file
media_body = MediaFileUpload(FILENAME, mimetype='text/plain', resumable=True)
body = {
'title': 'My document',
'description': 'A test document',
'mimeType': 'text/plain'
}
fil = drive_service.files().insert(body=body, media_body=media_body).execute()
pprint.pprint(fil)
if __name__ == '__main__':
main(sys.argv)
The script seems to run ok (no errors, pprint shows output that seems to be fine). However the google drive page for the account does not show the uploaded file. When trying to access one of the links from the pprint output to see the file I get a "You need permission" message from Google Drive, which is weird, as I am logged to the account in which I created the service account.
The file is owned by the service account, not your Google account. Service accounts have their own 5gb of space for Google Drive.
You'll need to either share the file with your user account or have the service account impersonate your user account (assuming you're in a Google Apps domain) so that the file is created and owned by your user account.