I want to add Google Service Account credentials using a groovy file initializer on init.groovy.d. So far I could define UserName-Password credentials and String credentials. However, the Google API key requires a special credential type: GoogleRobotPrivateKeyCredentials. I could not manage to add that type of credential.
How can I define Google Service Account credentials of type GoogleRobotPrivateKeyCredentials through a groovy script?
It seems there is an answer on that link: https://github.com/jenkinsci/google-oauth-plugin/issues/5
I have changed that slightly, as that answer did not work on the latest Jenkins installation:
import com.cloudbees.plugins.credentials.Credentials
import com.cloudbees.plugins.credentials.CredentialsStore
import com.cloudbees.plugins.credentials.SystemCredentialsProvider
import com.cloudbees.plugins.credentials.domains.Domain
import com.google.jenkins.plugins.credentials.oauth.GoogleRobotPrivateKeyCredentials
import com.google.jenkins.plugins.credentials.oauth.JsonServiceAccountConfig
import hudson.model.FileParameterValue.FileItemImpl
import org.apache.commons.fileupload.FileItem
def jsonFile = new File("/myfilepath/key.json")
def fileItem = new FileItemImpl(jsonFile)
def config = new JsonServiceAccountConfig()
config.setJsonKeyFileUpload(fileItem)
Credentials credentials = new GoogleRobotPrivateKeyCredentials("gpc", config, null)
SystemCredentialsProvider instance = SystemCredentialsProvider.getInstance()
CredentialsStore store = instance.getStore()
store.addCredentials(Domain.global(), credentials)
Related
import pandas as pd
from google.cloud import bigquery
import google.auth
# from google.cloud import bigquery
# Create credentials with Drive & BigQuery API scopes
# Both APIs must be enabled for your project before running this code
credentials, project = google.auth.default(scopes=[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/spreadsheets',
'https://www.googleapis.com/auth/bigquery',
])
client = bigquery.Client(credentials=credentials, project=project)
# Configure the external data source and query job
external_config = bigquery.ExternalConfig('GOOGLE_SHEETS')
# Use a shareable link or grant viewing access to the email address you
# used to authenticate with BigQuery (this example Sheet is public)
sheet_url = (
'https://docs.google.com/spreadsheets'
'/d/1uknEkew2C3nh1JQgrNKjj3Lc45hvYI2EjVCcFRligl4/edit?usp=sharing')
external_config.source_uris = [sheet_url]
external_config.schema = [
bigquery.SchemaField('name', 'STRING'),
bigquery.SchemaField('post_abbr', 'STRING')
]
external_config.options.skip_leading_rows = 1 # optionally skip header row
table_id = 'BambooHRActiveRoster'
job_config = bigquery.QueryJobConfig()
job_config.table_definitions = {table_id: external_config}
# Get Top 10
sql = 'SELECT * FROM workforce.BambooHRActiveRoster LIMIT 10'
query_job = client.query(sql, job_config=job_config) # API request
top10 = list(query_job) # Waits for query to finish
print('There are {} states with names starting with W.'.format(
len(top10)))
The error I get is:
BadRequest: 400 Error while reading table: workforce.BambooHRActiveRoster, error message: Failed to read the spreadsheet. Errors: No OAuth token with Google Drive scope was found.
I can pull data in from a BigQuery table created from CSV upload, but when I have a BigQuery table created from a linked Google Sheet, I continue to receive this error.
I have tried to replicate the sample in Google's documentation (Creating and querying a temporary table):
https://cloud.google.com/bigquery/external-data-drive
You are authenticating as yourself, which is generally fine for BQ if you have the correct permissions. Using tables linked to Google Sheets often requires a service account. Create one (or have your BI/IT team create one), and then you will have to share the underlying Google Sheet with the service account. Finally, you will need to modify your python script to use the service account credentials and not your own.
The quick way around this is to use the BQ interface, select * from the Sheets-linked table, and save the results to a new table, and query that new table directly in your python script. This works well if this is a one-time upload/analysis. If the data in the sheets will be changing consistently and you will need to routinely query the data, this is not a long-term solution.
I solved problem by adding scope object to client.
from google.cloud import bigquery
import google.auth
credentials, project = google.auth.default(scopes=[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/bigquery',
])
CLIENT = bigquery.Client(project='project', credentials=credentials)
https://cloud.google.com/bigquery/external-data-drive
import pandas as pd
from google.oauth2 import service_account
from google.cloud import bigquery
#from oauth2client.service_account import ServiceAccountCredentials
SCOPES = ['https://www.googleapis.com/auth/drive','https://www.googleapis.com/auth/bigquery']
SERVICE_ACCOUNT_FILE = 'mykey.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_credentials = credentials.with_subject('myserviceaccountt#domain.iam.gserviceaccount.com')
client = bigquery.Client(credentials=delegated_credentials, project=project)
sql = 'SELECT * FROM `myModel`'
DF = client.query(sql).to_dataframe()
You can try to update your default credentials through the console:
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/cloud-platform
To create a new user in Jenkins, admin needs to provide a username, emialID and password. Being an admin, is there a way to add a large number of users to Jenkins at a time by providing their username as their mail id, display name as their name and a common password*?
*Assuming that password will be reset at the time of each user logging in
I am using the below groovy script to add a user to Jenkins and provide only build permission.
import hudson.model.*
import hudson.security.*
import hudson.tasks.Mailer
def userId = args[0]
def password = args[1]
def email = args[2]
def fullName= args[3]
def instance = jenkins.model.Jenkins.instance
def existingUser = instance.securityRealm.allUsers.find {it.id == userId}
if (existingUser == null) {
def user = instance.securityRealm.createAccount(userId, password)
user.addProperty(new Mailer.UserProperty(email));
user.setFullName(fullName)
def strategy = (GlobalMatrixAuthorizationStrategy)
instance.getAuthorizationStrategy()
strategy.add(hudson.model.Item.BUILD,userId)
instance.setAuthorizationStrategy(strategy)
instance.save()
}
The script is invoked using jenkins-cli.
It is easier to connect Jenkins to LDAP. See this plugin here
looks like the Jenkins cli doesn't support add users , but check this one - using groovy script you can do it.
Creating user in Jenkins via API
if you want give specific permissions per job , maybe you can use the CLI get-job & update-job commands.
Or you can try check this one - Jenkins Add permissions to jobs using groovy it discuses almost the same ...
I am trying to authenticate my credentials to access the GMail API. Previously I did this using the run() method from OAuth2, and the code credentials = tools.run(flow, STORAGE, http=http) but this is now a deprecated method. I am now using the run_flow() method to authenticate my credentials.
import httplib2
import argparse
from apiclient import errors
from apiclient.discovery import build
from oauth2client import tools
from oauth2client.file import Storage
from oauth2client.client import flow_from_clientsecrets
CLIENT_SECRET_FILE = 'your_client_secret.json'
OAUTH_SCOPE = 'https://www.googleapis.com/auth/gmail.modify'
STORAGE = Storage('gmail.storage')
flow = flow_from_clientsecrets(CLIENT_SECRET_FILE, scope=OAUTH_SCOPE)
http = httplib2.Http()
credentials = STORAGE.get()there are credentials, no reauth is needed
#parser = argparse.ArgumentParser(parents=[tools.argparser])
#flags = parser.parse_args() #Put your arguments in the parenthesis
if credentials is None or credentials.access_token_expired:
credentials = run(flow, STORAGE, http=http)
#credentials = tools.run_flow(flow, STORAGE, flags, http=http)
http = credentials.authorize(http)
gmail_service = build('gmail', 'v1', http=http)
The commented lines are the code that uses run_flow() and not run().
The commented out code gives me the error: run.py: error: unrecognized arguments: AdminTests, AdminTests is not an argument I give to Python.
And when I change the arguments parsed to flags = parser.parse_args(['--noauth_local_webserver']) I get no error, but nothing happens.
Which flag should I use to simulate the run() as closesly as possible and how should I parse it?
Edit: When using the run() method to authenticate my credentials the URL accessed is:
http://localhost:8080/?code=4/myuniqueID (missing my unique ID in the example)
what you need to do for this is pass an empty list of args to the argparser like this
flags = tools.argparser.parse_args(args=[])
credentials = tools.run_flow(flow, storage, flags)
After comparing your code to the source code of OAuth's run and run_flow, it turns out that there is a significant difference between whether you include the http argument.
So,
tools.run(flow, STORAGE, http=http)
can be simulated with,
tools.run_flow(flow, STORAGE, flags, http=http)
but you have,
tools.run_flow(flow, STORAGE, flags)
Trying to use Oauth 2.0 server to server authentication (using a service account) to upload a file to google drive. Have used their sample code as a reference, the resulting script is something like this:
import httplib2
import pprint
import sys
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
from apiclient.http import MediaFileUpload
def main(argv):
# Load the key in PKCS 12 format that you downloaded from the Google API
# Console when you created your Service account.
f = open('key.p12', 'rb')
key = f.read()
f.close()
# Check https://developers.google.com/drive/scopes for all available scopes
OAUTH_SCOPE = 'https://www.googleapis.com/auth/drive'
# Path to the file to upload
FILENAME = 'testfile.txt'
# Create an httplib2.Http object to handle our HTTP requests and authorize it
# with the Credentials. Note that the first parameter, service_account_name,
# is the Email address created for the Service account. It must be the email
# address associated with the key that was created.
credentials = SignedJwtAssertionCredentials(
'xxxxx-xxxxxxx#developer.gserviceaccount.com',
key,
OAUTH_SCOPE)
http = httplib2.Http()
http = credentials.authorize(http)
drive_service = build('drive', 'v2', http=http)
# Insert a file
media_body = MediaFileUpload(FILENAME, mimetype='text/plain', resumable=True)
body = {
'title': 'My document',
'description': 'A test document',
'mimeType': 'text/plain'
}
fil = drive_service.files().insert(body=body, media_body=media_body).execute()
pprint.pprint(fil)
if __name__ == '__main__':
main(sys.argv)
The script seems to run ok (no errors, pprint shows output that seems to be fine). However the google drive page for the account does not show the uploaded file. When trying to access one of the links from the pprint output to see the file I get a "You need permission" message from Google Drive, which is weird, as I am logged to the account in which I created the service account.
The file is owned by the service account, not your Google account. Service accounts have their own 5gb of space for Google Drive.
You'll need to either share the file with your user account or have the service account impersonate your user account (assuming you're in a Google Apps domain) so that the file is created and owned by your user account.
I'm trying to run a groovy script that calls the Linkedin API. The question is, how do I authenticate using the grails oauth plugin from within the groovy script ?
This is my config :
oauth {
linkedin {
requestTokenUrl="https://api.linkedin.com/uas/oauth/requestToken"
accessTokenUrl="https://api.linkedin.com/uas/oauth/accessToken"
authUrl="https://www.linkedin.com/uas/oauth/authenticate"
consumer.key="xxxxxxxxx"
consumer.secret="xxxxxxxxxx"
}
}
This is my script, which I start with "grails run-script scriptname.groovy" :
import org.grails.plugins.oauth.OauthService
def oauthService = new OauthService()
oauthService.reset()
def URL = "http://api.linkedin.com/v1/people-search?country-code=us&postal-code=98102&distance=100&start=0&count=5"
def recs_response = oauthService.accessResource( URL, "linkedin", [key:"xxxxxxx", secret:"xxxxxxxxx"], 'GET')
println "it worked"
If I print the response I get "The token used in the OAuth request is not valid"
Thanks.
Take a look at http://code.google.com/p/oauth-signpost/wiki/TwitterAndSignpost
I downloaded the OAuthTwitterExample and put the commons-codec-1.3.jar and signpost-core-1.1-SNAPSHOT.jar that were included in the zip in my working directory
for flickr OAuth I used the following in oauth.groovy
import oauth.signpost.OAuth;
import oauth.signpost.OAuthConsumer;
import oauth.signpost.OAuthProvider;
import oauth.signpost.basic.DefaultOAuthConsumer;
import oauth.signpost.basic.DefaultOAuthProvider;
import oauth.signpost.signature.SignatureMethod;
def consumer = new DefaultOAuthConsumer('<API KEY>','<Signature>',
SignatureMethod.HMAC_SHA1)
def provider = new
DefaultOAuthProvider(consumer,"http://www.flickr.com/services/oauth/request_token",
"http://www.flickr.com/services/oauth/access_token",
"http://www.flickr.com/services/oauth/authorize");
String url =provider.retrieveRequestToken( OAuth.OUT_OF_BAND);
println "navigate to the following URL"
println url
enter this on commandline in the working directory
groovy -cp commons-codec-1.3.jar:signpost-core-1.1-SNAPSHOT.jar oauth.groovy
hope this helps