I'm working through the documentation to gspread (http://gspread.readthedocs.io/en/latest/oauth2.html ), and have been able to get the introductory code block working:
import gspread
gc = gspread.authorize(credentials)
However when I run the following script :
import gspread
from oauth2client.service_account import ServiceAccountCredentials
scope = ['https://spreadsheets.google.com/feeds']
credentials = ServiceAccountCredentials.from_json_keyfile_name('My Project-******97.json', scope)
gc = gspread.authorize(credentials)
wks = gc.open("rules_test_1.csv").sheet1
I get:
gspread.exceptions.SpreadsheetNotFound
How can I get access to this sheet?
OK, you're making a common mistake. If you search for "service account [google-drive-sdk]" in StackOverflow, you'll find many people asking the same question.
Basically a Service Account IS NOT your Google account, which is why your file isn't found. The authors of gspread should have explained that in their example code. You have two options:-
Share your spreadsheet, or the folder it's in, with the Service Account user. This is the easiest approach since all you need to do is make the share from GDrive and then your existing code will work. The downside is it can be clunky to keep sharing files. My suggestion would be to make a dedicated folder called say "shared with my service account" and make all files children of that folder. Remember that files can have multiple parents, so you can still preserve your folder structure. See the answer to How do I search sub-folders and sub-sub-folders in Google Drive? for a refresher on how Drive folders work.
Replace the Service Account auth with Standard Account auth. This requires some minor changes to your code. The result will be that your app is talking directly to your Drive Account, without having a Service Account acting as a proxy. Thus it's more elegant provided the security implications are acceptable. You will be using a stored Refresh Token. Some information on how to do this is at How do I authorise an app (web or installed) without user intervention? (canonical ?)
Neither way is right or wrong. It depends on your specific requirements.
Related
I am trying to collect all active TIs via the Beta Graph API by following this. But it doesn't return anything. Here is what I use in Postman:
https://graph.microsoft.com/beta/security/tiIndicators
Response (200):
{
"#odata.context": "https://graph.microsoft.com/beta/$metadata#security/tiIndicators",
"value": []
}
A bit of context for the environment I work in.
The tenant has multiple Sentinel workspaces & resource groups.
The application I use has the correct permissions:
ThreatIndicators.Read.All
ThreatIndicators.ReadWrite.OwnedBy
ThreatSubmission.Read.All
ThreatSubmission.ReadWrite.All
It is my current belief that this might be due to the limitations of the Beta API. My reasoning is that accourding to this documentation you need the ThreatIndicators.ReadWrite.OwnedBy permission to access the API. This would suggest that currently you can only view TI's that the resource itself created.
If more info is needed just ask.
According to the documentation, ThreatIndicators.ReadWrite.OwnedBy permission allow you to manage threat indicators your app creates or owns.
If you want to read all the threat indicators for your organization then your app needs ThreatIndicators.Read.All permission.
Although this is not a solution to the question it is a workaround. By using the Log Analytics API you can get the TI via a KQL.
ThreatIntelligenceIndicator
| where ExpirationDateTime > now() and
NetworkIP matches regex #"^(?:(?:25[0-5]|(?:2[0-4]|1\d|[1-9]|)\d)\.?\b){4}$" and
ConfidenceScore > 25
| summarize by NetworkIP
This is probably better as you can also use a watchlist to exclude specific IP addresses with one request.
One thing I struggled with this was Authorization. You must give your Application permission to use the api.loganalytics.io API, and the application needs the Log Analytics Reader role in the Log Analytic workspace you want to use.
Context of what I'm trying to accomplish:
User shares a file with the bot
Other users interact with the bot via a dialog
The bot shares the original file to the other users
For example, we want to share a file to the bot that contains this week's cafeteria menu. Each time users would interact with the bot in a certain way, it would share the cafeteria menu with them so that they can consult it.
I've tried calling files.share method but bots can't perform this action (get invalid token type error).
As far as I can tell, there is no way to do this currently. I've tried link unfurling in the message body but that only works if the file itself was already shared to the user. If not, the link simply won't unfurl and clicking it will fail.
The bot can perform a files.upload call and re-upload the contents of the file to each user individually. This seems incredibly wasteful but appears to be the only way to work currently.
Is there something I'm missing?
The reason your bot can not use file.share is that this is an undocumented API method and you need a legacy token to use it. No other token (user token, bot token) will work, because it requires the post scope, which only exists for legacy token.
Approach A: Legacy Token
So one approach would be to use a legacy token with your bot, which you can create here for your current workspace. That should work nicely if your Slack app is only used on your "own" Slack workspace where you can create and use a legacy token.
Approach B: File Mention
Another approach is to use the mention feature in messages to share a file. This works by sending the private link (url_private property) of an already shared file in a message to a new channel. This will automatically re-share the file in that channel. I believe this only works with files that how been previously shares in a public channel and can therefore be re-shared. Be aware though that the file mention feature is currently being reworked, so this behavior might change.
Example:
https://slack.com/api/chat.postMessage?token=TOKEN&channel=CHANNEL&as_user=true&text=URL_PRIVATE
For more details see the Slack tutorial Storing, retrieving, and modifying file uploads.
Approach C: External File / image file
If you host your file externally or create a public URL for a file uploaded to Slack you can share it in every channel by just adding the URL to a message. Slack will automatically unfurl it and therefore share it to the user in any channel. This is different to Approach B, because its not a file mention and requires a public URL. You get the public URL of an uploaded file by calling files.sharedPublicURL.
If i'm not wrong, you can do like this :
you share a file with your bot
you retrieve the file shared ID, so his url_private property (cf https://api.slack.com/types/file#authentication)
you then donwload the file
you can then re-share it several times later (without re-uploading to each user)...
I am currently working on a solution that is accessing OneDrive in Office 365 using Microsoft Graph. I am using the adal4j library to handle authentication and have configured the app in portal.azure.com.
My question relates the call to get the children for a specified drive. I am using a query similar to the one shown below, as I want to get folders and files at the root level of a specified users drive:
https://graph.microsoft.com/v1.0/users/*user id*/drives/*drive id*/root/children
When I login to the Graph Explorer and execute the query, I get a json result showing the root folder contents for the drive and user specified. All works as expected.
When I call it from my java application, the JSON node value is empty ([]).
Initially my thought was, because the Graph Explorer uses a different app id in the portal it was possibly something to do with access rights. However, I successfully read user profiles in our O365 tenant, the drive id's for each user, and if I execute the following:
https://graph.microsoft.com/v1.0/users/*user id*/drives/*drive id*/root/search(q='')
It provides me a complete list of all of the folders, sub folders etc within the appropriate user's drive.
Therefore, making me think this is a bug with the Graph query I am attempting to use rather than an authorization issue, but, that wouldn't explain why it works in the Graph Explorer.
The same java method is used for all calls, and the url is passed in as a parameter.
Just to follow up, the azure portal app permissions has the capability of adding permissions for the graph api. This was, indeed the problem. It would appear that the search was ignoring the permission and successfully reading the data whereas the /children call was honouring the security model. This caused a lot of confusion, but is now resolved.
Thanks Marc for your help.
My situation is as follows:
Google Account A has some data in BigQuery.
Google Account B manages Account A's BigQuery data, and has also been given editor privileges for Account A's Cloud Platform project.
Account B has a Sheet in Google Drive that has some cool reference data in it. Account B logs into the BQ Web console, and creates a table in Account A's BQ project that is backed by this sheet.
All is well. Account B can query and join to this table successfully within Account A's BQ data from the web UI.
Problem:
Google Account A also has a service account that is an editor for Google Account A's Cloud Platform Project. This service account manages and queries the data in BQ using the python google-cloud API. When this service account attempts to query the reference table that is backed by Account B's GDrive Sheet, the job fails with this error:
Encountered an error while globbing file pattern. JobID: "testing_gdrivesheet_query_job1"
Near as I can tell this is actually an authentication issue. How can I give Account A's service account appropriate access to Account B's GDrive so it can access that reference table?
Bonus Points:
Is there any performance difference between a table backed by a GDrive Sheet vs a native BQ table?
While Orbit's answer helped me to find a solution for the issue, there are a few more things you need to consider. Therefore, I like to add my detailed solution to the problem. This solution is required if Orbit's basic solution does not work, in particular, if you use the G Suite and your policies do not allow sharing sheets/docs with accounts outside of your domain. In this case you cannot directly share a doc/sheet with the service account.
Before you start:
Create or select a service account in your project
Enable Domain-wide Delegation (DwD) in the account settings. If not present, this generates an OAuth client ID for the service account.
Make sure the delegated user#company.com has access to the sheet.
Add the required scopes to your service account's OAuth client (you may need to ask a G Suite admin to do this for you):
https://www.googleapis.com/auth/bigquery
https://www.googleapis.com/auth/drive
If the delegated user can access your drive-based table in the BigQuery UI, your service account should now also be able to access it on behalf of the delegated user.
Here is a full code snippet that worked for me:
#!/usr/bin/env python
import httplib2
from google.cloud import bigquery
from oauth2client.service_account import ServiceAccountCredentials
scopes = [
"https://www.googleapis.com/auth/drive",
"https://www.googleapis.com/auth/bigquery",
]
delegated_user = "user#example.com"
project = 'project-name'
table = 'dataset-name.table-name'
query = 'SELECT count(*) FROM [%s:%s]' % (project, table)
creds = ServiceAccountCredentials.from_json_keyfile_name('secret.json', scopes=scopes)
creds = creds.create_delegated(delegated_user)
http = creds.authorize(httplib2.Http())
client = bigquery.Client(http=http)
bq = client.run_sync_query(query)
bq.run()
print bq.fetch_data()
Note that I was not able to setup the delegation directly and needed to create an HTTP client using creds = creds.create_delegated(delegated_user) and http = creds.authorize(httplib2.Http()). The authorized HTTP client can then be used as HTTP client for the BigQuery client: client = bigquery.Client(http=http).
Also note that the service account does not need to have any predefined roles assigned in the project settings, i.e., you do not have to make it a bigquery user or even a project owner. I suppose it acquires access primarily via delegation.
You should be able to get this working with the following steps:
First share the sheet with the email/"service account id" associated with the service account.
Then you'll be able to access your sheet-backed table if you create a Client with the bigquery and drive scopes. (You might need to have domain-wide-delegation enabled on the service account).
scopes = ['https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/drive']
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'<path_to_json>', scopes=scopes)
# Instantiates a client
client = bigquery.Client(project = PROJECT, credentials = credentials)
bqQuery = client.run_sync_query(q)
bqQuery.run()
bqQuery.fetch_data()
For those of you trying to do this via Airflow or Google Cloud Composer, there are two main steps you'll need to do to accomplish this.
Grant view access to the spreadsheet to the project_name#developer.gserviceaccount.com. This should be the same service account you're using to access Google BigQuery. This can be done in the Sheets GUI or programmatically.
Add the following scope to your Google Cloud Connection in Airflow:
You will then be able to query external tables that reference Google Sheets.
Just need to add step from Evan Kaeding answer. You can find airflow connection in Airflow UI menu "Admin" -> "Connections" -> choose your connection. In my case I also need to add keyfile path or keyfile JSON of your service account in the airflow connection
based on this references https://cloud.google.com/composer/docs/how-to/managing/connections#creating_a_connection_to_another_project
I have been using gspread (authenticated via ClientLogin) for a last year. Now I would like to use OAuth2. I've followed tutorial from gspread site: http://gspread.readthedocs.org/en/latest/oauth2.html
The problem is that this method creates new "Email address" (in console.developers.google) which doesn't have an access to spreadsheets - all spreadsheets should be shared again. This is really difficult if you have 1000+ spreadsheets.
The question is: how to authenticated with OAuth2 my default gmail account (that I've been using to access via ClientLogin)?
Thank you!
EDIT:
I've followed this tutorial: http://www.indjango.com/access-google-sheets-in-python-using-gspread/
But I modified code from point 1.2: http://www.indjango.com/access-google-sheets-in-python-using-gspread/#comment-2026863410
Result - some spreadsheets are available, some not and I have no idea why (same entries in access list)...
It seems that code from EDIT works. Thus, this is working solution:
I've followed this tutorial: http://www.indjango.com/access-google-sheets-in-python-using-gspread/
But I modified code from point 1.2: http://www.indjango.com/access-google-sheets-in-python-using-gspread/#comment-2026863410
The only problem is that Google Sheets API returns only 500 results (thus, if using gspread when you have more spreadsheets that are not among results -> gspread raises SpreadsheetNotFound).