I have a task using the GCSToGoogleSheetsOperator in Airflow where Im trying to add data to a sheet.
I have added the service credential email to the sheet I want to edit with editor privileges, and received this error:
googleapiclient.errors.HttpError:
<HttpError 403 when requesting
https://sheets.googleapis.com/v4/spreadsheets/<SHEET_ID>/values/Sheet1?valueInputOption=RAW&includeValuesInResponse=false&responseValueRenderOption=FORMATTED_VALUE&responseDateTimeRenderOption=SERIAL_NUMBER&alt=json
returned "Request had insufficient authentication scopes.".
Details: "[{
'#type': 'type.googleapis.com/google.rpc.ErrorInfo',
'reason': 'ACCESS_TOKEN_SCOPE_INSUFFICIENT',
'domain': 'googleapis.com',
'metadata': {
'service': 'sheets.googleapis.com',
'method': 'google.apps.sheets.v4.SpreadsheetsService.UpdateValues'}
}]>
I cant update the sheet, but the GCS and BigQuery operators work fine.
My connection configuration looks like the following:
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fopt%2Fairflow%2Fcredentials%2Fgoogle_credential.json
I tried following the instructions to add the scope https://www.googleapis.com/auth/spreadsheets.
Which URL encoded looks like:
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fopt%2Fairflow%2Fcredentials%2Fgoogle_credential.json&extra__google_cloud_platform__scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets
Now, operators which previously worked error out like this:
google.api_core.exceptions.Forbidden: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/my-project/jobs?prettyPrint=false: Request had insufficient authentication scopes.
And the GCSToGoogleSheetsOperator operator still error out like this:
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/download/storage/v1/b/my-bucket/o/folder%2Fobject.csv?alt=media: Insufficient Permission: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
How can I set the permissions correctly to use both BigQuery, GCS and Sheets operators?
Adding a scope seems to ignore the IAM roles, so its either one or the other.
The service account had roles needed to access GCS and BigQuery, but by adding the scope https://www.googleapis.com/auth/spreadsheets, the service would ignore the privileges granted by the roles and look only at the ones specified by the scopes.
So, to recover it, you must add both the spreadsheet and cloud-platform scopes (or more strict scopes). cloud-platform will provide access to GCS and BigQuery and spreadsheets to Google Sheets API.
If you set your connection using environment variables, you have to URL encode the arguments, so to create a GOOGLE_CLOUD connection, you will have to do something like this, which is not encoded...
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=/abs/path_to_file/credential.json&extra__google_cloud_platform__scope=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/spreadsheets
To encode, which is the version you have to use, replace /, , and ::
AIRFLOW_CONN_GOOGLE_CLOUD=google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fabs%2Fpath_to_file%2Fcredentials%2Fgoshare-driver-c08e0904285b.json&extra__google_cloud_platform__scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets
Related
I am writing a Slack integration that can boot certain users out of public channels when certain conditions are met. I have added several OAuth scopes to the bot token, including the following:
channels:history
channels:manage
channels:read
chat:write
chat:write.public
groups:write
im:write
mpim:write
users:read
I am writing my bot in Python using the slack-bolt library and asyncio. However when I try to invoke this code:
await app.client.conversations_kick(channel=channel_id, user=user_id)
I get the following error:
slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/conversations.kick)
The server responded with: {'ok': False, 'error': 'channel_not_found'}
I know for a fact that both the channel_id and user_id arguments I'm passing in are valid. The channel ID I'm using is the string C01PAE3DB0A. I know it is valid because I can use the very same value for channel_id in the following API call:
response = await app.client.conversations_info(channel=channel_id)
And when I call conversations_info like that I get all of the information about my channel. (The same is true for calling users_info with the user_id - it returns successfully.) So why is that when I pass my valid channel_id parameter to conversations_kick I consistently receive this channel_not_found error? What am I missing?
So I got in touch directly with Slack support about this and they confirmed that there is a bug on their end. Specifically, the bug is that I should have received a restricted_action error response instead of a channel_not_found response. Apparently this is a known issue that is on their backlog.
The reason the API call would (try to) return this restricted_action error is simply because there is a workspace setting that, by default, prevents non-admins from kicking people out of public channels. Furthermore, this setting can only be changed by the workspace owner - one tier above admins.
But assuming you are the owner of the Slack workspace, you simply have to log into the Settings & Permissions page, which should look something like this:
And then you have to change the setting labeled "People who can remove members from public channels" from "Workspace admins and owners only (default)" to "Everyone, except guests."
Once I made that change, my API calls started succeeding.
When I select the properties allowExternalSenders or autoSubscribeNewMembers for a group synced from an on-premise Active Directory, I get the following error via Graph API SDK:
403 Forbidden
Graph service exception Error code: AppOnlyAccessNotEnabledForTarget
Error message: App Only access is not allowed for target resource: 'a1d0c98e-fb7e-40c4-a7b5-ccf6d96cba57#34f0717f-f1a0-4894-b310-d379993504b9'.
The Graph Explorer displays the following 404 error, even though it is not a mailbox:
{
"error": {
"code": "MailboxNotEnabledForRESTAPI",
"message": "REST API is not yet supported for this mailbox.",
"innerError": {
"request-id": "5beae8eb-4a59-404f-bd42-0c6aa2a25abd",
"date": "2020-03-31T13:02:11"
}
}
}
An example request looks like this:
https://graph.microsoft.com/v1.0/groups/{id}?$select=id,deletedDateTime,assignedLicenses,description,displayName,groupTypes,licenseProcessingState,mail,mailEnabled,mailNickname,onPremisesLastSyncDateTime,onPremisesSecurityIdentifier,onPremisesSyncEnabled,onPremisesDomainName,onPremisesSamAccountName,preferredDataLocation,securityEnabled,securityIdentifier,visibility,resourceProvisioningOptions,allowExternalSenders,autoSubscribeNewMembers
Which property should I use to decide if I should try to select these properties or not? I want to avoid missing the properties for groups which actually do expose these properties, so I do not simply want to guess.
Slightly different take on this answer, Microsoft even thought only providing one API for groups, there are effectively two different objects returned, SecurityEnabled = True or False. If Security = True then you can not ask for the follow attributes
(allowExternalSenders,autoSubscribeNewMembers,hideFromAddressLists,hideFromOutlookClients)
Is this ok, NO its not ok, Microsoft needs to return null instead of providing THE WRONG ERROR MESSAGE and sending people like us into multi hour head scratchers.
In my code, I now check if securityEnabled is true or false and have a different set of attributes depending.
The mailbox is on a dedicated (on premise) Microsoft Exchange Server or is not a valid (configured) Office 365 mailbox. Contact your Exchange or Office admin to apply any of these recommended solutions.
To filter Office 365 groups you can do this:
https://graph.microsoft.com/v1.0/groups?$filter=groupTypes/any(c:c eq 'Unified')
In an answer to this 2014 post
Unable to retrieve members of a google group, getting Invalid Input
you read: "There is no API to manage consumer googlegroups.com groups programatically".
Is this still the situation in 2018?
I tried to follow the suggestion in answer 3 of the post How to get the list of members in a Google group in Google app script (Admin SDK)? but I get the following error message:
ReferenceError: "AdminDirectory" is not defined. (line 9, file "Code")
where line 9 (and following) is (are):
page = AdminDirectory.Members.list(groupKey,
{
domainName: 'googlegroups.#com',
maxResults: 500,
pageToken: pageToken,
});
Searching to understand the error I found the reference page for Members: list. Using the "Try this API" form in that page I get the error reported in the first post I mentioned.
If it is NOT true that "There is no API to manage consumer googlegroups.com groups programatically", is there a guide to copy the list of the members of a group I own in a google-sheet sheet? (I mean to copy via a function, non by hand exporting and reimporting the CSV)
Many thanks, Roberto Scotti
It's 2021 and I still can't find any evidence there's an API for #googlegroups.com groups, sadly.
Hi I am trying to save data in timeseries db but I am facing an error at timeseries ingest when I enter wss://gateway-predix-data-services.run.aws-usw02-pr.ice.predix.io/v1/stream/messages in websocket uri it shows an error like this when I click on open socket
The request failed with status code: 500
{
"error": "Error: unexpected server response (401)",
"url": "wss://gateway-predix-data-services.run.aws-usw02-pr.ice.predix.io/v1/stream/messages"
}
and see this screenshot once.
Thanks®ards
Eswar G
From the error log, it looks you got authorization error. Check if you are using correct predix-zone-id of the time-series instance in the request header or you have configured correct UAA instance in predix tool kit.
Have you updated your UAA client with timeseries authorities?
To inserting time series data with particular user or client need permission. For getting permission, you need to add authorities into your UAA client.
Following are the authorities for timeseries.
Experiencing a similar problem. I'm posting only one of several received error messages. I've cycled through combos of the following for uri and zone id:
uri:
wss://gateway-predix-data-services.run.aws-usw02-pr.ice.predix.io/v1/stream/messages
https://time-series-store-predix.run.aws-usw02-pr.ice.predix.io/v1/datapoints
zone-id:
44853a54-3ecd-4589-85da-b54f2fda7cce
and the zone id shown in the photo, with .ingest at the end.
so, 401 unauthorized.
Did you happen to re-create the timeseries or uaa? Perhaps they are not associated. Also, I'm assuming you did "login as client" in the top menu.
Might be a good idea to create a new timeseries tied to that uaa. Then use the new zoneId.
Did you add predix..ingest and predix..user to the scope and authorities section in uaa client in your predix.io account. Please check the client id to add permission for in websocketriver.config file.
I've read the whole Box.com developers api guide and spent hours on the web researching this particular question but I can't seem to find a definitive answer and I don't want to start creating a solution if I'm going down the wrong path. We have a production environment where as once we are finished working with files our production software system zips them up and saves them into a local server directory for archival purposes. This local path cannot be changed. My question is how can I programmatically upload these files to our Box.com account so we can archive these on the cloud? Everything I've read regarding this involves using OAuth2 to gain access to our account which I understand but it also requires the user to login. Since this is an internal process that is NOT exposed to outside users I want to be able to automate this otherwise it would not be feasable for us. I have no issues creating the programs to trigger everytime a new files gets saved all I need is to streamline the Box.com access.
I just went through the exact same set of questions and found out that currently you CANNOT bypass the OAuth process. However, their refresh token is now valid for 60 days which should make any custom setup a bit more sturdy. I still think, though, that having to use OAuth for an Enterprise setup is a very brittle implementation -- for the exact reason you stated: it's not feasible for some middleware application to have to rely on an OAuth authentication process.
My Solution:
Here's what I came up with. The following are the same steps as outlined in various box API docs and videos:
use this URL https://www.box.com/api/oauth2/authorize?response_type=code&client_id=[YOUR_CLIENT_ID]&state=[box-generated_state_security_token]
(go to https://developers.box.com/oauth/ to find the original one)
paste that URL into the browser and GO
authenticate and grant access
grab the resulting URL: http://0.0.0.0/?state=[box-generated_state_security_token]&code=[SOME_CODE]
and note the "code=" value.
open POSTMAN or Fiddler (or some other HTTP sniffer) and enter the following:
URL: https://www.box.com/api/oauth2/token
create URL encoded post data:
grant_type=authorization_code
client_id=[YOUR CLIENT ID]
client_secret=[YOUR CLIENT SECRET]
code= < enter the code from step 4 >
send the request and retrieve the resulting JSON data:
{
"access_token": "[YOUR SHINY NEW ACCESS TOKEN]",
"expires_in": 4255,
"restricted_to": [],
"refresh_token": "[YOUR HELPFUL REFRESH TOKEN]",
"token_type": "bearer"
}
In my application I save both auth token and refresh token in a format where I can easily go and replace them if something goes awry down the road. Then, I check my authentication each time I call into the API. If I get an authorization exception back I refresh my token programmatically, which you can do! Using the BoxApi.V2 .NET SDK this happens like so:
var authenticator = new TokenProvider(_clientId, _clientSecret);
// calling the 'RefreshAccessToken' method in the SDK
var newAuthToken = authenticator.RefreshAccessToken([YOUR EXISTING REFRESH TOKEN]);
// write the new token back to my data store.
Save(newAuthToken);
Hope this helped!
If I understand correctly you want the entire process to be automated so it would not require a user login (i.e run a script and the file is uploaded).
Well, it is possible. I am a rookie developer so excuse me if I'm not using the correct terms.
Anyway, this can be accomplished by using cURL.
First you need to define some variables, your user credentials (username and password), your client id and client secret given by Box (found in your app), your redirect URI and state (used for extra safety if I understand correctly).
The oAuth2.0 is a 4 step authentication process and you're going to need to go through each step individually.
The first step would be setting a curl instance:
curl_setopt_array($curl, array(
CURLOPT_URL => "https://app.box.com/api/oauth2/authorize",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "content-type: application/x-www-form-urlencoded",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS =>
"response_type=code&client_id=".$CLIENT_ID."&state=".$STATE,
));
This will return an html text with a request token, you will need it for the next step so I would save the entire output to a variable and grep the tag with the request token (the tag has a "name" = "request_token" and a "value" which is the actual token).
Next step you will need to send another curl request to the same url, this time the post fields should include the request token, user name and password as follows:
CURLOPT_POSTFIELDS => "response_type=code&client_id=".$CLIENT_ID."&state=".$STATE."&request_token=".$REQ_TOKEN."&login=".$USER_LOGIN."&password=".$PASSWORD
At this point you should also set a cookie file:
CURLOPT_COOKIEFILE => $COOKIE, (where $COOKIE is the path to the cookie file)
This will return another html text output, use the same method to grep the token which has the name "ic".
For the next step you're going to need to send a post request to the same url. It should include the postfields:
response_type=code&client_id=".$CLIENT_ID."&state=".$STATE."&redirect_uri=".$REDIRECT_URI."&doconsent=doconsent&scope=root_readwrite&ic=".$IC
Be sure to set the curl request to use the cookie file you set earlier like this:
CURLOPT_COOKIEFILE => $COOKIE,
and include the header in the request:
CURLOPT_HEADER => true,
At step (if done by browser) you will be redirected to a URL which looks as described above:
http://0.0.0.0(*redirect uri*)/?state=[box-generated_state_security_token]&code=[SOME_CODE] and note the "code=" value.
Grab the value of "code".
Final step!
send a new cur request to https//app.box.com/api/oauth2/token
This should include fields:
CURLOPT_POSTFIELDS => "grant_type=authorization_code&code=".$CODE."&client_id=".$CLIENT_ID."&client_secret=".$CLIENT_SECRET,
This will return a string containing "access token", "Expiration" and "Refresh token".
These are the tokens needed for the upload.
read about the use of them here:
https://box-content.readme.io/reference#upload-a-file
Hope this is somewhat helpful.
P.S,
I separated the https on purpuse (Stackoverflow wont let me post an answer with more than 1 url :D)
this is for PHP cURL. It is also possible to do the same using Bash cURL.
For anyone looking into this recently, the best way to do this is to create a Limited Access App in Box.
This will let you create an access token which you can use for server to server communication. It's simple to then upload a file (example in NodeJS):
import box from "box-node-sdk";
import fs from "fs";
(async function (){
const client = box.getBasicClient(YOUR_ACCESS_TOKEN);
await client.files.uploadFile(BOX_FOLDER_ID, FILE_NAME, fs.createReadStream(LOCAL_FILE_PATH));
})();
Have you thought about creating a box 'integration' user for this particular purpose. It seems like uploads have to be made with a Box account. It sounds like you are trying to do an anonymous upload. I think box, like most services, including stackoverflow don't want anonymous uploads.
You could create a system user. Go do the Oauth2 dance and store just the refresh token somewhere safe. Then as the first step of your script waking up go use the refresh token and store the new refresh token. Then upload all your files.