Given a Google Sheet which has data coming from BigQuery via the Sheets data connector for BigQuery (as described here), is it possible to force a refresh of this BigQuery data via the Sheets API? It is possible to do so through the Sheets UI (has a refresh button), but I want to do it via some other service invoking an API.
You can use Apps Script to interact with the data connector in your Spreadsheet.
Considerations
From the Sheet API you are not able to manipulate the DataSource. You can use Apps Script instead. It's relatively easy to set up a script that refreshes you BigQuery data.
Approach
If you want to do this operation with an external API you will have to use Apps Script as a proxy to achieve your goal. You can deploy your Apps Script as a API Executable and trigger its functions from an external service.
Apps Script
/** #OnlyCurrentDoc */
function refresh() {
var spreadsheet = SpreadsheetApp.getActive();
spreadsheet.getRange('A1').activate();
spreadsheet.setActiveSheet(spreadsheet.getSheetByName('Data Sheet 1'), true);
SpreadsheetApp.enableAllDataSourcesExecution();
spreadsheet.getCurrentCell().getDataSourceTables()[0].refreshData();
};
Your External Service
// [...] Oauth and other set ups...
public static void main(String[] args) throws IOException {
// ID of the script to call. Acquire this from the Apps Script editor,
// under Publish > Deploy as API executable.
String scriptId = "ENTER_YOUR_SCRIPT_ID_HERE";
Script service = getScriptService();
// Create an execution request object.
ExecutionRequest request = new ExecutionRequest()
.setFunction("refresh");
try {
// Make the API request.
Operation op =
service.scripts().run(scriptId, request).execute();
// [...] Error handling...
Refrences:
Big Query data connector
Apps Script API
Related
I see similar questionns has been asked multiple times before, but I cant seem to get them to work. I've also read that Google changed how their URLs are built up, so most of the solutions were deprecated unfortunately.
I'm looking for a link to a specific sheet of a workbook that has been published. I've made a simple workbook to test, and the published link looks like this:
https://docs.google.com/spreadsheets/d/e/2PACX-1vRrmEbjecLvXhbm409pa6JJXZd_ZXTG8Zt6OevIUs5Axq5oxlCZKU0QXk-2lW05HyXJ2B4Bzy3bG-4L/pubhtml
As you can see there is a top menu to change between the sheets, but that doesn't affect the URL.
Is there any way I can get a URL to "Sheet2" directly? Or is that dependant on having the Sheet ID (I'm not the owner of said spreadsheet)?
I believe your goal as follows.
You want to retrieve the values from Sheet2 from the URL of https://docs.google.com/spreadsheets/d/e/2PACX-1vRrmEbjecLvXhbm409pa6JJXZd_ZXTG8Zt6OevIUs5Axq5oxlCZKU0QXk-2lW05HyXJ2B4Bzy3bG-4L/pubhtml.
The owner of this Spreadsheet is not you.
You don't know the Spreadsheet ID and each sheet ID in the Spreadsheet. You know only the URL of https://docs.google.com/spreadsheets/d/e/2PACX-###/pubhtml.
Under above situation, you want to retrieve the direct URL of the sheet 2.
For above goal, how about this answer?
Issue and workarounds:
Unfortunately, in the current stage, it seems that the Spreadsheet ID and each sheet ID cannot be directly retrieved from the URL of https://docs.google.com/spreadsheets/d/e/2PACX-###/pubhtml. I think that this is the current specification. Also I think that this reason might be due to the security. So in order to achieve your goal, it is required to think of the workaround.
In this answer, as a workaround, I would like to achieve your goal using Web Apps created by Google Apps Script. When Web Apps is used, the directlink of Sheet2 can be retrieved.
Flow:
The flow of this workaround is as follows.
Download the Google Spreadsheet as a XLSX data from the URL of https://docs.google.com/spreadsheets/d/e/2PACX-###/pubhtml.
Convert the XLSX data to Google Spreadsheet.
Publish the converted Google Spreadsheet to Web.
Retrieve the URLs of each sheet.
Usage:
Please do the following flow.
1. Create new project of Google Apps Script.
Sample script of Web Apps is a Google Apps Script. So please create a project of Google Apps Script.
If you want to directly create it, please access to https://script.new/. In this case, if you are not logged in Google, the log in screen is opened. So please log in to Google. By this, the script editor of Google Apps Script is opened.
2. Prepare script.
Please copy and paste the following script (Google Apps Script) to the script editor. And please enable Google Drive API at Advanced Google services. This script is for the Web Apps.
function doGet(e) {
const prop = PropertiesService.getScriptProperties();
const ssId = prop.getProperty("ssId");
if (ssId) {
DriveApp.getFileById(ssId).setTrashed(true);
prop.deleteProperty("ssId");
}
const inputUrl = e.parameter.url;
const re = new RegExp("(https?:\\/\\/docs\\.google\\.com\\/spreadsheets\\/d\\/e\\/2PACX-.+?\\/)");
if (!re.test(inputUrl)) return ContentService.createTextOutput("Wrong URL.");
const url = `${inputUrl.match(re)[1]}pub?output=xlsx`;
const blob = UrlFetchApp.fetch(url).getBlob();
const id = Drive.Files.insert({mimeType: MimeType.GOOGLE_SHEETS, title: "temp"}, blob).id;
prop.setProperty("ssId", id);
Drive.Revisions.update({published: true, publishedOutsideDomain: true, publishAuto: true}, id, 1);
const sheets = SpreadsheetApp.openById(id).getSheets();
const pubUrls = sheets.map(s => ({[s.getSheetName()]: `https://docs.google.com/spreadsheets/d/${id}/pubhtml?gid=${s.getSheetId()}`}));
return ContentService.createTextOutput(JSON.stringify(pubUrls)).setMimeType(ContentService.MimeType.JSON);
}
In this case, the GET method is used.
In this script, when the below curl command is run, the Google Spreadsheet is downloaded as a XLSX data, and the XLSX data is converted to Google Spreadsheet. Then, the converted Spreadsheet is published to the web. By this, the direct links of each sheet can be retrieved.
Also, in this script, it supposes that the original Spreadsheet is changed. So if you run the curl command again, the existing Spreadsheet is deleted and new Spreadsheet is created by downloading from the original Spreadsheet. In this case, the URLs are updated.
So if the Spreadsheet is not changed, you can continue to use the retrieved URLs. Of course, you can also directly use the downloaded and converted Spreadsheet.
3. Deploy Web Apps.
On the script editor, Open a dialog box by "Publish" -> "Deploy as web app".
Select "Me" for "Execute the app as:".
By this, the script is run as the owner.
Select "Anyone, even anonymous" for "Who has access to the app:".
In this case, no access token is required to be request. I think that I recommend this setting for your goal.
Of course, you can also use the access token. At that time, please set this to "Anyone".
Click "Deploy" button as new "Project version".
Automatically open a dialog box of "Authorization required".
Click "Review Permissions".
Select own account.
Click "Advanced" at "This app isn't verified".
Click "Go to ### project name ###(unsafe)"
Click "Allow" button.
Click "OK".
Copy the URL of Web Apps. It's like https://script.google.com/macros/s/###/exec.
When you modified the Google Apps Script, please redeploy as new version. By this, the modified script is reflected to Web Apps. Please be careful this.
4. Run the function using Web Apps.
This is a sample curl command for requesting Web Apps. Please set your Web Apps URL.
curl -L "https://script.google.com/macros/s/###/exec?url=https://docs.google.com/spreadsheets/d/e/2PACX-1vRrmEbjecLvXhbm409pa6JJXZd_ZXTG8Zt6OevIUs5Axq5oxlCZKU0QXk-2lW05HyXJ2B4Bzy3bG-4L/pubhtml"
In this case, the GET method is used at Web Apps side. So you can also directly access to the above URL using your browser.
Note:
When you modified the script of Web Apps, please redeploy the Web Apps as new version. By this, the latest script is reflected to the Web Apps. Please be careful this.
In this answer, I thought that you might use this from outside. So I used Web Apps. If you want to directly retrieved from the Google Apps Script, you can also use the following script.
function myFunction() {
const inputUrl = "https://docs.google.com/spreadsheets/d/e/2PACX-1vRrmEbjecLvXhbm409pa6JJXZd_ZXTG8Zt6OevIUs5Axq5oxlCZKU0QXk-2lW05HyXJ2B4Bzy3bG-4L/pubhtml";
const prop = PropertiesService.getScriptProperties();
const ssId = prop.getProperty("ssId");
if (ssId) {
DriveApp.getFileById(ssId).setTrashed(true);
prop.deleteProperty("ssId");
}
const re = new RegExp("(https?:\\/\\/docs\\.google\\.com\\/spreadsheets\\/d\\/e\\/2PACX-.+?\\/)");
if (!re.test(inputUrl)) throw new Error("Wrong URL.");
const url = `${inputUrl.match(re)[1]}pub?output=xlsx`;
const blob = UrlFetchApp.fetch(url).getBlob();
const id = Drive.Files.insert({mimeType: MimeType.GOOGLE_SHEETS, title: "temp"}, blob).id;
prop.setProperty("ssId", id);
Drive.Revisions.update({published: true, publishedOutsideDomain: true, publishAuto: true}, id, 1);
const sheets = SpreadsheetApp.openById(id).getSheets();
const pubUrls = sheets.map(s => ({[s.getSheetName()]: `https://docs.google.com/spreadsheets/d/${id}/pubhtml?gid=${s.getSheetId()}`}));
console.log(pubUrls); // You can see the URLs for each sheet at the log.
}
References:
Web Apps
Taking advantage of Web Apps with Google Apps Script
Advanced Google services
publish a Google Spreadsheet through Google Apps Scripts
Added:
As another workaround, when the original Spreadsheet is often changed, and the number of sheet is constant in the original Spreadsheet, and then, you want to retrieve only values, you can also use the following script. In this script, the URL is not changed even when the script is run again. So you can continue to use the URL.
Sample script:
function myFunction() {
const inputUrl = "https://docs.google.com/spreadsheets/d/e/2PACX-1vRrmEbjecLvXhbm409pa6JJXZd_ZXTG8Zt6OevIUs5Axq5oxlCZKU0QXk-2lW05HyXJ2B4Bzy3bG-4L/pubhtml";
const re = new RegExp("(https?:\\/\\/docs\\.google\\.com\\/spreadsheets\\/d\\/e\\/2PACX-.+?\\/)");
if (!re.test(inputUrl)) throw new Error("Wrong URL.");
const url = `${inputUrl.match(re)[1]}pub?output=xlsx`;
const blob = UrlFetchApp.fetch(url).getBlob();
const prop = PropertiesService.getScriptProperties();
let sheets;
let ssId = prop.getProperty("ssId");
if (ssId) {
const temp = Drive.Files.insert({mimeType: MimeType.GOOGLE_SHEETS, title: "tempSpreadsheet"}, blob).id;
const tempSheets = SpreadsheetApp.openById(temp).getSheets();
sheets = SpreadsheetApp.openById(ssId).getSheets();
tempSheets.forEach((e, i) => {
const values = e.getDataRange().getValues();
sheets[i].getRange(1, 1, values.length, values[0].length).setValues(values);
});
DriveApp.getFileById(temp).setTrashed(true);
} else {
ssId = Drive.Files.insert({mimeType: MimeType.GOOGLE_SHEETS, title: "copiedSpreadsheet"}, blob).id;
Drive.Revisions.update({published: true, publishedOutsideDomain: true, publishAuto: true}, ssId, 1);
prop.setProperty("ssId", ssId);
sheets = SpreadsheetApp.openById(ssId).getSheets();
}
const pubUrls = sheets.map(s => ({[s.getSheetName()]: `https://docs.google.com/spreadsheets/d/${ssId}/pubhtml?gid=${s.getSheetId()}`}));
console.log(pubUrls); // You can see the URLs for each sheet at the log.
}
I'm looking for a way to either
1) Read/retrieve share notifications whenever a Sheet is shared with a specific service account
or
2) Get a list of all Sheets shared with a specific service account
Background: Users duplicate an existing Sheet template, modify its contents and share it with my service account email so I can retrieve the Sheet data programmatically. This still requires the users to input the resulting share link into my backend after sharing.
Instead I'd prefer using the API to receive either something like a "shared with service account" webhook event or an option to read all Sheets shared with this service account.
Does this require GSuite, or is there an API/webhook to achieve this?
Answer:
Yes, you can retrieve this information with the Drive API.
More Information:
If you make a Drive: files.list call as a service account, it will return the files of the Service Account's Drive.
If your users are sharing Sheets with the Service Account, you can retrieve them by making an API call to this method with the sharedWithMe flag set to true, and the mimeType set to application/vnd.google-apps.spreadsheet in the q parameter.
JavaScript example:
function execute() {
return gapi.client.drive.files.list({
"q": "sharedWithMe and mimeType = 'application/vnd.google-apps.spreadsheet'"
})
.then(function(response) {
// Handle the results here (response.result has the parsed body).
console.log("Response", response);
},
function(err) { console.error("Execute error", err); });
}
References:
Files: list | Google Drive API
Search for files and folders | Google Drive API
G Suite and Drive MIME Types | Google Drive API
I am trying to collect energy generation statistics like Watts and wattHour form external API. I have External Rest API endpoint available for it.
Is there a way in Thingsboard using rule chain to call external endpoint and store its as telemetry data. Later i want to show this data in dashboards.
I know it has been too long, but thingsboard has lacking documentation and it might be useful for someone else.
You'd have to use the REST API CALL external node (https://thingsboard.io/docs/user-guide/rule-engine-2-0/external-nodes/#rest-api-call-node)
If the Node was successful, it will output it's OutboundMessage containing the HTTP Response, with the metadata containing:
- metadata.status
- metadata.statusCode
- metadata.statusReason
and with the payload of the message containing the response body from your external REST service (i.e. your stored telemetry).
You then have to use a script transformation node in order to format the metadata, payload and msgType, into the POST_TELEMETRY_REQUEST message format, see: https://thingsboard.io/docs/user-guide/rule-engine-2-0/overview/#predefined-message-types
Your external REST API should provide the correct "deviceName" and "deviceType", as well as the "ts" in UNIX milliseconds.
Notice that you also need to change the messageType (msgType return variable) to "POST_TELEMETRY_REQUEST".
Finally, just transmit the result to the Save timeseries action node and it will be stored as a telemetry from the specified device. Hope this helps.
I have a query that runs and can see the results. But while trying to save the query as a view table, I get error message saying
Failed to save view. No suitable credentials found to access Google
Drive. Contact the table owner for assistance.
I think the problem is caused by a table used in the query. The table is uploaded from a google sheet (with source URI), own by me. I have tried to enable Google Drive API from the project but no luck. Not sure how I can give BigQuery access to Google Drive.
I suspect the problem you are hitting is one of OAuth Scopes. In order to talk to the Google Drive API to read data, you need to use credentials that were granted access to that API.
If you are using the BigQuery web UI and have not explicitly granted access to Drive, it won't work. For example, the first time I tried to "Save to Google Sheets", the BigQuery UI popped up an OAuth prompt asking me to grant access to my Google Drive. After this it could save the results. Try doing this to make sure your credentials have the Drive scope and then "Save View" again.
If you are using your own code to do this, you should request scope 'https://www.googleapis.com/auth/drive' in addition to the 'https://www.googleapis.com/auth/bigquery' scope you are already using to talk to BigQuery.
If you are using the bq client, it has been updated to request this scope, but you may need to re-initialize your authentication credentials. You can do this with bq init --delete_credentials to remove the credentials, then your next action we re-request credentials.
Using Google App Script this worked for me:
function saveQueryToTable() {
var projectId = '...yourprojectid goes here...';
var datasetId = '...yourdatesetid goes here...';
var sourceTable = '...your table or view goes here...';
var destTable = '...destination table goes here...';
var myQuery;
//just a random call to activate the Drive API scope
var test = Drive.Properties.list('...drive file id goes here...')
//list all tables for the particular dataset
var tableList = BigQuery.Tables.list(projectId, datasetId).getTables();
//if the table exist, delete it
for (var i = 0; i < tableList.length; i++) {
if (tableList[i].tableReference.tableId == destTable) {
BigQuery.Tables.remove(projectId, datasetId, destTable);
Logger.log("DELETED: " + destTable);
}
};
myQuery = 'SELECT * FROM [PROJECTID:DATASETID.TABLEID];'
.replace('PROJECTID',projectId)
.replace('DATASETID',datasetId)
.replace('TABLEID',sourceTable)
var job = {
configuration: {
query: {
query: myQuery,
destinationTable: {
projectId: projectId,
datasetId: datasetId,
tableId: destTable
}
}
}
};
var queryResults = BigQuery.Jobs.insert(job, projectId);
Logger.log(queryResults.status);
}
The 'trick' was a random call to the Drive API to ensure both the BigQuery and Drive scopes are included.
Google Apps Script Project Properties
I used to be able to query the Google Analytics API with my account's login & password.
Google is now using OAuth for authentication which is great...
The only issue is that I only need ONE access token.
I don't wanna allow other users to fetch THEIR analytics data.
I just wanna be able to fetch MY data.
Is there a way I can generate an access token only for my app or my analytics account?
I know such solutions exists... For instance, Twitter provides what they call a "single-user oauth" for apps that don't require a specific user to sign in.
One again, all I'm trying to accomplish here is to fetch MY OWN analytics data via the API.
Is there a way to properly do that?
I'm adding a PHP answer - you may be able to adjust or convert it to garb / ruby code.
You should be able to use Analytics with service accounts now. You will indeed have to use a private key instead of an access token.
Create an app in the API Console
Basically, you go to the Google API Console and create an App.
Enable Google Analytics in the services tab.
In the API Access tab, create a new OAuth ID (Create another client ID... button), select service account and download your private key (Generate new key... link). You'll have to upload the key to your web server later.
On the API Access page, in the Service account section, copy the email address (#developer.gserviceaccount.com) and add a new user with this email address to your Google Analytics profile. If you do not do this, you'll get some nice errors
Code
Download the latest Google PHP Client off SVN (from the command line svn checkout http://google-api-php-client.googlecode.com/svn/trunk/ google-api-php-client-read-only).
You can now access the Analytics API in code:
require_once 'Google_Client.php';
require_once 'contrib/Google_AnalyticsService.php';
$keyfile = 'dsdfdss0sdfsdsdfsdf44923dfs9023-privatekey.p12';
// Initialise the Google Client object
$client = new Google_Client();
$client->setApplicationName('Your product name');
$client->setAssertionCredentials(
new Google_AssertionCredentials(
'11122233344#developer.gserviceaccount.com',
array('https://www.googleapis.com/auth/analytics.readonly'),
file_get_contents($keyfile)
)
);
// Get this from the Google Console, API Access page
$client->setClientId('11122233344.apps.googleusercontent.com');
$client->setAccessType('offline_access');
$analytics = new Google_AnalyticsService($client);
// We have finished setting up the connection,
// now get some data and output the number of visits this week.
// Your analytics profile id. (Admin -> Profile Settings -> Profile ID)
$analytics_id = 'ga:1234';
$lastWeek = date('Y-m-d', strtotime('-1 week'));
$today = date('Y-m-d');
try {
$results = $analytics->data_ga->get($analytics_id,
$lastWeek,
$today,'ga:visits');
echo '<b>Number of visits this week:</b> ';
echo $results['totalsForAllResults']['ga:visits'];
} catch(Exception $e) {
echo 'There was an error : - ' . $e->getMessage();
}
Terry Seidler answered this nicely for php. I want to add a java code example.
Api console setup
Start by doing the required steps in the google api console as Terry explained:
Basically, you go to the Google API Console and create an App. Enable
Google Analytics in the services tab. In the API Access tab, create a
new OAuth ID (Create another client ID... button), select service
account and download your private key (Generate new key... link).
You'll have to upload the key to your web server later.
On the API Access page, in the Service account section, copy the email
address (#developer.gserviceaccount.com) and add a new user with this
email address to your Google Analytics profile. If you do not do this,
you'll get some nice errors
Get the necessary libraries
Download the google analytics java client from:
https://developers.google.com/api-client-library/java/apis/analytics/v3
Or add the following maven dependencies:
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-analytics</artifactId>
<version>v3-rev94-1.18.0-rc</version>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client-jackson</artifactId>
<version>1.18.0-rc</version>
</dependency>
Now for the code:
public class HellowAnalyticsV3Api {
private static final HttpTransport HTTP_TRANSPORT = new NetHttpTransport();
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
public void analyticsExample() {
// This is the .p12 file you got from the google api console by clicking generate new key
File analyticsKeyFile = new File(<p12FilePath>);
// This is the service account email address that you can find in the api console
String apiEmail = <something#developer.gserviceaccount.com>;
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(apiEmail)
.setServiceAccountScopes(Arrays.asList(AnalyticsScopes.ANALYTICS_READONLY))
.setServiceAccountPrivateKeyFromP12File(analyticsPrivateKeyFile).build();
Analytics analyticsService = new Analytics.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential)
.setApplicationName(<your application name>)
.build();
String startDate = "2014-01-03";
String endDate = "2014-03-03";
String mertrics = "ga:sessions,ga:timeOnPage";
// Use the analytics object build a query
Get get = analyticsService.data().ga().get(tableId, startDate, endDate, mertrics);
get.setDimensions("ga:city");
get.setFilters("ga:country==Canada");
get.setSort("-ga:sessions");
// Run the query
GaData data = get.execute();
// Do something with the data
if (data.getRows() != null) {
for (List<String> row : data.getRows()) {
System.out.println(row);
}
}
}
You can use a refresh token. Store the refresh token in a db or secure config file, then use it to show the stats.
Google API Offline Access Using OAuth 2.0 Refresh Token will give you an idea of how to capture then store your refresh token.
See also Using OAuth 2.0 for Web Server Applications - Offline Access
Hello I found a solution, it works for me
you have to change this one
immediate: true
to
immediate: false
and it looks like
function checkAuth() {
gapi.auth.authorize({
client_id: clientId, scope: scopes, immediate: false}, handleAuthResult);
}
Google has the 'Service Account' (Calls Google APIs on behalf of your application instead of an end-user), but the way it works is a bit different as it won't use access tokens but a private key instead.
You can find more details at https://developers.google.com/accounts/docs/OAuth2ServiceAccount