I'm having an issue getting a Drive ID when making a GET request.
First I'm creating the Team using "https://graph.microsoft.com/v1.0/teams". The request returns no response but a Status 202 accepted, the new Team is listed on Teams. I then extract the Team ID from Response Headers "Location".
Here is the failure, I make a request to https://graph.microsoft.com/v1.0/groups/{{TeamsID}}/drive
Here is my response back
{"error":{"code":"ResourceNotFound","message":"Resource is not found.","innerError":{"date":"2021-05-03T19:17:03","request-id":"75b4888d-de91-41aa-89ef-d983fcb5238d","client-request-id":"75b4888d-de91-41aa-89ef-d983fcb5238d"}}}
I make the same request on via postman and it works fine.
Here is my code
try {
let crt = await createTeam(msalData, data);
crt.headers.forEach( async (val, key) => {
if (key === "location") {
let id = val.split("/teams('")[1].split("')/operations")[0];
let gdd = await getDocumentDrive(msalData, id);
console.log(gdd)
}
});
} catch (err) {
console.log(err);
}
I can't figure out what I'm doing wrong.
Apparently the error (ResourceNotFound) occurs since the moment when Get Drive request is submitted the Group is not yet completely provisioned, documentations says the following in this regard:
If the group was created less than 15 minutes ago, it's possible for
the Create team call to fail with a 404 error code due to replication
delays. The recommended pattern is to retry the Create team call three
times, with a 10 second delay between calls.
Since you haven't revealed how exactly createTeam function is implemented, retry pattern come to the rescue here, for example:
async function withRetry(fn, maxRetry = 5, ms = 1000) {
try {
const val = await fn();
return val;
} catch (error) {
if (maxRetry) {
await new Promise((r) => setTimeout(r, ms));
return withRetry(fn, maxRetry - 1, ms);
} else throw new Error(`Max retries reached for function ${fn.name}`);
}
}
and now Teams could be created like this (note, in retry logic Get Group endpoint request is submitted to ensure it has been provisioned):
async function ensureTeam(client, teamName, defaultOwnerId) {
const teamResp = await createTeam(client, teamName, defaultOwnerId);
const teamloc = teamResp.headers.get("Content-Location");
return await withRetry(
async () => {
const groupUrl = teamloc.replace("teams", "groups");
return await client.api(groupUrl).get();
},
5,
10000
);
}
where createTeam could look like this:
async function createTeam(client, teamName, defaultOwnerId) {
const team = {
"template#odata.bind":
"https://graph.microsoft.com/v1.0/teamsTemplates('standard')",
displayName: teamName,
description: teamName,
members: [
{
"#odata.type": "#microsoft.graph.aadUserConversationMember",
roles: ["owner"],
"user#odata.bind": `https://graph.microsoft.com/v1.0/users('${defaultOwnerId}')`,
},
],
};
return await client.api("/teams").responseType(ResponseType.RAW).post(team);
}
Usage
const graphClient = Client.initWithMiddleware({ authProvider });
const group = await ensureTeam(graphClient,"Demo team",ownerId);
const driveUrl = `/groups/${group.id}/drive`;
const drive = await graphClient.api(driveUrl).get();
Related
We are trying web crawl and get contents from multiple pages. I am taking the advantage of async API with Promise ALL which can execute requests in parallel.
Is there a limitation on the number of contexts which can be opened parallel?
const fs = require('fs');
let browser;
const batch_size = 4; // control the number of async parallel calls
(async () => { // main function
let urls = [];
urls = fs.readFileSync('./resources/input_selenium_urls.csv').toString().split("\n");
browser = await chromium.launch();
let context_size = 0;
let processUrls = [];
let total_length = 0;
for (let i=0;i<urls.length;i++,total_length++) {
if ((context_size==batch_size)||(i==urls.length-1)){
await Promise.all(processUrls.map(x => getHTMLPageSource(x)));
context_size = 0;
processUrls = [];
} else {
processUrls.push(urls[i]);
context_size++;
}
}
await browser.close();
})();
async function getHTMLPageSource(url) {
const context = await browser.newContext();
const page = await context.newPage();
let response = {}
try {
await page.goto(url, { waitUntil: 'networkidle' });
response = {
url : url,
content: await page.title(),
error : null
}
console.log(response);
}
catch {
response = {
error : "Timeout error"
}
}
context.close;
return response;
}
Browser contexts are cheap to create, but it's not clear whether there is a hard-coded limit on them from the docs perhaps the limit might depend on the browser you chose and your OS resources. I think you might only be able to find out by creating a lot of contexts.
fine people of Stack Overflow.
I'm trying to solve a problem I'm having involving twilio functions, messaging services, and databases.
What I'm attempting to do is send a message to all members of a database at once.
My code is a mess, as Javascript isn't my native language and I'm rather new to twilio.
The problem I believe I'm having is with the async/await feature of javascript.
Here is my code so far:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
allItems = res.items;
// continue fetching until last is not seen
while (res.last){
res = await db.fetch({}, {last: res.last});
allItems = allItems.concat(res.items);
}
}
// Function to get total number of contacts.
async function ReturnNumberOfContacts(allItems) {
number_of_contacts = allItems.length;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message) {
allItems.forEach(contact => {
let users_name = contact.name
client.messages
.create({
body: `Hey ${users_name}! ${message}`,
messagingServiceSid: messaging_service,
to: contact.key
})
});
}
// Function to submit response to broadcaster.
async function SuccessResponse(user_name, number_of_contacts) {
responseObject = {
"actions": [
{
"say": `${user_name}, your broadcast has successfully sent to ${number_of_contacts} contacts.`
},
{
"listen": true
}
]
}
}
// Main Function
exports.handler = async function(context, event, callback) {
// Placeholder for number of contacts
let number_of_contacts;
// Place holder for object from database of all contacts
let allItems;
// Placeholder for users message
let message;
// Placeholder for response to user
let responseObject;
//Twilio and Deta, Etc Const
const client = require('twilio')(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messaging_service = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
await FetchDB(db, allItems).then(ReturnNumberOfContacts(allItems));
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message = memory.twilio.collected_data.broadcast_message.answers.message_input.answer;
}
// Check if verified and set name.
const current_user = await db.get(from);
// Get the current users name or set a default value
let user_name = current_user.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
responseObject = {
"actions": [
{
"say": `${user_name}, you have canceled your request and no messages have been sent.`
},
{
"listen": false
}
]
}
// Return Callback and end task
callback(null, responseObject);
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(message, client, messaging_service)
.then(SuccessResponse(user_name, number_of_contacts))
return callback(null, responseObject);
}
// The user is not authorized so return this.
}
return callback(null, {
"actions": [
{
"say": "You are not authorized to broadcast."
},
{
"listen": false
}
]
})
};
So when the Fetch() function is triggered, I want the database to load a list of everyone and have twilio send them the desired message saved in the message variable. I have the code working so that I can read from the database and get the proper values, and send a single text message with the desired message, but the problem I'm having now is integrating it all together.
Thanks if anyone can point me in the right direction here.
Again, I'm new to javascript and more specifically asynchronous programming.
Twilio developer evangelist here.
The issue from the error says that allItems is undefined when you call ReturnNumberOfContacts.
I think the issue comes from trying to use allItems as a sort of global variable, same for number_of_contacts. It would be better for FetchDB to resolve with the list of items and ReturnNumberOfContacts to resolve with the number of the items.
You also have some arguments missing when you call SendMessages in your function. I've updated it to the point that I think it will work:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message, client, messagingService) {
return Promise.all(
allItems.map((contact) => {
let usersName = contact.name;
return client.messages.create({
body: `Hey ${usersName}! ${message}`,
messagingServiceSid: messagingService,
to: contact.key,
});
})
);
}
// Main Function
exports.handler = async function (context, event, callback) {
// Placeholder for users message
let message;
//Twilio and Deta, Etc Const
const client = require("twilio")(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messagingService = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message =
memory.twilio.collected_data.broadcast_message.answers.message_input
.answer;
}
// Check if verified and set name.
const currentUser = await db.get(from);
// Get the current users name or set a default value
let userName = currentUser.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
// Return Callback and end task
callback(null, {
actions: [
{
say: `${userName}, you have canceled your request and no messages have been sent.`,
},
{
listen: false,
},
],
});
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(allItems, message, client, messagingService);
return callback(null, {
actions: [
{
say: `${userName}, your broadcast has successfully sent to ${numberOfContacts} contacts.`,
},
{
listen: true,
},
],
});
}
// The user is not authorized so return this.
}
return callback(null, {
actions: [
{
say: "You are not authorized to broadcast.",
},
{
listen: false,
},
],
});
};
What I did here was change FetchDB to only take the db as an argument, then to create a local allItems variable that collects all the contacts and then returns them.
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
This is then called in the main body of the function to assign a local variable. I also replaced the ReturnNumberOfContacts function with a simple assignment.
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
One thing you may want to consider is how many contacts you are trying to send messages to during this function. There are a few limits you need to be aware of.
Firstly, Function execution time is limited to 10 seconds so you need to make sure you can load and send all your messages within that amount of time if you want to use a Twilio Function for this.
Also, there are limits for the number of concurrent connections you can make to the Twilio API. That limit used to be 100 connections per account, but it may vary these days. When sending asynchronous API requests as you do in JavaScript, the platform will attempt to create as many connections to the API that it can in order to trigger all the requests asynchronously. If you have more than 100 contacts you are trying to send messages to here, that will quickly exhaust your available concurrent connections and you will receive 429 errors. You may choose to use a queue, like p-queue, to ensure your concurrent connections never get too high. The issue in this case is that it will then take longer to process the queue which brings me back to the original limit of 10 seconds of function execution.
So, I think the above code may work in theory now, but using it in practice may have other issues that you will need to consider.
fine people of Stack Overflow.
I'm trying to solve a problem I'm having involving twilio functions, messaging services, and databases.
What I'm attempting to do is send a message to all members of a database at once.
My code is a mess, as Javascript isn't my native language and I'm rather new to twilio.
The problem I believe I'm having is with the async/await feature of javascript.
Here is my code so far:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
allItems = res.items;
// continue fetching until last is not seen
while (res.last){
res = await db.fetch({}, {last: res.last});
allItems = allItems.concat(res.items);
}
}
// Function to get total number of contacts.
async function ReturnNumberOfContacts(allItems) {
number_of_contacts = allItems.length;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message) {
allItems.forEach(contact => {
let users_name = contact.name
client.messages
.create({
body: `Hey ${users_name}! ${message}`,
messagingServiceSid: messaging_service,
to: contact.key
})
});
}
// Function to submit response to broadcaster.
async function SuccessResponse(user_name, number_of_contacts) {
responseObject = {
"actions": [
{
"say": `${user_name}, your broadcast has successfully sent to ${number_of_contacts} contacts.`
},
{
"listen": true
}
]
}
}
// Main Function
exports.handler = async function(context, event, callback) {
// Placeholder for number of contacts
let number_of_contacts;
// Place holder for object from database of all contacts
let allItems;
// Placeholder for users message
let message;
// Placeholder for response to user
let responseObject;
//Twilio and Deta, Etc Const
const client = require('twilio')(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messaging_service = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
await FetchDB(db, allItems).then(ReturnNumberOfContacts(allItems));
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message = memory.twilio.collected_data.broadcast_message.answers.message_input.answer;
}
// Check if verified and set name.
const current_user = await db.get(from);
// Get the current users name or set a default value
let user_name = current_user.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
responseObject = {
"actions": [
{
"say": `${user_name}, you have canceled your request and no messages have been sent.`
},
{
"listen": false
}
]
}
// Return Callback and end task
callback(null, responseObject);
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(message, client, messaging_service)
.then(SuccessResponse(user_name, number_of_contacts))
return callback(null, responseObject);
}
// The user is not authorized so return this.
}
return callback(null, {
"actions": [
{
"say": "You are not authorized to broadcast."
},
{
"listen": false
}
]
})
};
So when the Fetch() function is triggered, I want the database to load a list of everyone and have twilio send them the desired message saved in the message variable. I have the code working so that I can read from the database and get the proper values, and send a single text message with the desired message, but the problem I'm having now is integrating it all together.
Thanks if anyone can point me in the right direction here.
Again, I'm new to javascript and more specifically asynchronous programming.
Twilio developer evangelist here.
The issue from the error says that allItems is undefined when you call ReturnNumberOfContacts.
I think the issue comes from trying to use allItems as a sort of global variable, same for number_of_contacts. It would be better for FetchDB to resolve with the list of items and ReturnNumberOfContacts to resolve with the number of the items.
You also have some arguments missing when you call SendMessages in your function. I've updated it to the point that I think it will work:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message, client, messagingService) {
return Promise.all(
allItems.map((contact) => {
let usersName = contact.name;
return client.messages.create({
body: `Hey ${usersName}! ${message}`,
messagingServiceSid: messagingService,
to: contact.key,
});
})
);
}
// Main Function
exports.handler = async function (context, event, callback) {
// Placeholder for users message
let message;
//Twilio and Deta, Etc Const
const client = require("twilio")(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messagingService = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message =
memory.twilio.collected_data.broadcast_message.answers.message_input
.answer;
}
// Check if verified and set name.
const currentUser = await db.get(from);
// Get the current users name or set a default value
let userName = currentUser.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
// Return Callback and end task
callback(null, {
actions: [
{
say: `${userName}, you have canceled your request and no messages have been sent.`,
},
{
listen: false,
},
],
});
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(allItems, message, client, messagingService);
return callback(null, {
actions: [
{
say: `${userName}, your broadcast has successfully sent to ${numberOfContacts} contacts.`,
},
{
listen: true,
},
],
});
}
// The user is not authorized so return this.
}
return callback(null, {
actions: [
{
say: "You are not authorized to broadcast.",
},
{
listen: false,
},
],
});
};
What I did here was change FetchDB to only take the db as an argument, then to create a local allItems variable that collects all the contacts and then returns them.
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
This is then called in the main body of the function to assign a local variable. I also replaced the ReturnNumberOfContacts function with a simple assignment.
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
One thing you may want to consider is how many contacts you are trying to send messages to during this function. There are a few limits you need to be aware of.
Firstly, Function execution time is limited to 10 seconds so you need to make sure you can load and send all your messages within that amount of time if you want to use a Twilio Function for this.
Also, there are limits for the number of concurrent connections you can make to the Twilio API. That limit used to be 100 connections per account, but it may vary these days. When sending asynchronous API requests as you do in JavaScript, the platform will attempt to create as many connections to the API that it can in order to trigger all the requests asynchronously. If you have more than 100 contacts you are trying to send messages to here, that will quickly exhaust your available concurrent connections and you will receive 429 errors. You may choose to use a queue, like p-queue, to ensure your concurrent connections never get too high. The issue in this case is that it will then take longer to process the queue which brings me back to the original limit of 10 seconds of function execution.
So, I think the above code may work in theory now, but using it in practice may have other issues that you will need to consider.
In my Electron App I would like to inject data (like Fixtures) when App launched.
I use typeorm library for managing my SQLite3 Database connection.
I created json file that represent Entity typeorm and I would like persist all of them in my DB with typeorm. For that It seems that use trasaction is more efficient.
I try two differents things but the result is the same and I don't uderstand why. The issue message is :
Error: Transaction already started for the given connection, commit
current transaction before starting a new one
My first implementation of transaction :
async setAll(entity, data)
{
let connection = await this.init()
const queryRunner = connection.createQueryRunner()
await queryRunner.connect()
for (const [key, value] of Object.entries(data))
{
await typeorm.getManager().transaction(transactionalEntityManager =>
{
})
}
}
My second implementation of transaction :
async setAll(entity, data)
{
let connection = await this.init()
const queryRunner = connection.createQueryRunner()
await queryRunner.connect()
for (const [key, value] of Object.entries(data))
{
let genre1 = new Genre()
genre1.name = 'toto'
genre1.identifier = 'gt'
genre1.logo = ''
genre1.isActivate = false
await queryRunner.startTransaction()
await queryRunner.manager.save(genre1)
await queryRunner.commitTransaction()
await queryRunner.release()
}
}
NB : The second implementation persist correctly the first object but not the others.
How can manage many typeorm Transaction created into loop for persist lot of data ?
async setAll(entity, data) {
let connection = await this.init()
const queryRunner = connection.createQueryRunner()
await queryRunner.connect()
await queryRunner.startTransaction()
try {
for await (const [key, value] of Object.entries(data)) {
let genre1 = new Genre()
genre1.name = 'toto'
genre1.identifier = 'gt'
genre1.logo = ''
genre1.isActivate = false
const newGenre= queryRunner.manager.create(Genre,genre1)
await queryRunner.manager.save(newGenre)
}
await queryRunner.commitTransaction()
} catch {
await queryRunner.rollbackTransaction()
} finally {
await queryRunner.release()
}
I want to monitor my dataflow jobs with an application. The application I'm developing is a nodejs application and ideally it would exist a package like #google-cloud/bigquery but for dataflow instead. I'm fully aware that I might not be able to start job, if it is not a template job, but it should be an easy way to list jobs or get job information.
Update:
I found this spec, https://dataflow.googleapis.com/$discovery/rest?version=v1b3, but I don't understand what location is for the list operation. The spec was linked from this page: https://cloud.google.com/dataflow/docs/reference/rest/
I did find the solution myself. There is a repo that basically has all the APIs for gcloud out there: https://github.com/google/google-api-nodejs-client
After I found that I could easily do what I wanted:
'use strict';
var google = require('googleapis');
var dataflow = google.dataflow('v1b3');
google.auth.getApplicationDefault(function (err, authClient, projectId) {
if (err) {
throw err;
}
// The createScopedRequired method returns true when running on GAE or a local developer
// machine. In that case, the desired scopes must be passed in manually. When the code is
// running in GCE or a Managed VM, the scopes are pulled from the GCE metadata server.
// See https://cloud.google.com/compute/docs/authentication for more information.
if (authClient.createScopedRequired && authClient.createScopedRequired()) {
// Scopes can be specified either as an array or as a single, space-delimited string.
authClient = authClient.createScoped([
'https://www.googleapis.com/auth/compute'
]);
}
// Fetch the list of GCE zones within a project.
// NOTE: You must fill in your valid project ID before running this sample!
var compute = google.compute({
version: 'v1',
auth: authClient
});
var result = dataflow.projects.jobs.list({
'projectId': projectId,
'auth': authClient
}, function (err, result) {
console.log(err, result);
});
});
For posterity . . . there is a way to do this without a client library, but it requires generating a jwt from service account credentials and exchanging the jwt for an access token to execute a Dataflow template. This example uses the Cloud_Bigtable_to_GCS_Avro template:
import axios from "axios";
import jwt from "jsonwebtoken";
import mem from "mem";
const loadCredentials = mem(function() {
// This is a string containing service account credentials
const serviceAccountJson = process.env.GOOGLE_APPLICATION_CREDENTIALS;
if (!serviceAccountJson) {
throw new Error("Missing GCP Credentials");
}
const credentials = JSON.parse(serviceAccountJson.replace(/\n/g, "\\n").replace(/\r/g, "\\r").replace(/\t/g, "\\t"));
return {
projectId: credentials.project_id,
privateKeyId: credentials.private_key_id,
privateKey: credentials.private_key,
clientEmail: credentials.client_email,
};
});
interface ProjectCredentials {
projectId: string;
privateKeyId: string;
privateKey: string;
clientEmail: string;
}
function generateJWT(params: ProjectCredentials) {
const scope = "https://www.googleapis.com/auth/cloud-platform";
const authUrl = "https://www.googleapis.com/oauth2/v4/token";
const issued = new Date().getTime() / 1000;
const expires = issued + 60;
const payload = {
iss: params.clientEmail,
sub: params.clientEmail,
aud: authUrl,
iat: issued,
exp: expires,
scope: scope,
};
const options = {
keyid: params.privateKeyId,
algorithm: "RS256",
};
return jwt.sign(payload, params.privateKey, options);
}
async function getAccessToken(credentials: ProjectCredentials): Promise<string> {
const jwt = generateJWT(credentials);
const authUrl = "https://www.googleapis.com/oauth2/v4/token";
const params = {
grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
assertion: jwt,
};
try {
const response = await axios.post(authUrl, params);
return response.data.access_token;
} catch (error) {
console.error("Failed to get access token", error);
throw error;
}
}
function buildTemplateParams(projectId: string, table: string) {
return {
jobName: `[job-name]`,
parameters: {
bigtableProjectId: projectId,
bigtableInstanceId: "[table-instance]",
bigtableTableId: table,
outputDirectory: `[gs://your-instance]`,
filenamePrefix: `${table}-`,
},
environment: {
zone: "us-west1-a" // omit or define your own,
tempLocation: `[gs://your-instance/temp]`,
},
};
}
async function backupTable(table: string) {
console.info(`Executing backup template for table=${table}`);
const credentials = loadCredentials();
const { projectId } = credentials;
const accessToken = await getAccessToken(credentials);
const baseUrl = "https://dataflow.googleapis.com/v1b3/projects";
const templatePath = "gs://dataflow-templates/latest/Cloud_Bigtable_to_GCS_Avro";
const url = `${baseUrl}/${projectId}/templates:launch?gcsPath=${templatePath}`;
const template = buildTemplateParams(projectId, table);
try {
const response = await axios.post(url, template, {
headers: { Authorization: `Bearer ${accessToken}` },
});
console.log("GCP Response", response.data);
} catch (error) {
console.error(`Failed to execute template for ${table}`, error.message);
}
}
async function run() {
await backupTable("my-table");
}
try {
run();
} catch (err) {
process.exit(1);
}