Related
I'm trying to call my solana contract from my frontend app.
Even though I added the baseAccount. Solana complains it's not added. Whats going on?
const baseAccount = Keypair.generate()
async function createNFT({ nftPrice, nftRoyalty, nftInfo }) {
// console.log('image', image)
// const added = await client.add(image)
// console.log(
// '`https://ipfs.infura.io/ipfs/${added.path}`',
// `https://ipfs.infura.io/ipfs/${added.path}`
// )
const provider = await getProvider()
const program = new Program(idl, programID, provider)
const price = new BN(nftPrice * 10 ** 9)
const result = await program.rpc.mintNft(
wallet.publicKey,
price,
nftRoyalty,
wallet.publicKey,
nftInfo,
{
accounts: {
baseAccount: baseAccount.publicKey,
user: provider.wallet.publicKey,
systemProgram: SystemProgram.programId,
},
signers: [baseAccount],
}
)
console.log('result', result)
}
I inherited a script to manage a deploy of Salesforce code to multiple orgs in one go, to ensure all orgs are on the same version. The code is maintained in a Github respository and the final step is the update of the main branch, so the deploy therefore has to be successful for all orgs before it updates the main branch. Currently we have 32 orgs for which the deploys run simultaneously (with more to be added).
The final step after the code has deployed successfully is to check all the Salesforce to Salesforce connections and mappings, since all the orgs update a 'hub' org. It is in this step that I've started getting Puppeteer timeouts. Sometimes it completes, sometimes it fails. It seems to be getting worse in that I have to rerun it 2 or 3 times to get it pass without timing out. I'm not experienced in Node or Puppeteer or scripts like these so don't know how to stop this happening. I've tried increasing the timeout from the default 30000 to 90000 but even then it fails sometimes so that is not a solution, obviously.
Interestingly a few of us have also been having problems lately with Chrome being dreadfully slow and timing out just in the browser (we run on the latest version of Chrome) and I read that Puppeteer uses Chrome. I tried googling but haven't found anything that helps me hence posting this query here.
I would appreciate any help to sort this out because running it multiple times for each deploy is not a viable solution, especially with the length of time it takes to complete.
This is the function from where it sets the timeout.
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000); // added this but it still times out
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
And this is the called function where I believe the timeout happens:
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
This is the error thrown (after running 2hrs 40min!)It is close to the end of the process so has completed most of the org checks at this stage. It doesn't always fail in the same place or on the same org checks so the timeout is not related to a specific connection.
The full script is here:
import puppeteer, { Browser, Page } from "puppeteer";
import { flatten } from "lodash";
import yargs from "yargs";
import pAll from "p-all";
import {
loginAndGetConnections,
Connection
} from "../page-objects/sf2sf-home.page-object";
import {
getSubscribedObjects,
ConnectionObject
} from "../page-objects/sf2sf-connection.page-object";
import {
SubscribedField,
SubscribedFieldOption,
getSubscribedFields,
gotoObject,
selectMapping,
getPicklistValues,
save as saveSubscribedFields,
getPicklistMappingLinks,
savePicklistMapping
} from "../page-objects/sf2sf-subscribed-fields.page-object";
import { SClusterConfig } from "../s-cluster-config";
class Config {
configFile: string;
clusterConfigFile: string;
dryRun: boolean;
strict: boolean;
concurrency: number;
skipPicklists: boolean;
constructor() {
// eslint-disable-next-line #typescript-eslint/no-explicit-any
const argv: any = yargs
.scriptName("publish-connections")
.describe("config-file", "The file configuring the SF2SF sync.")
.alias("config-file", "c")
.default("config-file", "./sf2sf.config.json")
.describe("cluster-config-file", "The file configuring the SF2SF sync.")
.alias("cluster-config-file", "f")
.string("cluster-config-file")
.required("cluster-config-file")
.describe(
"dry-run",
"don't make any changes, just print what you're going to do."
)
.boolean("dry-run")
.default("dry-run", false)
.describe("strict", "Prevents associations from being unassigned")
.boolean("strict")
.default("strict", false)
.number("concurrency")
.default("concurrency", 10)
.describe("skip-picklists", "Skip assigning the picklists")
.boolean("skip-picklists")
.default("skip-picklists", false).argv;
this.configFile = argv["config-file"];
this.clusterConfigFile = argv["cluster-config-file"];
this.dryRun = argv["dry-run"];
this.strict = argv["strict"];
this.concurrency = argv["concurrency"];
this.skipPicklists = argv["skip-picklists"];
}
}
interface SubscribedFieldUpdate {
connection: Connection;
connectionObject: ConnectionObject;
connectionField: SubscribedField;
newValue?: SubscribedFieldOption;
}
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000);
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
(async (): Promise<void> => {
const argv = new Config();
const { clusterConfigFile, concurrency } = argv;
const clusterConfig = await SClusterConfig.fromPath(clusterConfigFile);
const browser = await puppeteer.launch({});
const connections = flatten(
await pAll(
clusterConfig.usernames.map(
(username) => (): Promise<Connection[]> =>
loginAndGetConnections(browser, username)
),
{ concurrency }
)
).filter((conn) => conn.isActive);
const differences: SubscribedFieldUpdate[] = [];
await pAll(
connections.map(
(connection) => (): Promise<void> =>
checkDifferencesForConnectionSafely(
argv,
browser,
connection,
differences
)
),
{ concurrency }
);
const result = differences.map(
({ connection, connectionObject, connectionField, newValue }) => ({
username: connection.username,
connection: connection.name,
object: connectionObject.name,
field: connectionField.name,
oldValue: (connectionField.value && connectionField.value.name) || "",
newValue: (newValue && newValue.name) || ""
})
);
console.log(JSON.stringify(result, null, " "));
await browser.close();
})();
fine people of Stack Overflow.
I'm trying to solve a problem I'm having involving twilio functions, messaging services, and databases.
What I'm attempting to do is send a message to all members of a database at once.
My code is a mess, as Javascript isn't my native language and I'm rather new to twilio.
The problem I believe I'm having is with the async/await feature of javascript.
Here is my code so far:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
allItems = res.items;
// continue fetching until last is not seen
while (res.last){
res = await db.fetch({}, {last: res.last});
allItems = allItems.concat(res.items);
}
}
// Function to get total number of contacts.
async function ReturnNumberOfContacts(allItems) {
number_of_contacts = allItems.length;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message) {
allItems.forEach(contact => {
let users_name = contact.name
client.messages
.create({
body: `Hey ${users_name}! ${message}`,
messagingServiceSid: messaging_service,
to: contact.key
})
});
}
// Function to submit response to broadcaster.
async function SuccessResponse(user_name, number_of_contacts) {
responseObject = {
"actions": [
{
"say": `${user_name}, your broadcast has successfully sent to ${number_of_contacts} contacts.`
},
{
"listen": true
}
]
}
}
// Main Function
exports.handler = async function(context, event, callback) {
// Placeholder for number of contacts
let number_of_contacts;
// Place holder for object from database of all contacts
let allItems;
// Placeholder for users message
let message;
// Placeholder for response to user
let responseObject;
//Twilio and Deta, Etc Const
const client = require('twilio')(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messaging_service = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
await FetchDB(db, allItems).then(ReturnNumberOfContacts(allItems));
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message = memory.twilio.collected_data.broadcast_message.answers.message_input.answer;
}
// Check if verified and set name.
const current_user = await db.get(from);
// Get the current users name or set a default value
let user_name = current_user.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
responseObject = {
"actions": [
{
"say": `${user_name}, you have canceled your request and no messages have been sent.`
},
{
"listen": false
}
]
}
// Return Callback and end task
callback(null, responseObject);
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(message, client, messaging_service)
.then(SuccessResponse(user_name, number_of_contacts))
return callback(null, responseObject);
}
// The user is not authorized so return this.
}
return callback(null, {
"actions": [
{
"say": "You are not authorized to broadcast."
},
{
"listen": false
}
]
})
};
So when the Fetch() function is triggered, I want the database to load a list of everyone and have twilio send them the desired message saved in the message variable. I have the code working so that I can read from the database and get the proper values, and send a single text message with the desired message, but the problem I'm having now is integrating it all together.
Thanks if anyone can point me in the right direction here.
Again, I'm new to javascript and more specifically asynchronous programming.
Twilio developer evangelist here.
The issue from the error says that allItems is undefined when you call ReturnNumberOfContacts.
I think the issue comes from trying to use allItems as a sort of global variable, same for number_of_contacts. It would be better for FetchDB to resolve with the list of items and ReturnNumberOfContacts to resolve with the number of the items.
You also have some arguments missing when you call SendMessages in your function. I've updated it to the point that I think it will work:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message, client, messagingService) {
return Promise.all(
allItems.map((contact) => {
let usersName = contact.name;
return client.messages.create({
body: `Hey ${usersName}! ${message}`,
messagingServiceSid: messagingService,
to: contact.key,
});
})
);
}
// Main Function
exports.handler = async function (context, event, callback) {
// Placeholder for users message
let message;
//Twilio and Deta, Etc Const
const client = require("twilio")(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messagingService = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message =
memory.twilio.collected_data.broadcast_message.answers.message_input
.answer;
}
// Check if verified and set name.
const currentUser = await db.get(from);
// Get the current users name or set a default value
let userName = currentUser.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
// Return Callback and end task
callback(null, {
actions: [
{
say: `${userName}, you have canceled your request and no messages have been sent.`,
},
{
listen: false,
},
],
});
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(allItems, message, client, messagingService);
return callback(null, {
actions: [
{
say: `${userName}, your broadcast has successfully sent to ${numberOfContacts} contacts.`,
},
{
listen: true,
},
],
});
}
// The user is not authorized so return this.
}
return callback(null, {
actions: [
{
say: "You are not authorized to broadcast.",
},
{
listen: false,
},
],
});
};
What I did here was change FetchDB to only take the db as an argument, then to create a local allItems variable that collects all the contacts and then returns them.
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
This is then called in the main body of the function to assign a local variable. I also replaced the ReturnNumberOfContacts function with a simple assignment.
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
One thing you may want to consider is how many contacts you are trying to send messages to during this function. There are a few limits you need to be aware of.
Firstly, Function execution time is limited to 10 seconds so you need to make sure you can load and send all your messages within that amount of time if you want to use a Twilio Function for this.
Also, there are limits for the number of concurrent connections you can make to the Twilio API. That limit used to be 100 connections per account, but it may vary these days. When sending asynchronous API requests as you do in JavaScript, the platform will attempt to create as many connections to the API that it can in order to trigger all the requests asynchronously. If you have more than 100 contacts you are trying to send messages to here, that will quickly exhaust your available concurrent connections and you will receive 429 errors. You may choose to use a queue, like p-queue, to ensure your concurrent connections never get too high. The issue in this case is that it will then take longer to process the queue which brings me back to the original limit of 10 seconds of function execution.
So, I think the above code may work in theory now, but using it in practice may have other issues that you will need to consider.
I'm having an issue getting a Drive ID when making a GET request.
First I'm creating the Team using "https://graph.microsoft.com/v1.0/teams". The request returns no response but a Status 202 accepted, the new Team is listed on Teams. I then extract the Team ID from Response Headers "Location".
Here is the failure, I make a request to https://graph.microsoft.com/v1.0/groups/{{TeamsID}}/drive
Here is my response back
{"error":{"code":"ResourceNotFound","message":"Resource is not found.","innerError":{"date":"2021-05-03T19:17:03","request-id":"75b4888d-de91-41aa-89ef-d983fcb5238d","client-request-id":"75b4888d-de91-41aa-89ef-d983fcb5238d"}}}
I make the same request on via postman and it works fine.
Here is my code
try {
let crt = await createTeam(msalData, data);
crt.headers.forEach( async (val, key) => {
if (key === "location") {
let id = val.split("/teams('")[1].split("')/operations")[0];
let gdd = await getDocumentDrive(msalData, id);
console.log(gdd)
}
});
} catch (err) {
console.log(err);
}
I can't figure out what I'm doing wrong.
Apparently the error (ResourceNotFound) occurs since the moment when Get Drive request is submitted the Group is not yet completely provisioned, documentations says the following in this regard:
If the group was created less than 15 minutes ago, it's possible for
the Create team call to fail with a 404 error code due to replication
delays. The recommended pattern is to retry the Create team call three
times, with a 10 second delay between calls.
Since you haven't revealed how exactly createTeam function is implemented, retry pattern come to the rescue here, for example:
async function withRetry(fn, maxRetry = 5, ms = 1000) {
try {
const val = await fn();
return val;
} catch (error) {
if (maxRetry) {
await new Promise((r) => setTimeout(r, ms));
return withRetry(fn, maxRetry - 1, ms);
} else throw new Error(`Max retries reached for function ${fn.name}`);
}
}
and now Teams could be created like this (note, in retry logic Get Group endpoint request is submitted to ensure it has been provisioned):
async function ensureTeam(client, teamName, defaultOwnerId) {
const teamResp = await createTeam(client, teamName, defaultOwnerId);
const teamloc = teamResp.headers.get("Content-Location");
return await withRetry(
async () => {
const groupUrl = teamloc.replace("teams", "groups");
return await client.api(groupUrl).get();
},
5,
10000
);
}
where createTeam could look like this:
async function createTeam(client, teamName, defaultOwnerId) {
const team = {
"template#odata.bind":
"https://graph.microsoft.com/v1.0/teamsTemplates('standard')",
displayName: teamName,
description: teamName,
members: [
{
"#odata.type": "#microsoft.graph.aadUserConversationMember",
roles: ["owner"],
"user#odata.bind": `https://graph.microsoft.com/v1.0/users('${defaultOwnerId}')`,
},
],
};
return await client.api("/teams").responseType(ResponseType.RAW).post(team);
}
Usage
const graphClient = Client.initWithMiddleware({ authProvider });
const group = await ensureTeam(graphClient,"Demo team",ownerId);
const driveUrl = `/groups/${group.id}/drive`;
const drive = await graphClient.api(driveUrl).get();
I want to monitor my dataflow jobs with an application. The application I'm developing is a nodejs application and ideally it would exist a package like #google-cloud/bigquery but for dataflow instead. I'm fully aware that I might not be able to start job, if it is not a template job, but it should be an easy way to list jobs or get job information.
Update:
I found this spec, https://dataflow.googleapis.com/$discovery/rest?version=v1b3, but I don't understand what location is for the list operation. The spec was linked from this page: https://cloud.google.com/dataflow/docs/reference/rest/
I did find the solution myself. There is a repo that basically has all the APIs for gcloud out there: https://github.com/google/google-api-nodejs-client
After I found that I could easily do what I wanted:
'use strict';
var google = require('googleapis');
var dataflow = google.dataflow('v1b3');
google.auth.getApplicationDefault(function (err, authClient, projectId) {
if (err) {
throw err;
}
// The createScopedRequired method returns true when running on GAE or a local developer
// machine. In that case, the desired scopes must be passed in manually. When the code is
// running in GCE or a Managed VM, the scopes are pulled from the GCE metadata server.
// See https://cloud.google.com/compute/docs/authentication for more information.
if (authClient.createScopedRequired && authClient.createScopedRequired()) {
// Scopes can be specified either as an array or as a single, space-delimited string.
authClient = authClient.createScoped([
'https://www.googleapis.com/auth/compute'
]);
}
// Fetch the list of GCE zones within a project.
// NOTE: You must fill in your valid project ID before running this sample!
var compute = google.compute({
version: 'v1',
auth: authClient
});
var result = dataflow.projects.jobs.list({
'projectId': projectId,
'auth': authClient
}, function (err, result) {
console.log(err, result);
});
});
For posterity . . . there is a way to do this without a client library, but it requires generating a jwt from service account credentials and exchanging the jwt for an access token to execute a Dataflow template. This example uses the Cloud_Bigtable_to_GCS_Avro template:
import axios from "axios";
import jwt from "jsonwebtoken";
import mem from "mem";
const loadCredentials = mem(function() {
// This is a string containing service account credentials
const serviceAccountJson = process.env.GOOGLE_APPLICATION_CREDENTIALS;
if (!serviceAccountJson) {
throw new Error("Missing GCP Credentials");
}
const credentials = JSON.parse(serviceAccountJson.replace(/\n/g, "\\n").replace(/\r/g, "\\r").replace(/\t/g, "\\t"));
return {
projectId: credentials.project_id,
privateKeyId: credentials.private_key_id,
privateKey: credentials.private_key,
clientEmail: credentials.client_email,
};
});
interface ProjectCredentials {
projectId: string;
privateKeyId: string;
privateKey: string;
clientEmail: string;
}
function generateJWT(params: ProjectCredentials) {
const scope = "https://www.googleapis.com/auth/cloud-platform";
const authUrl = "https://www.googleapis.com/oauth2/v4/token";
const issued = new Date().getTime() / 1000;
const expires = issued + 60;
const payload = {
iss: params.clientEmail,
sub: params.clientEmail,
aud: authUrl,
iat: issued,
exp: expires,
scope: scope,
};
const options = {
keyid: params.privateKeyId,
algorithm: "RS256",
};
return jwt.sign(payload, params.privateKey, options);
}
async function getAccessToken(credentials: ProjectCredentials): Promise<string> {
const jwt = generateJWT(credentials);
const authUrl = "https://www.googleapis.com/oauth2/v4/token";
const params = {
grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
assertion: jwt,
};
try {
const response = await axios.post(authUrl, params);
return response.data.access_token;
} catch (error) {
console.error("Failed to get access token", error);
throw error;
}
}
function buildTemplateParams(projectId: string, table: string) {
return {
jobName: `[job-name]`,
parameters: {
bigtableProjectId: projectId,
bigtableInstanceId: "[table-instance]",
bigtableTableId: table,
outputDirectory: `[gs://your-instance]`,
filenamePrefix: `${table}-`,
},
environment: {
zone: "us-west1-a" // omit or define your own,
tempLocation: `[gs://your-instance/temp]`,
},
};
}
async function backupTable(table: string) {
console.info(`Executing backup template for table=${table}`);
const credentials = loadCredentials();
const { projectId } = credentials;
const accessToken = await getAccessToken(credentials);
const baseUrl = "https://dataflow.googleapis.com/v1b3/projects";
const templatePath = "gs://dataflow-templates/latest/Cloud_Bigtable_to_GCS_Avro";
const url = `${baseUrl}/${projectId}/templates:launch?gcsPath=${templatePath}`;
const template = buildTemplateParams(projectId, table);
try {
const response = await axios.post(url, template, {
headers: { Authorization: `Bearer ${accessToken}` },
});
console.log("GCP Response", response.data);
} catch (error) {
console.error(`Failed to execute template for ${table}`, error.message);
}
}
async function run() {
await backupTable("my-table");
}
try {
run();
} catch (err) {
process.exit(1);
}