In ormconfig.ts, it can set seeds to point to a path for all the files.
{
...
seeds: ['db/seed/*.ts'],
},
Is it possible to run only one file with an option like
$ npm run seed:run -f db/seed/file1.ts
In other words, if the data already been seeded into db, run it again will create duplicated data. How to avoid it?
I just encountered this. What I did was check first if the data exists (get the records first then loop to each record), if not, push the updates to an array. Then call insert command with the updates.
const updates = [];
const records = await getRecords();
records.foreach(record => {
if (!exists) updates.push(...);
});
await connection
.createQueryBuilder()
.insert()
.into(TableName)
.values(updates)
.execute()
Related
I am using the realtime database and I am using transactions to ensure the integrity of my data set. In my example below I am updating currentTime on every update.
export const updateTime = functions.database.ref("/users/{userId}/projects/{projectId}")
.onUpdate((snapshot) => {
const beforeData = snapshot.before.val();
const afterData = snapshot.after.val();
if (beforeData.currentTime !== afterData.currentTime) {
return Promise.resolve();
} else {
return snapshot.after.ref.update( {currentTime: new Date().getTime()})
.catch((err) =>{
console.error(err);
});
}
});
It seems the cloud function is not part of the transaction, but triggers multiple updates in my clients, which I try to avoid.
For example, I watched this starter tutorial which replaces :pizza: with a pizza emoji. In my client I would see :pizza: for one frame before it gets replaced with the emoji. I know, the pizza tutorial is just an example, but I am running into a similar issue. Any advice is highly appreciated!
Cloud Functions don't run as part of the database transaction indeed. They run after the database has been updated, and receive "before" and "after" snapshots of the affected data.
If you want a Cloud Function to serve as an approval process, the idiomatic approach is to have the clients write to a different location (typically called a pending queue) that the function listens to. The function then performs whatever operation it wants, and writes the result to the final location.
Snowflake provides Snowpipe to Copy data into a Table as soon as it is available in a Stage, but it misses the option of Purge.
Is there another way to achieve this?
There's no direct way to achieve Purge in case of Snowpipe but it can be achieved through the combination of Snowpipe, Stream and Task
Let's assume we have our Data files to be loaded residing in GCS Bucket
Step 1: Create a Snowpipe on Snowflake with an External stage
Refer to this Documentation
// Create a Staging Table
CREATE TABLE SNOWPIPE_DB.PUBLIC.GCP_STAGE_TABLE (COL1 STRING);
// Create Destination Table
CREATE TABLE SNOWPIPE_DB.PUBLIC.GCP_DESTINATION_TABLE (COL1 STRING);
// Create an External Stage
CREATE STAGE SNOWPIPE_DB.PUBLIC.GCP_STAGE
URL='gcs://bucket/files/'
STORAGE_INTEGRATION = '<STORAGE_INTEGRATION>';
// Create Snowpipe
CREATE PIPE SNOWPIPE_DB.PUBLIC.GCP_Pipe
AUTO_INGEST = true
INTEGRATION = '<NOTIFICATION_INTEGRATION>'
AS
COPY INTO SNOWPIPE_DB.PUBLIC.GCP_STAGE_TABLE
FROM #SNOWPIPE_DB.PUBLIC.GCP_STAGE;
Step 2: Create Stream on Table GCP_STAGE_TABLE
A stream records data manipulation language (DML) changes made to a table, including information about inserts, updates, and deletes.
Refer to this Documentation
// Create Stream in APPEND_ONLY Mode since we are concerned with INSERTS only
CREATE OR REPLACE STREAM SNOWPIPE_DB.PUBLIC.RESPONSES_STREAM
ON TABLE SNOWPIPE_DB.PUBLIC.GCP_STAGE_TABLE
APPEND_ONLY = TRUE;
Now, whenever some data is uploaded on the GCS Bucket, GCP_STAGE_TABLE is populated by Snowpipe and so is our Stream RESPONSES_STREAM
RESPONSES_STREAM Would look like this
COL1
METADATA$ACTION
METADATA$ISUPDATE
METADATA$ROW_ID
MOHAMMED
INSERT
FALSE
kjee941e66d4ca4hhh1e2b8ddba12c9c905a829
TURKY
INSERT
FALSE
b7c5uytba6c1jhhfb6e9d85e3d3cfd7249192b0d8
Since the Stream has APPEND_ONLY Mode, we will only see INSERT in METADATA$ACTION
Step 3: Create a Procedure to PURGE the stage and Populate GCP_DESTINATION_TABLE
// Create a Procedure
CREATE OR REPLACE Load_Data()
RETURNS VARCHAR
LANGUAGE JAVASCRIPT
AS
$$
var purgeStage = `REMOVE #SNOWPIPE_DB.PUBLIC.GCP_STAGE`;
var populateTable = `INSERT INTO SNOWPIPE_DB.PUBLIC.GCP_DESTINATION_TABLE
SELECT * FROM RESPONSES_STREAM`;
try {
snowflake.execute ( {sqlText: purgeStage} );
snowflake.execute ( {sqlText: populateTable} );
return "Succeeded.";
}
catch (err) {
return "Failed: " + err;
}
$$
The above Procedure uses REMOVE command to purge the Stage and Populates the Table GCP_DESTINATION_TABLE.
Populating the Table GCP_DESTINATION_TABLE from the Stream RESPONSES_STREAM Clears the Stream.
Step 4: Create a Task to call the Procedure Load_Data()
Refer to this Documentation
We create a Task with a 5 mins interval which 1st checks the Stream RESPONSES_STREAM for any Data loaded to GCP_STAGE_TABLE, if True, executes a Procedure Load_Data()
// Task DDL
CREATE OR REPLACE TASK MASTER_TASK
WAREHOUSE = LOAD_WH
SCHEDULE = '5 MINUTE'
WHEN SYSTEM$STREAM_HAS_DATA('SNOWPIPE_DB.PUBLIC.RESPONSES_STREAM') //Checks the stream for Data
AS
CALL Load_Data();
SYSTEM$STREAM_HAS_DATA('RESPONSES_STREAM') evaluates to True when Data is loaded to GCP_STAGE_TABLE, this then makes the Task execute the Procedure call.
Even though the Procedure is not called every 5 mins, it's worth noting that WHEN SYSTEM$STREAM_HAS_DATA('RESPONSES_STREAM') does consume a minute Compute resource, to reduce this, the frequency can be changed from 5 mins to a greater duration.
To make this an ELT task, Procedure can have some transformation logic and a Tree of Tasks can be made.
Note:
REMOVE is not officially supported for External stages but it still worked for GCS Bucket for me.
Let me know if it works for you in the case of AWS S3 and Azure.
I have React Native app with SQL database. I want to clear all local data at a certain place (for example: after log out I want to clear all the local database). I have multiple tables so I am not sure if there is any particular way to clear all data in one go.
i am using React Native SQLite Storage
Please suggest the best way to delete all local databases.
Here is one way to achieve this. You can list out all the tables and then discard tables you do not want to reset and then run the delete command.
let [results] = await this.db.executeSql('SELECT name FROM sqlite_master WHERE type="table" ORDER BY name');
var len = results.rows.length;
for (let i = 0; i < len; i++) {
let tableName = results.rows.item(i).name;
if (tableName !== 'sqlite_sequence' && tableName !== 'android_metadata') {
await this.db.executeSql(`DELETE FROM ${results.rows.item(i).name}`);
}
}
I am not sure what if anything I am doing wrong or if this is a bug.
My IndexedDB exists and has data in it. I am currently using alasql to query the data AFTER i retrieve it from IndexedDB, however I would much rather do this in a singe step as this make joins and other queries across multiple tables much simpler.
Here is my code:
const queryDB = async (dbName, query) => {
const result = await alasql.promise([
`CREATE INDEXEDDB DATABASE IF NOT EXISTS ${dbName};`,
`ATTACH INDEXEDDB DATABASE ${dbName};`,
`USE ${dbName};`,
]);
console.log(result);
return await alasql.promise(query);
};
(also tried without semicolon...)
the first call to this function fails with 'Error: Database ... does not exist' where ... is the dbName
while the second one succeeds and returns the value from the query.
I have been trying just about everything I can everything I can think of and the only working solution is using callbacks (as in the example on the wiki) which is NOT what I want to do.
Oddly enough this work:
const queryDB = async (dbName, query) => {
await alasql.promise(`ATTACH INDEXEDDB DATABASE ${dbName};USE DATABASE ${dbName};`);
return await alasql.promise(query);
};
Recently, a program that creates an Access db (a requirement of our downstream partner), adds a table with all memo columns, and then inserts a bunch of records stopped working. Oddly, there were no changes in the environment that I could see and nothing in any diffs that could have affected it. Furthermore, this repros on any machine I've tried, whether it has Office or not and if it has Office, whether it's 32- or 64-bit.
The problem is that when you open the db after the program runs, the destination table is empty and instead there's a MSysCompactError table with a bunch of rows.
Here's the distilled code:
var connectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=corrupt.mdb;Jet OLEDB:Engine Type=5";
// create the db and make a table
var cat = new ADOX.Catalog();
try
{
cat.Create(connectionString);
var tbl = new ADOX.Table();
try
{
tbl.Name = "tbl";
tbl.Columns.Append("a", ADOX.DataTypeEnum.adLongVarWChar);
cat.Tables.Append(tbl);
}
finally
{
Marshal.ReleaseComObject(tbl);
}
}
finally
{
cat.ActiveConnection.Close();
Marshal.ReleaseComObject(cat);
}
using (var connection = new OleDbConnection(connectionString))
{
connection.Open();
// insert a value
using (var cmd = new OleDbCommand("INSERT INTO [tbl] VALUES ( 'x' )", connection))
cmd.ExecuteNonQuery();
}
Here are a couple of workarounds I've stumbled into:
If you insert a breakpoint between creating the table and inserting the value (line 28 above), and you open the mdb with Access and close it again, then when the app continues it will not corrupt the database.
Changing the engine type from 5 to 4 (line 1) will create an uncorrupted mdb. You end up with an obsolete mdb version but the table has values and there's no MSysCompactError. Note that I've tried creating a database this way and then upgrading it to 5 programmatically at the end with no luck. I end up with a corrupt db in the newest version.
If you change from memo to text fields by changing the adLongVarWChar on line 13 to adVarWChar, then the database isn't corrupt. You end up with text fields in the db instead of memo, though.
A final note: in my travels, I've seen that MSysCompactError is related to compacting the database, but I'm not doing anything explicit to make the db compact.
Any ideas?
As I replied to HasUp:
According MS support, creation of Jet databases programmatically is deprecated. I ended up checking in an empty model database and then copying it whenever I needed a new one. See http://support.microsoft.com/kb/318559 for more info.