I have written a migration in nestjs which iterates through some data in a for each loop. Each loop iterations saves entry in the database. The problem is when an error occurs, migration deletes all the entries in the database it has added. How can I prevent that? I don't want the database to roll back to the previous state but keep all the entries that have been added. I a, using typeorm
Related
Here is the problem:
I have Ruby on Rails project that has a table that have almost 100k rows and I have a binary column and I want to make changes in the content data on this column.
So I must iterate over those 100k rows, making changes on that row on particular column, saving it back on database.
But I must keep track of changes because these changes could fail and I should have someway to re-start data change from where I stopped.
Here is what I thought of a way of doing it:
Create a Migration to have a table MigrationTrack to track all records that have being migrated
Create a model of the above migration
Create a rake task that grabs all 100k from TableToUpdate and iterate over them, saving data back to row and save its ID on MigrationTrack. Create a logic to have a join on TableToUpdate and MigrationTrack to filter only ids that I haven't updated yet
After above migration finished create another migration to drop MigrationTrack table and remove its model.
Is there any other "Railsh way" to do that? Anyone have done such change?
Thanks
I would do it like this:
Add and deploy a migration adding a new column with the desired data type to the database table.
Add code to your model that save the value from the old column into the new column too.
Run a rake task or a simple one-liner in the console that touches all records to make sure the code introduced in step one ran on each record.
After this step, you can manually verify if all records in the database have both columns set as expected.
Switch using the new attribute instead of the old attribute in the code.
Drop the old column.
For simple cases, try running a simple view to check how it will turn out to be, for example, if your migration is
change_column :table, :boolean_field, 'integer USING CASE boolean_field THEN ...'
then you try do a simple select query with your cast, if you need more safey, you can create 'up' and 'down' methods on your migrations, then you can create a backup table on up, and on down, you can revert the values
The EF7 migration add command (to date, beta5) compares the model classes defined in the DbContext against the current model snapshot, creates a new migration class, and updates the model snapshot.
I need to modify the migration to make it generate different DDL SQL. As an example, EF7 uses sequences for SQL Server auto-increment values, and I would like it to use identity. However it could be any other reason. The migration remove command would physically delete the migration files and revert the model snapshot, so it's useless in this case.
There are 3 files that contain related code that look like they need to be edited:
The primary migration class: The Up and Down methods should be modified.
The DbContextModelSnapshot file contains Annotations that need to be modified
The secondary migration partial class: The badly named [migration].Designer.cs file also contains a model snapshot for the migration. I'm assuming this snapshot needs to match the model snapshot in item 2, but am not certain. The only information I have about the purpose of it is from Brice's blog , which says "It’s there in case you or a provider need to inspect the model for additional information during a migration."
Specific questions:
Do the two model snapshots need to be kept in sync in order to correctly perform migrations?
Is modifying 3 separate files the only way to edit the migration? (Although depending on the changes, the model snapshots may not have to be touched in some cases.)
Is there some EF command that would regenerate just the model snapshots, but not the migration methods?
Specific answers:
Do the two model snapshots need to be kept in sync in order to correctly perform migrations?
No, the snapshot in the migration is a last resort for provider writers. For example, SQLite can't rename columns so it could use migration's model snapshot to do a table rebuild for this operation. 99% of the time, it won't ever be used.
Is modifying 3 separate files the only way to edit the migration?
Most of the time, you should only ever edit the main migration file. In rare cases, you may need to edit the model snapshot if you're working in a team environment and you encounter a merge conflict. You can ignore the designer file; it just captures some metadata about the migration.
You may not have to edit anything if you configure your model correctly. For example, to use identity instead of sequences, override DbContext.OnModelCreating() and add modelBuilder.ForSqlServer().UseIdentity().
Is there some EF command that would regenerate just the model snapshots, but not the migration methods?
No, it shouldn't be needed since you almost never edit these files.
We use Sunspot Solr for indexing and searching in our Ruby on Rails application.
We wanted to reindex some objects and someone accidentally ran the Product.reindex command from the Rails Console. The result was that indexing of all products started from scratch and our catalogue appeared empty while indexing was taking place.
Since we have a vast amount of data the reindexing has been taken three days so far. This morning when I checked on the progress of the reindexing, it seems like there was one corrupt data entry which resulted in the reindexing stopping without completing.
I cannot restart the entire Product.reindex operation again as it takes way too long. Is there a way to only run reindexing on selected products? I want to select a range of products that aren't indexed and then just run indexing on thise. How can I add a single product to the index without having to run a complete reindex of entire data set?
Sunspot does index an object in the save callback so you could save each object but maybe that would trigger other callbacks too. A more precise way to do it would be
Sunspot.index [post1, post2]
Sunspot.commit
or with autocommit
Sunspot.index! [post1, post2]
You could even pass in object relations as they are just an array too
Sunspot.index! post1.comments
I have found the answer on https://github.com/sunspot/sunspot#reindexing-objects
Whenever an object is saved, it is automatically reindexed as part of the save callbacks. So all that was needed was to add all the objects that needed reindexing to an array and then loop through the array, calling save on each object. This successfully updated the required objects in the index.
In my project, i have a situation like when user runs the application.
The system should insert some values in a table which is used allover the application.
(These value should be inserted only once when the project is executed at first time)
I am trying to find out if there is any initialization function like Constructors in Rails.
I tried to use the config/application.rb, but i am not sure its the right way to do this.
Please suggest.
Thanks
If you looking for inserting some default dictionary data like month names etc you should look into seed.rb file or even better consider using seed_fu gem (https://github.com/mbleigh/seed-fu)
Yes you can insert/edit/delete records into table with migration :
1) Create the migration .
2) Run db query inside the execute. like :
execute "insert into users (name, role) values ('vik', 'admin')"
3) After all the insertion operation run the migration.
How if you update have boolean field or any kind of integer field to maintain status in your application. And for the very first time, user runs the application, your code will insert necessary values for that user in db and update boolean/status field and will be cached(for better performance only rather than fetching value from db every time). However after every time cache is cleared, it will send the query to db; but fetching boolean value(checking user status) is more faster than checking all inserted values for that user.
Yesterday i was absolutely sertain that all migrations data for EF placed in classes, placed in my solution as nested from DbMigration. But today i was dig a slightly deeper(just try to fallback to old migration with enable data loss not with nu-get and visual studio, but with code())
DbMigrator fg = new DbMigrator(new Settings() { AutomaticDataLossEnabled = true});
fg.Update("MigrationName");
And get exception, smth like "string should be truncated", those means that migrator tried to update column from big to small MaxLength attribute. So, i had excluded migration that caused this update and move this changes to migration, those create tables. The error still was occured. I got to intellitrace and it said that those(deleted) migration still was called. Looking to requests told me things like this:
SELECT [Extent1].[MigrationId] AS [MigrationId] FROM [dbo].[__MigrationHistory] AS [Extent1]
Looking to a table __MigrationsHistory and get my deleted migration there with field model that contains crypted data(don't decrypt this yet) . I was realy shocked. Does this means that all code, have written in classes is just the fake and really executed code placed here? And does anyone know, how to work with this table, register projections of migration classes to it etc. Or the once way to work with migrations is nu-get console?
I am not fully sure what your primary question is, so I will first try to answer last part about __MigrationHistory table.
Code in classes is not fake, your code in classes is compiled and run.
This table, however, really contains your database model, but it is not encrypted, it is compressed. The reason why Migrations API needs to store your model, is to be able to compare it against your current actual model and track changes for you (for example when you add a new property it will be able to tell what property you added and to perform automatic db migration).
In previous version of EF there was an EdmMetadata table where hash of your model was stored, and EF was able to detect if you made some changes to model by comparing stored and current model hash value. New version when migrations are enabled stores entire model as compressed blob, so it can do diff between the model that was used to create database and current model you are using, and make automatic migrations accordingly.
You should not work directly with this table, it is automatically populated by migrations API, but nuget console is not the only way to do migrations, you can check this resource for some insights how to do it from code.
Now, regarding your question from question title (where they are stored?), migrations are stored in code, in a class inheriting from DbMigration class that migrations API creates for you when you do Add-Migration command in nuget console. When you perform an migration (Update-Database), either from nuget package manager console or from code, API will compare your current model with versions in __MigrationsHistory to find initial version (if you have not specified it) and perform all migrations in between initial and target version (if not specified otherwise target is latest version).
I'm not really clear how you did exclude your migration that causes problems, as you need to migrate your database to version before that migration, and then delete and recreate all subsequent migrations from there.
Maybe you could solve your fallback to old version problem by implementing public override void Down() method in your migration that causes problems when trying to rollback? This method can be used to execute code which performs inverse any operations for migration.
Not directly related to question but worth mentioning, there is also pretty detailed tutorial here for EF CF.