The (occasionally very large) output of some JSON-calls get written into my database to speed things up a little; the app gets the JSON-output pre-rendered from the database.
Every once there's a little change in the content so the output would have been different. I'm looking for a rails migration that I could just include in my deployment for clearing out the specific table storing my JSON code.
You can put that in your migration. Warning, this will delete all data from the table !
Model.connection.execute('DELETE FROM table_name');
Or altenatively you could run some code every day (crontab) to delete everything that is X months old :
def self.delete_old_stuff
month = 3
Model.where("updated_at < ?" , month.months.ago).each{|c| c.destroy}
end
You can call that through the runner command of rails or integrate that a in rake task... I use a similar method to clear old basket in a small ecommerce app.
Related
I've some .json files with data that are automatically updated from time to time. I also have a Ruby on Rails app where I want to insert the information that's in those files.
By now, I'm parsing the JSON and inserting the data in the database in the seeds.rb file, but I'll want to add more data without having to restart the app, I mean, on the go.
From time to time, I'll check those files and if they have modifications, I want to insert those new items into my database.
What's the best way to do it?
Looks like a job for a cron.
Create the code you need in a rake task (in /lib/tasks):
task :import, => :environment do
Importer.read_json_if_modified # your importer class
end
Then run this with the period you want using your system's cron.
I have a question regarding my migrations in rails.
Normally when i want to add a colum to a model for i dont make extra migrations but instead i perform this steps:
rake db:rollback
next i change the migration file in db/migrations and rerune:
rake db:migrate
The biggest problem is that when i do this i loose my data.
Previous i wrote migrations from the command line with for example
rake g migration Add_Column_House_to_Users house:string
The problem with this approach is that my db/migrations folder afterwards get very large and not very clear! I mean at the end i dont know wich variables the object has! Im not an expert in rails and would like to ask you how to keep the overview over the migrations!Thanks
Just a minor thought - I just use the file db/migrate/schema.rb to determine whats in the database as opposed to tracking through the migrations
You definitely shouldn't use db:rollback with a table with existing data.
I have a few production RonR apps with a ton of data and there are 100+ entries in the migrations table and adding new migrations to tweak tables is the rails way to do things. Not sure what you mean by lucid, but your schema and data model are going to change over time and that is ok and expected.
One tip. The migrations are great, but they are just the beginning, you can include complex logic as needed to fix your existing data (like so)
Changing data in existing table:
def up
add_column :rsvps, :column_name_id, :integer
update_data
end
def update_data
rsvps = Rsvp.where("other_column is not null")
rsvps.each do |rsvp|
invite = Blah.find(rsvp.example_id)
...
rsvp.save
end
end
Another tip: backup your production database often (should do this anyway), but use it to test all of your migrations before deploying. I run scripts like this all the time for local testing:
mysql -u root -ppassword
drop database mydatabase_dev;
create database mydatabase_dev;
use mydatabase_dev;
source /var/www/bak/mydatabase_backup_2013-10-04-16.28.06.sql
exit
rake db:migrate
EDIT: I have totally rewritten this question for clarity. I got no comments and no answers earlier.
I am maintaining a 2.x Rails app with plenty of statistical data. Some data is real and some is estimated for the future years. Every year I need to update estimated data with real data and calculate new estimates.
I have been using BIG yml-files and migrations for loading the data into the app every year. My migrations are full of estimation calculations and data corrections.
Problem
My migrations are full of none-schema related material and I can't even dream of doing db:migrate:reset without waiting few hours (if it even works). I'd love to see my migrations nice and clean - only with schema related modifications. But how I am suppose to update the data every year if not using migrations?
Help needed
I'd like to hear your comments and answers. I'm not looking for a silver bullet - more like best practises and ideas how people are handling similar situation.
It sounds like you have a large operation (data load using yml files) once a year but smaller operations once a month.
From my experience with statistical data you will probably end up doing more and more of these operations to clean and add more data.
I would use a job processing framework like resque and resque scheduler.
You can schedule the jobs to run once a month, year, day or constantly running. A job is something like loading yml files (or sets of yml files) or cleaning up data. You can control parameters to send to your job so you can use one class but alternate how it updates or cleans your data based on the way you enqueue or schedule the job.
First of all, I have to say that this is a very interesting question. As far as i know, it isn't a good idea loading data from migrations. Generally speaking you should use db/seeds.rb for data loading in your db and I think it could be a good idea to write a little class helper to put in your lib dir and then call it from db/seeds.rb. I image you could organize you files in the following way:
lib/data_loader.rb
lib/years/2009.rb
lib/years/2010.rb
Obviously, you should clear your migrations and write code for lib/data_loader.rb in the way you should prefer but I was only trying to offer a general idea of how I'd organize my code if I have to face a problem like that.
I'm not sure I've replied to your question in a way that helps but I hope it does.
If I were you I would go with creating custom rake task. You will have access to all you models and activerecord connections and once a year you will end up doing:
rake calculate
I have a situation where I need to load data from CSV files that change infrequently and update data from the Internet daily. I'll include a somewhat complete example on how to do the former.
First I have a rake file in lib/tasks/update.rake:
require 'update/from_csv_files.rb'
namespace :update do
task :csvfiles => :environment do
Dir.glob('db/static_data/*.csv') do |file|
Update::FromCsvFiles.load(file)
end
end
end
The => :environment means we will have access to the database via the usual models.
Then I have code in the lib/update/from_csv_files.rb file to do the actual work:
require 'csv'
module Update
module FromCsvFiles
def FromCsvFiles.load(file)
csv = CSV.open(file, 'r')
csv.each do |row|
id = row[0]
s = Statistic.find_by_id(id)
if (s.nil?)
s = Statistic.new
s.id= id
end
s.survey_area = row[1]
s.nr_of_space_men = row[2]
s.save
end
end
end
end
Then I can just run rake update:csvfiles whenever my CSV files changes to load the new data. I also have another task that is set up in a similar way to update my daily data.
In your case you should be able to write some code to load your YML files or make your calculations directly. To handle your smaller corrections you could make a generic method for loading YML files and call it with specific files from the rake task. That way you only need to include the YML file and update the rake file with a new task. To handle execution order you can make a rake task that calls the other rake tasks in the appropriate order. I'm just throwing around some ideas now, you know better than me.
How does one create a Rails migration properly so that a table gets changed to MyISAM in MySQL? It is currently InnoDB. Running a raw execute statement will change the table, but it won't update db/schema.rb, so when the table is recreated in a testing environment, it goes back to InnoDB and my fulltext searches fail.
How do I go about changing/adding a migration so that the existing table gets modified to MyISAM and schema.rb gets updated so my db and respective test db get updated accordingly?
I didn't find a great way to do this. You could change your schema.rb like someone suggested and then run: rake db:schema:load, however, this will overwrite your data.
The way I did it was (assuming you are trying to convert a table called books):
Save the existing data from the CLI: CREATE TABLE tmp SELECT * FROM books;
In your new migration file, drop the books table and recreate it with :options => "ENGINE=MyISAM" like someone said in the comment
Copy the contents back: INSERT INTO books SELECT * FROM tmp
i think that if you change your schema format (config.active_record.schema_format) from :ruby to :sql, all sql will be saved there.
i'd do some tests on a fresh app first if i were you, see how it works.
You can run any sql in migrations. This worked for me:
class ChangeMapOnlyUsersEngine < ActiveRecord::Migration[5.1]
def change
MyModel.connection.execute("ALTER TABLE my_models ENGINE = 'MyISAM';")
end
end
When I did this in the other direction (InnoDB -> MyISAM) it worked fine, without loss of data so I don't think it's neccesary to create temporary tables or similar. Note that MyISAM doesn't support transactions, so any tests against the database for a corresponding ActiveRecord model will be persisted, with a risk of test pollution.
I have a bunch of rails models that I'm re-writing into a single model to simplify my code and reduce unnecessary tables.
I'm wondering what the best way to delete a model class and its table is. I want past migrations to still succeed, but I don't want to leave the empty models lying around.
Do I have to manually delete the old migrations that reference these models, then manually delete the class files?
Does anyone have any tips for the best way to do this?
All in one solution.
Run the following commands:
rails destroy model ModelName
rails g migration DropTableModelName
The former will generate a new migration file which should looks like this:
class DropTableModelName < ActiveRecord::Migration
def change
drop_table :model_name
end
end
Now run db:migrate and you're done.
If you'd like to completely get rid of of a model and its table do this:
rails destroy model Name
The question is a bit stale now, but I just did:
rails destroy scaffold <ModelName> -p
The -p flag shows "pretend" output, which is good for seeing what will happen. Remove the '-p' flag and the results will match the output. This cleaned the entire collection of M-V-C files + testing + js files + the original migration, no problem.
I guess if you are one who likes to hand edit your migrations and include multiple steps in each, losing the original migration could break db:setup, so buyer beware. Keeping one action == one migration file should avoid this potential snafu.
What about doing ruby script/destroy model? That should take care of the model and the migration.
Depending on how far into development or production you are, you may want to migrate the models out safely using a migration to remove/backup data or what not. Then as bobbywilson0 suggested, using
script/destroy model
or if you rspec anything
script/destroy rspec_model
This will remove any spec tests as well.
Or you can always just drag them to the trash folder.
You can take a look at this one at rails guide.
But I suggest, if it is possible, you should delete the models and all references to the models. This will probably save time later as you don't need to maintain the dead code in the codebase.
If you'd rather have a manual based answer:
First run the following command to identify which migrations you want removed:
rake db:migrate:status
Feel free to grep -i on it too if you're confident of your naming scheme.
Make note of all the "add x to model name" and similar alterations to your Model. These can be removed using:
rails d migration AddXToModelName
Do this for every migration besides the initial create migration. The following command will take care of the initial create migration and the files associated with the model:
rails d model ModelName