I have a couple of scripts that generate & gather large amounts of data that I will need both to seed my database with and in the future to add large amounts more to it. What is the best way to import lots of relational data into a rails database as both seed data and intermittently during production?
I haven't settled on an output format for my script yet but the data's structure largely mirrors my rails model's and contains has_many associations that I would like the import to preserve.
I've googled a fair bit and seen ar-extensions and seed_fu as well as the idea of using fixtures.
With ar-extensions all the examples seem to be staightforward csv imports (likely from table dumps which seems to be its primary use case) with no mention of handling associations or avoiding duplicate updates. In my case I have no id's, foreign keys, or join tables in my script so this seems like it wouldn't work for me unless I was prepared to handle that complexity myself.
With seed_fu, it looks like it could handle the relational aspects of the data creation but would still require me to specify ids (how can you know which ones are available in production?) and mix code with data.
Fixtures also have the same id problem though now it requires objects to be named (I'd probably just end up using numbers for names) and I am not sure how I would avoid accidental duplications of records.
Or would i be better off just putting my data into a local sqlite db first and then using the straight table dumping techniques?
I've done this before with CSV files. I have a cron job that collects the data and puts it in accessible CSV files. Then I have a rake task (also in cron, but on another box) that looks for CSVs, and if there are any, calls model methods to create objects out of the CSV rows.
The models have a csv_create_or_update method that takes a CSV row. This technique sidesteps the id issue, and also allows validations to run (or not) on the data coming in.
I was handed a rails app recently that required some imported data and it was all handled in a rake task. All the data came in form of csv formatted files and was handled per class/model etc as necessary. This worked out relatively well for me being new to the system as it was very easy to see where data was going and how it was being applied. As you import the data you can check for id conflicts/collisions and handle them accordingly.
Related
I have a Rails application where there are standard cities and areas rows that I want to have initialized in my tables in production and development. I have the data in separate files in array form.
Example cairo_areas.txt file has the following:
["Downtown - Abdin",
"Downtown - Abu El Rish",
"Downtown - Ahmed Helmy",
"Downtown - Ahmed Maher",
"Downtown - Gamea' El Banat"]
This is only part of the array of course to serve as an example.
Now I've tried to look for the best way to initialize in the database, but most of the answers I found seem very old.
In short the two approaches I found are:
Insert the data in seeds.rb. The downside is that most of the data there are for testing purposes, and I'm trying to separate the original data from the Faker data.
Approach 2 is to create a task to seed the city and area data into the database, but most of the answers I found are very old in older versions of Rails.
I want to know what is the best way to insert authentic production data into the database at initializing?
According to what Mark told you on his answer there is a third option which is widely used for adding data to production environments which is creating a migration that creates those records or modifies existing ones.
Two things must be said about the seeds.rb, this file should contain the minimal amount of data, for which according to the business logic of your system, makes the system functional. E.g, creation of roles, permissions and other information which your systems makes no sense without it. Also, this seeds.rb script should be idempotent, so you can re run the seeds.rb at any time without breaking the logic of the data. So you should use Model.find before doing Model.create basically. In that sense you can keep adding information to that seed file and re run it any time.
You might wonder how to choose over a migration or a rake task. I would suggest you to use migrations, because when used correctly you have the possibility to rollback those changes. Option that you do not have when using tasks.
Sorry if I was to verbose, I hope it helps !
Let's say for example I'm managing a Rails application that has static content that's relevant in all of my environments but I still want to be able to modify if needed. Examples: states, questions for a quiz, wine varietals, etc. There's relations between your user content and these static data and I want to be able to modify it live if need be, so it has to be stored in the database.
I've always managed that with migrations, in order to keep my team and all of my environments in sync.
I've had people tell me dogmatically that migrations should only be for structural changes to the database. I see the point.
My counterargument is that this mostly "static" data is essential for the app to function and if I don't keep it up to date automatically (everyone's already trained to run migrations), someone's going to have failures and search around for what the problem is, before they figure out that a new mandatory field has been added to a table and that they need to import something. So I just do it in the migration. This also makes deployments much simpler and safer.
The way I've concretely been doing it is to keep my test fixture files up to date with the good data (which has the side effect of letting me write more realistic tests) and re-importing it whenever necessary. I do it with connection.execute "some SQL" rather than with the models, because I've found that Model.reset_column_information + a bunch of Model.create sometimes worked if everyone immediately updated, but would eventually explode in my face when I pushed to prod let's say a few weeks later, because I'd have newer validations on the model that would conflict with the 2 week old migration.
Anyway, I think this YAML + SQL process works explodes a little less, but I also find it pretty kludgey. I was wondering how people manage that kind of data. Is there other tricks available right in Rails? Are there gems to help manage static data?
In an app I work with, we use a concept we call "DictionaryTerms" that work as look up values. Every term has a category that it belongs to. In our case, it's demographic terms (hence the data in the screenshot), and include terms having to do with gender, race, and location (e.g. State), among others.
You can then use the typical CRUD actions to add/remove/edit dictionary terms. If you need to migrate terms between environments, you could write a rake task to export/import the data from one database to another via a CSV file.
If you don't want to have to import/export, then you might want to host that data separate from the app itself, accessible via something like a JSON request, and have your app pull the terms from that request. That seems like a lot of extra work if your case is a simple one.
I must say that i currently use fixtures to populate my database. In an app i'm making, i will need to pre-populate the database with lots of data. I find that fixtures are a very nice way to describe this data, but there are some efficiency problems.
One important problem is managing the big yaml files. I think that it can get a bit overwhelming when i will have like 200 entries there.
Then, using something like Factories is not really to my liking, because it kinda messes data with code and i just want data representation to be available for easy changes.
Thus, i think of writing a small program to convert from csv to yaml and vice versa, in order to manage my entries through excel (i know that such a script exists already).
Do you know of another better way to do this sort of managing ? Notice that my data is not relevant with each other, meaning that a collection.each to populate is out of the question. Each entry is truly individual with lots of different attributes.
You can do all code you want directly in your db/seed.rb file. Inside you can add all script you want.
You can load a YAML file and save the return of this YAML or you can add your object in format you want.
After you just need call the rake task rake db:seed to launch this task
We are just giving MongoDB a test run and have set up a Rails 3 app with Mongoid. What are the best practices for inserting large datasets into MongoDB? To flesh out a scenario: Say, I have a book model and want to import several million records from a CSV file.
I suppose this needs to be done in the console, so this may possibly not be a Ruby-specific question.
Edited to add: I assume it makes a huge difference whether the imported data includes associations or is supposed to go into one model only. Any comments on either scenario welcome.
MongoDB comes with import/export tools that parse JSON formatted data.
Assuming you have an existing database in SQL, the easiest way to migrate that data is to output your SQL data as JSON strings, then use the import tool for each collection.
This includes denormalization and nesting/embedding - so don't migrate a relational model to MongoDB, you should consider also refactoring your data model to leverage MongoDB features.
For example, a common task is to merge articles and tags to an articles collection with the tags embedded as an array. Do that in your export script, so all MongoDB sees is nice clean JSON coming in through the import :-)
You can still import all your tables as collections, but you're missing out on some of the true strengths of MongoDB by doing that.
If you want add this dataset only one time. You can use the db/seed.rb file. you can read your CSV and generate all Document.
If you want made that a lot of times, you can made a runner or task.
With task, you need define a lib/task/file.rake and generate task with your file and again parse it and generate all documents.
You can made a runner too.
It's the same thing that a ActiveRecord stuff.
I've got multiple massive (multi gigabyte) datasets I need to import into a Rails app. The datasets are currently each in their own database on my development machine, and I need to read from them and create rows in tables in my Rails database based on the information they contain. The tables in my Rails database will not be exactly the same as the tables in the source databases.
What's the smartest way to go about this?
I was thinking migrations, but I'm not exactly sure how to connect the migration to the databases, and even if that is possible, is that going to be ridiculously slow?
without seeing the schemas or knowing the logic you want to apply to each row, I would say the fastest way to import this data is to create a view of the table you want to export in the column order you want (and process it using sql) and the do a select into outfile on that view. You can then take the resulting file and import it into the target db.
This will not allow you to use any rails model validations on the imported data, though.
Otherwise, you have to go the slow way and create a model for each source db/table to extract the data (http://programmerassist.com/article/302 tells you how to connect to a different db for a given model) and import it that way. This is going to be quite slow, but you could set up an EC2 monster instance and run it as fast as possible.
Migrations would work for this, but I wouldn't recommend it for something like this.
Since georgian suggested it, I'll post my comment as an answer:
If the changes are superficial (column names changed, columns removed, etc), then I would just manually export them from the old database and into the new, and then run a migration to change the columns.