I have a json file with a lot of movies in it. I want to create a model 'Movie' and fill it with all movies from that json file. How do i do it? I know that I can parse json file into a hash but that's not the thing I am looking for.
The correct term you're looking for is "seeding"!
You're going to need a database however, and a migration to create the database along with the associated movies table. (There are plenty of guides on how to do this, along with the official documentation).
After that's done, you'll need to "seed" your database with the data in your json file.
In the seeds.rb file, assuming that the JSON file is an array of Movies in JSON form, you should be able to loop over every Movie JSON object and insert it into your database.
To add to docaholic's helpful response, here's some steps/pseudo-code that may help.
Assuming you're using a SQL database and need to create a model:
# creates a migration file.
rails generate migration create_movies title:string #duration_in_minutes:integer or whatever fields you have
# edit the file to add other fields/ensure it has what you want.
rake db:migrate
Write a script to populate your database. There are many patterns for this (rake task, test fixtures, etc) and which one you'd want to use would depend on what you need (whether it's for testing, for production environment, as seed data for new environments, etc).
But generally what the code would look like is:
text_from_file = File.read(file_path)
JSON.parse(text_from_file).each do |json_movie_object|
Movie.create!(title: json_movie_object[:title], other_attribute: json_movie_object[:other_attribute])
# if the json attributes exactly match the column names, you can do
# Movie.create!(json_movie_object)
end
This is not the most performant option for large amounts of data. For large files you can use insert_all for much greater efficiency, but this bypasses activerecord validations and callbacks so you'd want to understand what that means.
For my case, I need to seed around 200 hundred datas for production from a JSON file, so I tried to insert data from the json files in a database.
SO at first I created a database in rails project
rails generate migration create_movies title:string duration_in_minutes:integer
# or whatever fields you have
# edit the file to add other fields/ensure it has what you want.
rake db:migrate
Now its time to seed datas!
Suppose your movies.json file has:
[
{"id":"1", "name":"Titanic", "time":"120"},
{"id":"2", "name":"Ruby tutorials", "time":"120"},,
{"id":"3", "name":"Something spetial", "time":"500"}
{"id":"4", "name":"2HAAS", "time":"320"}
]
NOW, like this 4 datas, think your JSON file has 400+ datas to input which is a nightmare to write in seed files manually.
You need to add JSON gem to work. Run
bundle add json
to work with JSON file.
In db/seed.rb file, add these lines to add those 400+ infos in your DATABASE:
infos_from_json = File.read(your/JSON/file/path/movies.json)
JSON.parse(infos_from_json).each do |t|
Movie.create!(title: t['name'], duration_in_minutes:
t['time'])
end
Here:
infos_from_json variable fixing the JSON file from file directory
JSON.parse is calling the JSON datas then the variable is declared to indicate where the file is located and each do loop is dividing each data and |t| will be used to add part JSON.parse(infos_from_json). in every call.
Movie.create!(title: t['title'], duration_in_minutes: t['time']) is adding the data in database
This is how we can add datas in our database from a JSON file easily.
For more info about seeding data in database checkout the documentation
THANKS
Edit: This operation requires JSON gem just type bundle add json
Related
I have an issue re-creating my app DB from JSON backup file.
What I am trying to do now is to create all the records from rake task reading lines in the file and rendering json like:
new_record = key.camelize.constantize.new(hs['key'])
new_record.save
The problem is that I can not skip callbacks for all the models, to be sure I am not creating any dup stuff there. .camelize.constantize.skip_callback(:after_create) is just not working giving me an error undefined method '_after_create_callbacks' for #<Class:.
So two questions here:
1) Is there any way to skip AR callbacks other way?
2) Are there any other options for re-creating db from JSON except SQL queries?
1. Skipping callbacks
You can ask AR to skip validation callbacks when changing a record like this:
new_record.class.skip_callback(:create, :after, :specific_callback)
new_record.save
2. Importing JSON into PSQL
Since your data is in JSON, there is no magical way of importing it into your DB. But, Postgres allows you easily import it using SQL like it is shown in this SO answer. Also, many others have written utilities to make your life easier.
I have a ruby script written which takes a CSV file and converts the input into a hash:
Culper = File.open('.\CulperCSV.csv')
culper_hash = {}
# set up culper code hash from provided CSV
CSV.foreach(Culper) do |row|
number, word = row
culper_hash[word] = number
end
and I am trying to make a Rails app using the script.
My question: How do I store the Hash persistently (or the CSV data so I can build the hash) so that I can minimize load times?
My thoughts:
1) Load the CSV data into a database (seed it) and each time I get a visitor on my site, do the above assignment into a hash but from the db. (not sure how to do this but I can research it).
or
2) Load the complete hash into the database (I think I would have to serialize it?) so that I can do just one fetch from the db and have the hash ready to go.
I am very new to building apps, especially in Rails so please ask questions if what I am trying to do doesn't make sense.
I suggest you should go with 2nd approach. Here are steps to do that:
Setup new app:
rails new app_name
bundle install
rake db:create
Create Model:
rails g model model_name column_name:text
rake db:migrate
Open model_name.rb file and add the following line
serialize :column_name
Now all sets. Just run the your script to parse .csv file and store hash in db. Your column is now able to store the hash.
Culper = File.open('.\CulperCSV.csv')
# get the object from database
obj = ModelName.first
# set up culper code hash from provided CSV
CSV.each(Culper) do |row|
number, word = row
obj.column_name[word] = number
end
obj.save
Your .csv file seems to be already in your Rails app directory, so load times shouldn't be bad (unless it's really big). However, if that file isn't going to change and you need only a small piece of it at a time, then I would store that in your database.
Create a model/migration that corresponds to the data you have in the .csv file and then (after migrating the migration) run a script to parse the data from your .csv file to your database.
I managed to solve my problem following the advice of #Kh Ammad: setting up a new app, creating a model for it, and marking my column serializable.
However, I had some problems with running the script to populate the model with the hash so instead, after some research, I created the rake task below:
#lib/tasks/import.rake
require 'csv'
task :import, [:filename] => :environment do
culper_hash = {}
Culper = File.open('.\CulperCSV.csv')
CSV.foreach(Culper) do |row|
number, word = row
culper_hash[word] = number
end
# culper_hash == column_name
obj = CulperDict.create(culper_hash: culper_hash)
obj.save
end
and ran it with:
$ bundle exec rake import
and my model contained the entire hash table in one entry!
I used this article to figure out how to run a rake task:
http://erikonrails.snowedin.net/?p=212
Specifically, the last comment on the page by Lauralee (posted on December 20th, 2012 at 8:47 am) who ran into a similar problem.
Hi i probably have a simple question.
I have this database with moodboard templates.
I want to transport it to the server with capistrano but in my seeds.rb file there is only all the seeds and if i run them again a lot of data gets inserted twice.
I normally run:
rake db:seed
But i would like to see another command
How can i make a separate seed file to execute on its own.
You can either delete your seeds before insert in seeds.rb or check the count of the model rows or check for the id if the item you're looking to insert before you insert it in your seeds.rb. Basically, I think you just need to add some checks before you insert additional data. I could provide a more specific answer if you posted your seeds.rb.
I've some .json files with data that are automatically updated from time to time. I also have a Ruby on Rails app where I want to insert the information that's in those files.
By now, I'm parsing the JSON and inserting the data in the database in the seeds.rb file, but I'll want to add more data without having to restart the app, I mean, on the go.
From time to time, I'll check those files and if they have modifications, I want to insert those new items into my database.
What's the best way to do it?
Looks like a job for a cron.
Create the code you need in a rake task (in /lib/tasks):
task :import, => :environment do
Importer.read_json_if_modified # your importer class
end
Then run this with the period you want using your system's cron.
This may well be a duplicate, but I couldn't find anyone asking quite this question.
My understanding* is that if I want to migrate data from an outside source to my Rails app's database, I should do so using a migration. It seems from my preliminary research that what I could do is use a tool like FasterCSV to parse a CSV file (for example) right in the migration (.rb) file itself.
Is this the correct approach? And if so, where should I actually put that CSV file -- it seems that if migrations are, after all, meant to be reversible/repeatable, that CSV data ought to be kept in a stable location.
*Let me know if I am completely mistaken about how to even go about this as I am still new to RoR.
You can write this to a rake job without FasterCSV, though I use both.
Write rows to 'csvout' file.
outfile = File.open('csvout', 'wb')
CSV::Writer.generate(outfile) do |csv|
csv << ['c1', nil, '', '"', "\r\n", 'c2']
...
end
outfile.close
This file will output where the rake file is written. In your case, you can put it in a seperate folder for CSV's. I would personally keep it out of the rest of the app structure.
You may want to look into seed_fu to manage it. It has the benefit of being able to easily update the data already in database. You can convert the CSV into a seed file, which is just a Ruby code (example code is provided there).