I am using a rakefile to fetch information from one website and saving it to my database.
Using the TMDB-Gem, this code #movie = TmdbMovie.browse(:order_by => "release", :order => "asc", :page => 1, :per_page => 2, :language => "en", :expand_results => true) browses the oldest movies (:order_by => "release") and saves them to my database, but as i'll be running this rake quite frequently, the returned and saved movies will be the same.
Every movie has a tmdb_id and every id is unique
How can i make the rakefile check that the returned movie's tmdb_id is unique, and if there is already a movie with that id, skip and save the next movie.
I tried it in my Movies model validates_uniqueness_of :tmdb_id but it gives an error when running the rake command and it doesn't save movies.
Basically, how can I, through the rakefile, validate the uniqueness of the tmdb_id
This is my rake file
namespace :db do
task :pull_tmdb_data => :environment do
Tmdb.api_key = "API KEY"
Tmdb.default_language = "en"
#movie = TmdbMovie.browse(:order_by => "release", :order => "asc", :page => 1, :per_page => 2, :language => "en", :expand_results => true)
#movie.each do |movie|
Movie.create(title: movie.name, description: movie.overview, release_date: movie.released, tmdb_id: movie.id)
end
end
end
You're right to use validates_uniqueness_of :tmdb_id in your Movie model. You didn't show how you were saving the movies to your database, but .save doesn't cause an exception when validations fail, whereas .save! does. The key would be to use a save method that doesn't raise an error when validations fail.
Edit - Now that I understand what you're actually trying to do, you should be able to do something like:
per_page = 100
number_of_movies = Movie.count
page = number_of_movies/per_page+1
#movies = TmdbMovie.browse(:order_by => "release", :order => "asc", :page => page, :per_page => per_page, :language => "en", :expand_results => true) browses the oldest movies (:order_by => "release")
So if you've already pulled 435 movies, it'll only return movies 400-500 in the next call .. I did it this way because I wasn't sure if there was an offset option, but if there is you could just offset the query by Movie.count, which would be better.
Related
I'm successfully following some of the commands listed here and applying them to my own tables.
How do I turn the output into actual records/data ? Presently it's returning some output which I can assign to a variable, but I do not understand this output at all.. it doesn't look like an array or hash (but perhaps it is), and I do not understand how to access its contents (I want to use the output in a rails view, but at the least would like to access the table records in the console where I'm executing the Arel code)
For example, say I have two tables; User and Product as follows:
User = [{:id => 1, :name => "Joe", :email => "joe#gmail.com"}, {:id => 2, :name => "Jane", :email => "jane#gmail.com"}]
Product = [{:product_name => "Car Tire", :brand => "BMW", :last_edited_by => 1}, {:product_name => "Paint Brush", :brand => "Dulux", :last_edited_by => nil}, {:product_name => "Dog Biscuits", :brand => "Bowow Snacks", :last_edited_by => 2}, {:product_name => "Game", :brand => "Westwood", :last_edited_by => 1}]
Suppose I want to join the two tables User.id = Product.last_edited_by
In SQL it would be as simple as:
SELECT *
FROM User
JOIN Product
WHERE users.id = products.last_edited_by
Now if I plug this into scuttle.io to convert to Arel, it gives:
User.select(Arel.star).where(
User.arel_table[:id].eq(Product.arel_table[:last_edited_by])
).joins(
User.arel_table.join(Product.arel_table).on(null).join_sources
)
But trying this returns: ArgumentError: wrong number of arguments (given 1, expected 0)
Attempting to emulate advice given here, I come up with another approach:
users.project(Arel.star).join(User).on(Product[:last_edited_by].eq(User[:id]))
But this returns: NoMethodError: undefined method `project' for Array..
A final point, if I run
users = Arel::Table.new(User)
output = users.project(users[:id].as("id"), users[:id].count.as("count")).group("id")
I get some output which looks almost meaningful, but I don't know how to return that into something I can make sense of, and I wouldn't know how to use it if I provided it to a rails view
This is just a simple question. I was trying to create a new object in Rails by passing in parameters to the constructor. However, when I execute the code, I get
SQLite3::SQLException: no such column: awards.user_id: SELECT "awards".* FROM "awards" WHERE "awards"."user_id" = 1
which means the object isn't being constructed properly. Should I be using create instead of new? That isn't working either.
def refresh_awards(user)
new_awards = []
if (user.karma < 40 ) #test award
a = Award.new(:name => "Nobody Award", :description => "From Jonathan", :category => "Community", :value => 1337, :level => 0, :handle => "nobody_award")
user.awards.append(a)
new_awards.append(a)
end
new_awards.each do |a|
flash[:notice] = "You received the " + a.name + "!"
end
end
Have you add has_many :awards to the User model? Have you added belongs_to :user to the Award model? Have you added the column user_id to the Award model (using a migration)? You'll need to do these three things to be able to use the user.awards method you're using. Read the Rails Guide on Associations for more detail.
Also, append isn't a Ruby method - the closest method would be <<. You would use it like this:
a = Award.new(:name => "Nobody Award", :description => "From Jonathan", :category => "Community", :value => 1337, :level => 0, :handle => "nobody_award")
user.awards << a
But you could neaten this into one line of code using the create method:
a = user.awards.create(:name => "Nobody Award", :description => "From Jonathan", :category => "Community", :value => 1337, :level => 0, :handle => "nobody_award")
EDIT: To create the user_id column in the Award model, run the following code from terminal (while in your app's directory):
rails generate migration AddUserIdToAward user_id:integer
rake db:migrate
I have some trouble with mongoid:
test "Test candidate" do
User.create(:id => 1, :sex => User::Male, :country => 1, :city => 1)
User.create(:id => 2, :sex => User::Female, :country => 1, :city => 1)
User.create(:id => 3, :sex => User::Female, :country => 1, :city => 1)
user = User.not_in(:id => [2]).second
assert_not_equal(user.id, 2)
end
Test failed. I've tried to use where(:id => {'$nid' => [2]}), but it have same effect.
What is wrong? How to use "not in" condition with mongoid?
PS, "second" is ok, with "first" test passed, because id=1
Try this query:
user = User.not_in(:_id => [2]).second
In MongoDB primary key has name _id. Mongoid tries to be friendly and partially hides this fact from the developer by aliasing it to id in the object model. But when you do queries, it cannot tell if you want primary key _id or some completely ordinary field id.
user = User.where(:id.nin => [2,3,4,5])
This is as per mongoid official doc : http://mongoid.org/en/origin/docs/selection.html
I have the following Rake file. Using RoR 2.3.8.
desc "Create shops sitemap"
task(:shops => :environment) do
sitemap = Sitemap.new
#add every item
for i in shop.find(:all, :select => 'id, updated_at', :order => 'updated_at DESC', :limit => 50000)
sitemap.add_url("http://abc.com/shops/#{i.id}",w3c_date(i.updated_at),'daily','1.0')
end
puts "#{sitemap.urls.length} total urls"
#delete the file
FileUtils.rm(File.join(RAILS_ROOT, "public/sitemap_shops_1.xml.gz"), :force => true)
f =File.new(File.join(RAILS_ROOT, "public/sitemap_shops_1.xml"), 'w')
sitemap.write(f,2)
f.close
system("gzip #{File.join(RAILS_ROOT, 'public/sitemap_shops_1.xml')}")
end
The file above searches the first 50,000 records based on last updated, then save in a file numbered 1.
How do I modify the code to have it search the next 50,000, and save the file numbered 2, then next 50,000, save as file numbered 3, etc.?
Thanks.
Instead of find, you can use find_in_batches which will return groups of 1,000 at a time (but you can override this to be 50,000 with the :batch_size option). Throw in a counter variable (since I don't think find_in_batches has anything like an each_with_index) and you can handle all the files you need.
desc "Create shops sitemap"
task(:shops => :environment) do
file_name_index = 0
Shop.find_in_batches(:all, :select => 'id, updated_at', :order => 'updated_at DESC', :batch_size => 50000) do |group_of_50000|
file_name_index += 1
sitemap = Sitemap.new
#add every item
for i in group_of_50000
sitemap.add_url("http://abc.com/shops/#{i.id}",w3c_date(i.updated_at),'daily','1.0')
end
puts "#{sitemap.urls.length} total urls"
#delete the file
FileUtils.rm(File.join(RAILS_ROOT, "public/sitemap_shops_#{file_name_index}.xml.gz"), :force => true)
f =File.new(File.join(RAILS_ROOT, "public/sitemap_shops_#{file_name_index}.xml"), 'w')
sitemap.write(f,2)
f.close
system("gzip #{File.join(RAILS_ROOT, 'public/sitemap_shops_#{file_name_index}.xml')}")
end
end
while writing some application for personal use. I find out the child query is not as great as it look.
For instance, I have 2 object
Category has_many Files
File belongs_to Category
File.category will access its parent category. But this lead to the famous N+1 problem.
For example, I want my homepage to list out 20 newest files and its corresponding category using something like this
for file in #files
%p
= file.name
= file.category.name
How should I solve this problem?
#files = File.find(:all, :limit => 20, :order => "created at desc", :include => :category)
In your find if you say :include => :category then this will eager load the categories for you and avoid a separate query to retrieve each category's name. So for your example of the 20 most recent files you could do:
#files = File.find :all, :limit => 20, :include => :category,
:order => 'created at desc'