The error is ActiveRecord::RecordInvalid: Validation failed: Route must exist.
This is the code:
new_route = Route.create!(new_route_data)
new_points.each_with_index do |point, index|
new_point_data = { route_id: new_route.id,
latitude: point[0],
longitude: point[1],
timestamp: point[2] }
new_point = Point.create!(new_point_data)
end
It is being reported for the new_point = Point.create!(new_point_data) line.
Related Details:
This code is running in a single Sidekiq worker as you see it above (so, the Route isn't being created in one worker, with the Points being created in another worker - this is all inline)
The routes table has almost 3M records
The points table has about 2.5B records
The Point model contains belongs_to :route, counter_cache: true
There are no validations on the Route model
In case it's relevant, the Route model does contain belongs_to :user, counter_cache: true
There are only about 5k records in the users table
Software versions:
Rails 5.1.5
Ruby 2.5.0
PostgreSQL 9.6.7
First of all, your code does not make sense. You are iterating new_point and assigning new_point to some value in the block. So I am assuming that you meant iteration over some collection called data_points
Try this.
In Route model
has_many :points
then:
new_route = ...
data_points.each do |point|
point_data = {latitude: point[0], ...} # do not include route_id
new_route.points.create(point_data) # idiomatic
end
`
You don't need the index, so don't use each_with_index.
It's tough to say what the issue is without seeing what type of validations you have in the Point model.
My guess is that your validation in point.rb is:
validates :route, presence: true
With ActiveRecord relations, you can use this shortcut to avoid explicitly assigning route_id:
new_point_data = { latitude: point[0],longitude: point[1], timestamp: point[2] }
new_route.points.create!(new_point_data)
where new_point data does not have route_id.
You should also rename the new_point that you are assigning in the block since you are writing over the array that you're iterating.
Related
I'm kinda struggling with validations in my Rails application.
I have the following Setup:
class MealDay < ApplicationRecord
belongs_to :meal
belongs_to :day
has_many :meal_day_canteens
has_many :canteens,
through: :meal_day_canteens
validates :meal_id, uniqueness: {scope: :day_id}
end
#MainController
helping_hash.each { |days, meal|
dayRef = Day.where(date: days).first
mealRef = Meal.where(name: meal)
dayRef.meals << mealRef #This line is obviously throwing the error because a record exists already
}
The Error is: "Validation failed: Meal has already been taken"
But I'm not sure on how to handle the error. I just want it, so that Rails is not inserting it into the database and skips it. If you need more information just tell me.
Thanks.
Edit: Some more code which I can't get to work now.
Edit2: Forgot to add validation to that model. Works fine now
helping_hash.each { |days, meal|
dayRef = Day.where(date: days).first
mealRef = Meal.where(name: meal)
meal_day_ref = MealDay.where(day: dayRef, meal: mealRef)
#canteenNameRef.meal_days << meal_day_ref rescue next
}
How about rescueing the error?
helping_hash.each do |days, meal|
dayRef = Day.where(date: days).first
mealRef = Meal.where(name: meal)
dayRef.meals << mealRef rescue next
end
Rails uniqueness constraint is basically throwing a query under the hood to look for a record in the DB. More of a side comment but this is already not safe concurrently so I recommend you adding a constraint at the database level.
You basically need to do the manual work of skipping the ones that already exist.
Basically something like:
helping_hash.each do |day, meal|
day_ref = Day.find_by!(date: day)
meal_ref = Meal.find_by!(name: meal)
unless day_ref.meal_ids.include?(meal_ref.id)
# Depending on the number of records you expect
# using day_ref.meals.exists?(id: meal_ref.id) may be a better choice
day_ref.meals << meal_ref
end
end
Basically, I have a model, Degree, and it has three attributes: degree_type, awarded_by, and date_awarded.
There are two arrays of values that should be valid for awarded_by. The two valid values for degree_type are "one" and "two", and the valid values for awarded_by depend on "one" and "two".
If degree_type is "one" (has a value of "one", that a user would put in), I want the valid values for awarded_by to be array_one. If degree_type has a value of "two", I want the valid values for awarded_by to be array_two.
Here is the code so far:
class Degree < ActiveRecord::Base
extend School
validates :degree_type, presence: true,
inclusion: { in: ["one",
"two"],
message: "is not a valid degree type"
}
validates :awarded_by, presence: true,
inclusion: { in: Degree.schools(awarded_by_type) }
end
Degree.schools outputs an array depending on the degree type, so Degree.schools("one") would return array_one, where
array_one = ['school01', 'school02'...]
My problem is, I don't know how to access the value of degree_type within the model.
What I tried below doesn't work:
validates :awarded_by, presence: true,
inclusion: { in: Degree.schools(:degree_type) }
I tried using before_type_cast but I was either using it incorrectly or there was another problem, as I couldn't get that to work either.
When I test this I get:
An object with the method #include? or a proc, lambda or symbol is required, and must be supplied as the :in (or :within) option of the configuration hash
Help me out? :) If any more info is needed, let me know.
EDIT: To add to this, I double checked it wasn't my Degree.schools method acting up - if I go into the rails console and try Degree.schools("one") or Degree.schools("two") I do get the array I should get. :)
EDIT again: When I tried #Jordan's answer, I got errors in the cases where the awarded_by was incorrect because in those cases, valid_awarded_by_values was nil and there is no include? method for a nil object. Therefore I added an if statement checking for whether valid_awarded_by_values was nil or not (so as to return if it was), and that solved the problem!
I put this inside the method, before the unless statement and after the valid_awarded_by_values declaration:
if valid_awarded_by_values.nil?
error_msg = "is not a valid awarded_by"
errors.add(:awarded_by, error_msg)
return
end
The easiest way will be to write a custom validation method, as described in the Active Record Validations Rails Guide.
In your case, it might look something like this:
class Degree < ActiveRecord::Base
validate :validate_awarded_by_inclusion_dependent_on_degree_type
# ...
def validate_awarded_by_inclusion_dependent_on_degree_type
valid_awarded_by_values = Degree.schools(degree_type)
unless valid_awarded_by_values.include?(awarded_by)
error_msg = "must be " << valid_awarded_by_values.to_sentence(two_words_connector: ' or ', last_word_connector: ', or ')
errors.add(:awarded_by, error_msg)
end
end
end
Rails 4, Mongoid instead of ActiveRecord (but this should change anything for the sake of the question).
Let's say I have a MyModel domain class with some validation rules:
class MyModel
include Mongoid::Document
field :text, type: String
field :type, type: String
belongs_to :parent
validates :text, presence: true
validates :type, inclusion: %w(A B C)
validates_uniqueness_of :text, scope: :parent # important validation rule for the purpose of the question
end
where Parent is another domain class:
class Parent
include Mongoid::Document
field :name, type: String
has_many my_models
end
Also I have the related tables in the database populated with some valid data.
Now, I want to import some data from an CSV file, which can conflict with the existing data in the database. The easy thing to do is to create an instance of MyModel for every row in the CSV and verify if it's valid, then save it to the database (or discard it).
Something like this:
csv_rows.each |data| # simplified
my_model = MyModel.new(data) # data is the hash with the values taken from the CSV row
if my_model.valid?
my_model.save validate: false
else
# do something useful, but not interesting for the question's purpose
# just know that I need to separate validation from saving
end
end
Now, this works pretty smoothly for a limited amount of data. But when the CSV contains hundreds of thousands of rows, this gets quite slow, because (worst case) there's a write operation for every row.
What I'd like to do, is to store the list of valid items and save them all at the end of the file parsing process. So, nothing complicated:
valids = []
csv_rows.each |data|
my_model = MyModel.new(data)
if my_model.valid? # THE INTERESTING LINE this "if" checks only against the database, what happens if it conflicts with some other my_models not saved yet?
valids << my_model
else
# ...
end
end
if valids.size > 0
# bulk insert of all data
end
That would be perfect, if I could be sure that the data in the CSV does not contain duplicated rows or data that goes against the validation rules of MyModel.
My question is: how can I check each row against the database AND the valids array, without having to repeat the validation rules defined into MyModel (avoiding to have them duplicated)?
Is there a different (more efficient) approach I'm not considering?
What you can do is validate as model, save the attributes in a hash, pushed to the valids array, then do a bulk insert of the values usint mongodb's insert:
valids = []
csv_rows.each |data|
my_model = MyModel.new(data)
if my_model.valid?
valids << my_model.attributes
end
end
MyModel.collection.insert(valids, continue_on_error: true)
This won't however prevent NEW duplicates... for that you could do something like the following, using a hash and compound key:
valids = {}
csv_rows.each |data|
my_model = MyModel.new(data)
if my_model.valid?
valids["#{my_model.text}_#{my_model.parent}"] = my_model.as_document
end
end
Then either of the following will work, DB Agnostic:
MyModel.create(valids.values)
Or MongoDB'ish:
MyModel.collection.insert(valids.values, continue_on_error: true)
OR EVEN BETTER
Ensure you have a uniq index on the collection:
class MyModel
...
index({ text: 1, parent: 1 }, { unique: true, dropDups: true })
...
end
Then Just do the following:
MyModel.collection.insert(csv_rows, continue_on_error: true)
http://api.mongodb.org/ruby/current/Mongo/Collection.html#insert-instance_method
http://mongoid.org/en/mongoid/docs/indexing.html
TIP: I recommend if you anticipate thousands of rows to do this in batches of 500 or so.
I am running a high traffic test with Sidekiq that creates MongoDB based objects using Mongoid as my driver in a Rails 4 app. The issue I'm seeing is that when a PlayByPlay document is supposed to have a unique game_id, I see multiple PlayByPlay objects getting created with the same exact game_id. I've enforced the unique constraint on MongoDB as well and this is still happening. Here's my document, it's embedded document, and a glimpse into how I'm creating the documents. The issue is that this is all happening in a threaded environment using Sidekiq, and I'm not sure if there is a way to work around it. My write concern is set to 1 in mongoid.yml and it looks like the safe option was removed in master as was persist_in_safe_mode. Code below -- any suggestions on how to properly work this would be appreciated. This is not a replica set, it's a single MongoDB server performing all read/write requests at this time.
module MLB
class Play
include Mongoid::Document
include Mongoid::Timestamps
embedded_in :play_by_play
field :batter#, type: Hash
field :next_batter#, type: Hash
field :pitchers#, type: Array
field :pitches#, type: Array
field :fielders#, type: Array
field :narrative, type: String
field :seq_id, type: Integer
field :inning, type: Integer
field :outs
field :no_play
field :home_team_score
field :away_team_score
end
class PlayByPlay
include Mongoid::Document
include Mongoid::Timestamps
embeds_many :plays, cascade_callbacks: true
accepts_nested_attributes_for :plays
field :sport
field :datetime, type: DateTime
field :gamedate, type: DateTime
field :game_id
field :home_team_id
field :away_team_id
field :home_team_score
field :away_team_score
field :season_year
field :season_type
field :location
field :status
field :home_team_abbr
field :away_team_abbr
field :hp_umpire
field :fb_umpire
field :sb_umpire
field :tb_umpire
index({game_id: 1})
index({away_team_id: 1})
index({home_team_id: 1})
index({season_type: 1})
index({season_year: 1})
index({"plays.seq_id" => 1}, {unique: true, drop_dups: true})
#validates 'play.seq_id', uniqueness: true
validates :game_id, presence: true, uniqueness: true
validates :home_team_id, presence: true
validates :away_team_id, presence: true
validates :gamedate, presence: true
validates :datetime, presence: true
validates :season_type, presence: true
validates :season_year, presence: true
def self.parse!(entry)
#document = Nokogiri::XML(entry.data)
xslt = Nokogiri::XSLT(File.read("#{$XSLT_PATH}/mlb_pbp.xslt"))
transform = xslt.apply_to(#document)
json_document = JSON.parse(transform)
obj = find_or_create_by(game_id: json_document['game_id'])
obj.sport = json_document['sport']
obj.home_team_id = json_document['home_team_id']
obj.away_team_id = json_document['away_team_id']
obj.home_team_score = json_document['home_team_score']
obj.away_team_score = json_document['away_team_score']
obj.season_type = json_document['season_type']
obj.season_year = json_document['season_year']
obj.location = json_document['location']
obj.datetime = DateTime.strptime(json_document['datetime'], "%m/%d/%y %H:%M:%S")
obj.gamedate = DateTime.strptime(json_document['game_date'], "%m/%d/%Y %H:%M:%S %p")
obj.status = json_document['status']
obj.home_team_abbr = json_document['home_team_abbr']
obj.away_team_abbr = json_document['away_team_abbr']
obj.hp_umpire = json_document['hp_umpire']
obj.fb_umpire = json_document['fb_umpire']
obj.sb_umpire = json_document['sb_umpire']
obj.tb_umpire = json_document['tb_umpire']
p=obj.plays.build(seq_id: json_document['seq_id'])
p.batter = json_document['batter']
p.next_batter = json_document['next_batter'] if json_document['next_batter'].present? && json_document['next_batter'].keys.count >= 1
p.pitchers = json_document['pitchers'] if json_document['pitchers'].present? && json_document['pitchers'].count >= 1
p.pitches = json_document['pitches'] if json_document['pitches'].present? && json_document['pitches'].count >= 1
p.fielders = json_document['fielders'] if json_document['fielders'].present? && json_document['fielders'].count >= 1
p.narrative = json_document['narrative']
p.seq_id = json_document['seq_id']
p.inning = json_document['inning']
p.outs = json_document['outs']
p.no_play = json_document['no_play']
p.home_team_score = json_document['home_team_score']
p.away_team_score = json_document['away_team_score']
obj.save
end
end
end
** NOTE **
This problem goes away if I limit sidekiq to 1 worker, which obviously in the real world I'd never do.
You already have an index on game_id, why not making it unique? that way the db will not allow a duplicate entry, even if mongoid fails to do the validation correctly (#vidaica's answer describes how mongoid could fail to validate the uniqueness).
Try adding a unique index
index({"game_id" => 1}, {unique: true})
and then
rake db:mongoid:create_indexes
to create them in mongo (please make sure that it is created from a mongo shell).
After that, mongodb should not persist any records with duplicate game_id and you'll have to do on the ruby layer is to handle the insert errors that you'll receive from mongodb.
This is because many threads inserting objects with the same game_id. Let me paraphrase it.
For example, you have two sidekiq threads t1 and t2. They run in parallel. Assuming you have a document with game_id 1 and it has not been inserted into the database.
t1 enters parse method, it sees no document in the database with game_id 1, it creates a document with game_id 1 and continues to populate other data, but it has not saved the document.
t2 enters parse method, it sees no document in the database with game_id 1 because at this point t1 has not saved the document. t2 creates a document with the same game_id 1.
t1 save the document
t2 save the document
The result: you have two documents with the same game_id 1.
To prevent this you can use a Mutex to serialize the access of the parsing code. To know how to use a Mutex, read this: http://www.ruby-doc.org/core-2.0.0/Mutex.html
Whatever you do, you will want to solve this on the database level because you will almost certainly do a worst job of implementing unique constraints then what mongo people did.
Assuming you will want to shard one day or consider mongo due to its horizontal scalability features (you're doing high volume testing so I assume this is something you don't want to rule out by design), there may be no reliable way to do this (see Ramifications of working with a mongodb cluster and sharding concepts):
Suppose we were sharding on email and wanted to have a unique index on username. This is not possible to enforce with a cluster.
However, if you're sharding on game_id or you're not considering sharding at all then setting a unique index on game_id should prevent double records (see #xlembouras answer).
However, that answer may not prevent exceptions when this index is violated due to race conditions so be sure to rescue that exception and perform an update instead of create in the rescue block (possibly by playing with #new_record (click 'Show source'), will try to find time to give you exact code).
UPDATE, short fast answer
begin
a = Album.new(name: 'foo', game_id: 3)
a.save
rescue
a.id = id_of_the_object_with_same_id_already_in_db
a.instance_variable_set('#new_record', false)
a.save
end
#vidaica's answer is helpful. If you were fetching and incrementing an ID from memory or a database, it might solve your problem.
However, your game_id is not being generated in parse, it is being passed into parse via the entry JSON object.
How / where is your game_id being generated?
Maybe you should do an upsert instead of an insert:
obj = new(game_id: json_document['game_id'])
obj.upsert
A naive approach is to change the last line of #parse to:
obj.save if where(game_id: obj.game_id).count == 0
Or if you hand to handle it somehow:
if where(game_id: obj.game_id).count == 0
# handle it here
end
Note however that this still leaves possibilities for duplicates.
I want to validate my date (which actually have DATE type) in model. So, i try to write for that simle method and run it via validation.
Teacher
class Teacher < ActiveRecord::Base
attr_accessible :teacher_birthday # DATE type!
belongs_to :user
validates :teacher_birthday, :presence => true,
:unless => :date_is_correct?
########
def date_is_correct?
parsed_data = Date._parse(:teacher_birthday)
input_day = parsed_data[:mday]
input_month = parsed_data[:mon]
input_year = parsed_data[:year]
correct_days = 1..31
correct_months = 1..12
correct_year = 1900..2000
if ( correct_days.member? input_day ) and ( correct_months.member? input_month) and
( correct_year.member? input_year)
true
else
errors.add(:teacher_birthday, 'date is invalid')
false
end
end
When i run rspec a lot of tests fail.
TypeError: can't convert Symbol into String
# ./app/models/teacher.rb:56:in `_parse'
# ./app/models/teacher.rb:56:in `date_is_correct?'
I suppose i do something wrong. Can someone tell me what is wrong?
This isn't necessary at all. If Date.parse(:teacher_birthday) returns a date and doesn't raise an exception, you have a valid date.
Date._parse expects a string-value containing a date and will in your code always try to parse 'teacher_birthday'. You need to get the value of the field first and pass the value to Date._parse. ActiveRecord creates methods with the same name as the field to get the value.
Any of the following will work:
Short way
parsed_data = Date._parse(teacher_birthday)
Identically to the first (the self. is added for you during parsing)
parsed_data = Date._parse(self.teacher_birthday)
Explicit way
parsed_data = Date._parse(self[:teacher_birthday])
A new gem has been created to help validate types in rails and an explanatory blog post exists to answer more of the "why" it was created in the first place.
With this library your code would simple be:
class Post < ActiveRecord::Base
validates_type :teacher_birthday, :date
end
This will throw an exception when anything except a a Date is assigned to :teacher_birthday.