I have several sidekiq workers in my rails 4 app and can't figure out why the processes are failing. I am passing the User_id to the worker as a string but it can't seem to find the user. But when I search the same id in my console it finds the user. What am i doing wrong? Here is my code.
Controller.rb
def update
#user = current_user
if #user.bill.update_attributes(bill_params)
RandomWorker.perform_async(#user.id.to_s)
redirect_to users_dashboard_path, :notice => "Payment Informaton Updated"
else
...
end
end
Randomworker.rb
def perform(user_id)
user_id = user_id["$oid"] unless user_id.is_a?(String)
user = User.find(user_id)
user.process.update_report
user.process.save
end
My error comes back as
RandomWorker "575a..." Mongoid::Errors::DocumentNotFound: message: Document(s) not found for class User with id(s) 575a.... summary: When calling User.find with an id or array of ids, each parameter...
--EDIT--
My db config file is as shown:
development:
adapter: sqlite3
database: db/development.sqlite3
pool: 5
timeout: 5000
production:
adapter: sqlite3
database: db/production.sqlite3
pool: 25
timeout: 5000
And my mongoid.yml
development:
# Configure available database clients. (required)
clients:
# Defines the default client. (required)
default:
# Defines the name of the default database that Mongoid can connect to.
# (required).
database: development_db
# Provides the hosts the default client can connect to. Must be an array
# of host:port pairs. (required)
hosts:
- localhost:27017
options:
I found a solution. I had to put a monkeypatch in the sidekiq initalizer to treat the user.id as json. Apparently, mongoid struggles with this with sidekiq and although I struggled to find some documentation on it I stumbed across the answer in another unrelated question. Have to_json return a mongoid as a string
I added this in the initalizer and it seems to have fixed the issue
class BSON::ObjectId
def as_json
self.to_s
end
end
I'd advise that you include the Sidekiq::Web to view the enqueued jobs on a web interface so you can see the parameters and possibly the failures triggered.
However, this is an error I also faced a while ago too, which quite frustrated me for a while because of the number of emails I received from my error notifier(Bugsnag).
I've not found the best solution yet, but I imagine that the database was being locked for reading or some records weren't committed before attempting to use them.
Sidekiq's documentation says in your model use, after_commit :perform_sidekiq_job, on: :create.
However, I've not really tried that because I found another approach, the flip side to my new approach is that my jobs are executed much later, at times about 10minutes later.
RandomWorker.perform_in(5.seconds, user_id)
This is less likely to fail with the Sidekiq::retry option.
Read more here
Related
I am building an application in Rails which requires me to access multiple SQL databases. The resources I found online suggested I use ActiveRecord::Base.establish_connection, however, this seems to interfere with my site's models. After connecting to a different database, when I next do <model>.<command>, it gives me Mysql2::Error: Table '<Database I had to access through establish_connection>.<Model's table>' doesn't exist eg: Mysql2::Error: Table 'test.words' doesn't exist , which means rails tries to look for the table associated with its models in the database I had to access through establish_connection instead of the site development database.
Steps to reproduce error:
Here are some steps I found which seem to reproduce the problem;
First, I create a new rails app:
rails new sqldbtest -d mysql
cd sqldbtest
Then I set the config file:
default: &default
adapter: mysql2
encoding: utf8mb4
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
username: root
password: <omitted>
host: localhost
Then create a controller, model, and some data in the mysql database:
rails generate controller test test
rails generate model Word name:string txt:text
rails db:create
rails db:migrate
rails c
cat=Word.new
cat.name="Cat"
cat.txt="Cat."
cat.save
exit
mysql -u root -p # I already have a database called "test".
use test;
create table extst (id int primary key, name varchar(8), txt text);
insert into extst (id,name,txt) values (0,"Bob","Bob.");
quit
I then made the controller and the view:
class TestController < ApplicationController
def test
itemOne=Word.find_by(name:"Cat")
#textOne=itemOne.txt
con=ActiveRecord::Base.establish_connection(adapter: 'mysql2', encoding: 'utf8mb4', username: 'root', password: <omitted>, host: 'localhost', database: 'test').connection
#textTwo=con.execute('select txt from extst where name="Bob"').to_a[0][0]
end
end
I wrote this in the view:
<%= #textOne %><br>
<%= #textTwo %>
added 'root "test#test"' to config/routes.rb
rails s
Result:
When I load the page, it shows "Cat." and "Bob." on separate lines as expected, but when I refresh, It shows the error as described above.
I have tried adding con.close to the controller, but this does not work.
Since this error comes from making a connection to another database, adding ActiveRecord::Base.establish_connection(adapter: 'mysql2', encoding: 'utf8mb4', username: 'root', password: <omitted>, host: 'localhost', database: 'sqldbtest_development').connection to the controller after #textTwo=con.execute('select txt from extst where name="Bob"').to_a[0][0] stops the error from happening, however I don't know if this is the best practice or if it has any side effrcts. For example, the database will need to be changed when moving to test or production.
This solution is most likely just a temporary workaround.
I am trying to build an app which will use one of latest Rails 6.1 features: multiple databases - you can read in sources at the bottom.
I would like to implement multitenancy by being able to switch between databases based on request domain.
Each club would have his own database, and one primary database would store app specific data like Users, Clubs.
I created two types of records: GlobalRecord using primary database and ShardRecord using horizontal shards databases.
And also tried to use around_action to select current club database.
# config/database.yml
default: &default
adapter: postgresql
encoding: unicode
development:
primary:
<<: *default
database: primary
migrations_paths: db/migrate
club_1:
<<: *default
database: club_1
migrations_paths: db/shard_migrate
club_2:
<<: *default
database: club_2
host: 1.1.1.1
username: deploy
password: pass
migrations_paths: db/shard_migrate
class Club < GlobalRecord; end
class GlobalRecord < ActiveRecord::Base
self.abstract_class = true
connects_to shards: {
club_1: { writing: :primary, reading: :primary },
club_2: { writing: :primary, reading: :primary }
}
end
class MemberRecord < ShardRecord; end
class ShardRecord < ActiveRecord::Base
self.abstract_class = true
connects_to shards: {
club_1: { writing: :club_1, reading: :club_1 },
club_2: { writing: :club_2, reading: :club_2 }
}
end
class ApplicationController < ActionController::API
before_action :set_club
around_action :connect_to_shard
private
def set_club
#club = SelectClubByDomain.new(request).slug
end
def connect_to_shard
ActiveRecord::Base.connected_to(role: :writing, shard: #club.slug) do
yield
end
end
end
I have few design questions/issues I would like to ask YOU about:
I would like to set GlobalRecord with connects_to database: { writing: :primary } but because I added around_action
I have to set it as above.
Is it bad design?
Can I refactor it so I could always call GlobalRecord without ActiveRecord::Base.connected_to(role: :writing, shard: :club_1) block?
Tried to use ActiveRecord::Base.connected_to_many(GlobalRecord, ShardRecord, role: :writing, shard: #club_slug.to_sym) in around_action but on request(clubs#index => [Club.all]) I get an error:
ActiveRecord::ConnectionNotEstablished (No connection pool for 'GlobalRecord' found for the 'club_1' shard.)
On production server on start I see this warning, how to fix it?
=> Run `bin/rails server --help` for more startup options
Failed to define attribute methods because of ActiveRecord::ConnectionNotEstablished: ActiveRecord::ConnectionNotEstablished
Sources:
https://guides.rubyonrails.org/active_record_multiple_databases.html
https://www.freshworks.com/horizontal-sharding-in-a-multi-tenant-app-with-rails-61-blog/
https://api.rubyonrails.org/classes/ActiveRecord/ConnectionHandling.html#method-i-connected_to_many
In general it's absolutely fine to switch connections with an around filter, we have been doing this for years (even before Rails 6) in my previous company.
Btw: did you see that the guide is mentioning your posted exception?
Note that connected_to with a role will look up an existing connection and switch using the connection specification name. This means that if you pass an unknown role like connected_to(role: :nonexistent) you will get an error that says ActiveRecord::ConnectionNotEstablished (No connection pool for 'ActiveRecord::Base' found for the 'nonexistent' role.)
https://guides.rubyonrails.org/active_record_multiple_databases.html#using-manual-connection-switching
I have a Rails 5 application that uses three different existing databases. No migrations were needed in this application. I want to build db/schema.rb to include all three databases, not just the main database. Executing rake db:schema:dump rebuilds the schema using the main database. I believe there is a way to do this but for some reason the way I've been searching I cannot find anything about this. All of the posts I have found discuss how to use models from different databases, not how to rebuild the schema to include all databases.
I started looking at this again about a month ago. After reviewing several options I found a solution created by #ostinelli that worked really well and easy to implement. This solution creates separate migrations for each database being maintained instead of putting them all in a single schema file. My goal mainly was to have a schema for each database which this accomplishes for me. There are some parts I did not implement because they were not needed.
http://www.ostinelli.net/setting-multiple-databases-rails-definitive-guide/
I created separate database yml files for each database like this:
config/database_stats.yml
development:
adapter: postgresql
encoding: utf8
host: localhost
pool: 10
database: myapp_stats_development
username: postgres
password:
test:
adapter: postgresql
encoding: utf8
host: localhost
pool: 10
database: myapp_stats_test
username: postgres
password:
production:
adapter: postgresql
encoding: utf8
url: <%= ENV["DATABASE_STATS_URL"] %>
pool: <%= ENV["DB_POOL"] || 5 %>
Then I created separate database rake tasks for each database:
lib/tasks/db_stats.rake
task spec: ["stats:db:test:prepare"]
namespace :stats do
namespace :db do |ns|
task :drop do
Rake::Task["db:drop"].invoke
end
task :create do
Rake::Task["db:create"].invoke
end
task :setup do
Rake::Task["db:setup"].invoke
end
task :migrate do
Rake::Task["db:migrate"].invoke
end
task :rollback do
Rake::Task["db:rollback"].invoke
end
task :seed do
Rake::Task["db:seed"].invoke
end
task :version do
Rake::Task["db:version"].invoke
end
namespace :schema do
task :load do
Rake::Task["db:schema:load"].invoke
end
task :dump do
Rake::Task["db:schema:dump"].invoke
end
end
namespace :test do
task :prepare do
Rake::Task["db:test:prepare"].invoke
end
end
# append and prepend proper tasks to all the tasks defined here above
ns.tasks.each do |task|
task.enhance ["stats:set_custom_config"] do
Rake::Task["stats:revert_to_original_config"].invoke
end
end
end
task :set_custom_config do
# save current vars
#original_config = {
env_schema: ENV['SCHEMA'],
config: Rails.application.config.dup
}
# set config variables for custom database
ENV['SCHEMA'] = "db_stats/schema.rb"
Rails.application.config.paths['db'] = ["db_stats"]
Rails.application.config.paths['db/migrate'] = ["db_stats/migrate"]
Rails.application.config.paths['db/seeds'] = ["db_stats/seeds.rb"]
Rails.application.config.paths['config/database'] = ["config/database_stats.yml"]
end
task :revert_to_original_config do
# reset config variables to original values
ENV['SCHEMA'] = #original_config[:env_schema]
Rails.application.config = #original_config[:config]
end
end
Then I created custom migration generators for the additional databases.
lib/generators/stats_migration_generator.rb
require 'rails/generators/active_record/migration/migration_generator'
class StatsMigrationGenerator < ActiveRecord::Generators::MigrationGenerator
source_root File.join(File.dirname(ActiveRecord::Generators::MigrationGenerator.instance_method(:create_migration_file).source_location.first), "templates")
def create_migration_file
set_local_assigns!
validate_file_name!
migration_template #migration_template, "db_stats/migrate/#{file_name}.rb"
end
end
I created database migrations for each database similar to:
rails g stats_migration create_clicks
create db_stats/migrate/20151201191642_create_clicks.rb
rake stats:db:create
rake stats:db:migrate
Then created separate database initializers for the additional databases then modified each model.
config/initializers/db_stats.rb
DB_STATS = YAML::load(ERB.new(File.read(Rails.root.join("config","database_stats.yml"))).result)[Rails.env]
class Click < ActiveRecord::Base
establish_connection DB_STATS
end
I had three databases in my application when I first wrote this question. I'm now adding another one database. I'm so grateful that #ostinelli wrote this. It made my life easier.
I have a Student model that has_many CustomField (polymorphic) rows. After adding a couple CustomFields to a student in the rails console, I then started up a rails server and navigated to the page that is supposed to display this information, however it is not displaying any of the updated information.
I have a method get_custom_fields_with_values that returns a Hash of field names and values.
In the rails console:
student = Student.all.first
field_attrs = FieldAttributes.create(name: 'Favorite Color') # => true
student.custom_fields.build(field_attributes: field_attrs, value: 'Blue').save # => true
student.get_custom_fields_with_values # => { 'Favorite Color' => 'Blue' }
However, in the view when I make the call to the get_custom_fields_with_values, all that is returned is an empty Hash.
I'm using a Postgres database, and inspecting the data inside it shows that all the rows are there, and all the foreign keys are correct.
This is the query that is made when the page is requested:
CustomField Load (0.3ms) SELECT "custom_fields".* FROM "custom_fields"
WHERE "custom_fields"."custom_fieldable_id" = $1
AND "custom_fields"."custom_fieldable_type" = $2
[["custom_fieldable_id", "34df8e09-148d-5660-bb6c-dd5ca7c9e0ff"],
["custom_fieldable_type", "Student"]]
The database rows:
I've restarted the server multiple times, and still only get an empty hash or empty ActiveRecord::CollectionProxy object. However, upon restarting the console, it is able to retrieve the fields just fine and I get the expected Hash back. And running the query that is run when loading the page also returns the expected behavior.
I've cleared my browser cache, restarted the browser, and tried multiple browsers all to the same effect. Restarted the server multiple times, and restarted the console multiple times.
The HAML:
= bootstrap_panel do |panel|
- panel.content = capture do
.panel-heading
%h2 Custom Fields
.panel-body
= #student.get_custom_fields_with_values
The function:
def get_custom_fields_with_values
custom_fields.inject(Hash.new) do |h, cf|
field_name = cf.field_attributes.name
h[field_name] = cf.value; h
end
end
And all that is being displayed in the panel body:
{}
But the method call in the console returns:
student.get_custom_fields_with_values
=> {"Favorite Car"=>"Prius", "Favorite Color"=>"Blue"}
config/database.yml
default: &default
adapter: postgresql
database: <%= Rails.application.secrets.database %>
host: <%= Rails.application.secrets.database_host %>
username: <%= Rails.application.secrets.database_username %>
password: <%= Rails.application.secrets.database_password %>
port: 5432
pool: 10
development:
<<: *default
test:
<<: *default
staging:
<<: *default
production:
<<: *default
SOLVED
The problem was my database is multi tenanted and rails was looking in the wrong tenant. I added it to the excluded models so it'd look in public, and it fixed the issue.
The issue was that my database is multi tenanted, so rails was looking in the current schema when it should have been looking in the public schema. I just added the CustomFields model to the exclusion list so rails would look in the public schema and that fixed my issue.
is it possible to use 2 types of database in a Rails application?
I am using 2 databases - Postgres to store data which might not change much, and MongoDB
to store data which change dynamically.
is this valid approach? and is it possible to connect between these 2 databases and operate
in a single Rails application?
Please correct, if i am wrong.
regards,
Balan
yes this is possible here is an example from my old code (but here we use mysql for both the DB, but I think you can get an idea)
in database.yml file define two databases
development: &defaults
adapter: mysql
encoding: utf8
database: DB1
enable_call: true
username:
password:
host:
portal_development: &defaults
adapter: mysql
encoding: utf8
database: DB2
enable_call: true
username:
password:
host:
in your models have to base models coupled with above databases
portal_base.rb
class PortalBase < ActiveRecord::Base
self.abstract_class = true
establish_connection "portal_#{Rails.env}"
def self.table_name_prefix
"DB1."
end
end
default_base.rb
class DefaultBase < ActiveRecord::Base
self.abstract_class = true
establish_connection "#{Rails.env}"
def self.table_name_prefix
"DB2."
end
end
and derive your models accordingly
class Client < PortalBase
#code
end
Hope you have an idea now :)
Found solution :
We can use mongoid gem to achieve this.
Steps to install
1) Install mongoid by adding "gem mongoid" in Gemfile and running bundle command
2) Generate mongoid configuration by typing "rails g mongoid:config", this will create a
config file mongoid.yml near database.yml file, you can add the configuration to Mongo server
in this file.
Note : After adding Mongoid, all the models created will be created for MongoDB by default, you can specify --orm option to generate Models for Postgres.