create elasticsearch index transactional - ruby-on-rails

Sorry for my bad English,This is not my first language.
I tried to create an index on elasticsearch but I want to be transactional.
Means if error occurred on saving process or on creating index both processes rollback.
I have these lines to save an entry
ActiveRecord::Base.transaction do
entries.map do |entry|
entry = entries.where(source_entry_id: entry.entry_id).first_or_initialize
entry.data = feed_entry.to_hash(self)
entry.save!
end
end
then I defined this class on concern to have searchable functionality for some entities
require 'elasticsearch/model'
module Searchable
extend ActiveSupport::Concern
included do
include Elasticsearch::Model
after_save { index_document }
after_destroy { delete_document }
end
module ClassMethods
def create_index!
self.__elasticsearch__.create_index! force: true
end
def search_index(*args)
self.__elasticsearch__.search(*args)
end
end
def index_document
return if Rails.env.test?
self.__elasticsearch__.index_document
rescue StandardError => e
handle_error(e)
end
def delete_document
return if Rails.env.test?
self.__elasticsearch__.delete_document
rescue StandardError => e
handle_error(e)
end
def format_date(date)
date.to_s(:iso8601) if date
end
def handle_error(e)
error = Searchable.parse_error(e.message)
if error['result'] != 'not_found'
Raven.capture_message("Elasticsearch: #{error['result']}", extra: error)
end
end
def self.parse_error(string)
parts = string.match(/\[(\d*)\]\s(.*)/)
return ({ 'result': string }) unless parts
result = JSON.parse(parts[2])
result['status'] = parts[1]
result
end
end
and my model include searchable
class
Entry < ApplicationRecord
include Searchable
...
Problem that I faced is if I rescue the error entry will save into database if I throw exception and not handle the exception process will freeze and not doing for the rest entries. how can I do this correctly?

Elasticsearch is not transactional.
You may want to add a transactional queue system in the middle or just log when something goes wrong and manually deal with inconsistencies.
You can also send errors in a message queue system to process them later.

Related

How to show the dependent records that will be deleted before destroying a record

Django admin shows you the dependent records that will be deleted when you delete a record as a confirmation.
Is there a way to do the same on Ruby on Rails?
I have been researching how to do it, but I am still looking for a way.
I couldn't find a gem, so I wrote this concern using association reflections:
module DependentDestroys
extend ActiveSupport::Concern
DEPENDENT_DESTROY_ACTIONS = %i[destroy delete destroy_async]
class_methods do
def dependent_destroy_reflections
#dependent_destroy_reflections ||= reflections.filter_map do |name, r|
r if DEPENDENT_DESTROY_ACTIONS.include?(r.options[:dependent])
end
end
end
def total_dependent_destroys
dependent_destroy_counts.sum { |r| r[1] }
end
def any_dependent_destroys?
dependent_destroy_counts.any?
end
# If you want all affected records...
def dependent_destroy_records
self.class.dependent_destroy_reflections.flat_map do |r|
relation = self.public_send(r.name)
if r.collection?
relation.find_each.to_a
else
relation
end
end
end
# If you only want the record type and ids...
def dependent_destroy_ids
self.class.dependent_destroy_reflections.flat_map do |r|
relation = self.public_send(r.name)
if r.collection?
relation.pluck(:id).map { |rid| [r.klass, rid] }
else
[[r.klass, relation.id]] if relation
end
end.compact
end
# If you only want counts...
def dependent_destroy_counts
self.class.dependent_destroy_reflections.filter_map do |r|
relation = self.public_send(r.name)
if r.collection?
c = relation.count
[r.klass, c] if c.positive?
else
[r.klass, 1] if relation
end
end
end
def dependent_destroy_total_message
"#{total_dependent_destroys} associated records will be destroyed"
end
def dependent_destroy_message
# Using #human means you can define model names in your translations.
"The following dependent records will be destroyed: #{dependent_destroy_ids.map { |r| "#{r[0].model_name.human}/#{r[1]}" }.join(', ')}"
end
def dependent_destroy_count_message
"The following dependent records will be destroyed: #{dependent_destroy_counts.map { |r| "#{r[0].model_name.human(count: r[1])} (#{r[1]})" }.join(', ')}"
end
end
Usage:
class User
include DependentDestroys
belongs_to :company
has_many :notes
has_one :profile
end
user = User.first
user.any_dependent_destroys?
# => true
user.total_dependent_destroys
# => 60
user.dependent_destroy_total_message
# => "60 associated records will be destroyed"
user.dependent_destroy_message
# => "The following dependent records will be destroyed: Note/1, Note/2, ..., Profile/1"
user.dependent_destroy_count_message
# => "The following dependent records will be destroyed: Notes (59), Profile (1)"
You can then use these methods in the controller to deal with the user flow.
With some improvements, options (like limiting it to the associations or modes (destroy, delete, destroy_async) you want) and tests, this could become a gem.

Rails gem multiple NOSQL datastores

I'm creating my own gem and I want to enable user to save data to multiple NOSQL data stores. How can I make this happen? Where should I place the necessary files?
I've done the same thing in my gem. I think you have created an App folder in your gem/engine. Create another folder called "backend" and create classes for each datastore. For my case I created a seperate for Mongo and Redis
module Memberfier
class RedisStore
def initialize(redis)
#redis = redis
end
def keys
#redis.keys
end
def []=(key, value)
value = nil if value.blank?
#redis[key] = ActiveSupport::JSON.encode(value)
end
def [](key)
#redis[key]
end
def clear_database
#redis.keys.clone.each {|key| #redis.del key }
end
end
end
module Memberfier
class MongoStore
def initialize(collection)
#collection = collection
end
def keys
#collection.distinct :_id
end
def []=(key, value)
value = nil if value.blank?
collection.update({:_id => key},
{'$set' => {:value => ActiveSupport::JSON.encode(value)}},
{:upsert => true, :safe => true})
end
def [](key)
if document = collection.find_one(:_id => key)
document["value"]
else
nil
end
end
def destroy_entry(key)
#collection.remove({:_id => key})
end
def searchable?
true
end
def clear_database
collection.drop
end
private
def collection; #collection; end
end
end
You may have already seen one of Uncle Bob's presentations on application architecture. If not, it's here. I'd recommend having a single boundary object that select models inherit from. That boundary object could have multiple CRUD methods such as find, create, delete. That boundary object could inherit from whatever NOSQL adapter you configure. Example/source: http://hawkins.io/2014/01/pesistence_with_repository_and_query_patterns/

How to test such class behavior in Rspec

I have a class which is responsible for dealing with some response from payments gateway.
Let's say:
class PaymentReceiver
def initialize(gateway_response)
#gateway_response = gateway_response
end
def handle_response
if #gateway_response['NC_STATUS'] != '0'
if order
order.fail_payment
else
raise 'LackOfProperOrder'
# Log lack of proper order
end
end
end
private
def order
#order ||= Order.where(id: #gateway_response['orderID']).unpaid.first
end
end
In payload from payment I've NC_STATUS
which is responsible for information if payment succeed and orderID which refers to Order ActiveRecord class byid`.
I would like to test behavior(in rspec):
If PaymentReceiver receives response where NC_STATUS != 0 sends fail_payment to specific Order object referred by orderID.
How you would approach to testing this ? I assume that also design could be bad ...
You have to make refactorization to remove SRP and DIR principles violations.
Something below I'd say:
class PaymentReceiver
def initialize(response)
#response = response
end
def handle_response
if #response.success?
#response.order.pay
else
#response.order.fail_payment
end
end
end
# it wraps output paramteres only !
class PaymentResponse
def initialize(response)
#response = response
end
def order
# maybe we can check if order exists
#order ||= Order.find(#response['orderID'].to_i)
end
def success?
#response['NCSTATUS'] == '0'
end
end
p = PaymentReceiver.new(PaymentResponse({'NCSTATUS' => '0' }))
p.handle_response
Then testing everything is easy.

Organic Rails Data-Model with NoSQL Database

I'd like to make full use of the organic character of a NoSQL document and build a dynamic data model which can grow, be changed, and is different for most datasets. Below is the model SomeRequest.rb with the code to set and get from Couchbase, but I can't get the function addOrUpdate(key, value) to work:
undefined method `each' for "0":String
Completed 500 Internal Server
Error in 16ms NoMethodError (undefined method `each' for "0":String):
config/initializers/quiet_assets.rb:7:in `call_with_quiet_assets'
Is the returning error. Is there a way to make this work, to add (or update existing) keys and save the document to the database afterwards?
class SomeRequest < Couchbase::Model
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Callbacks
extend ActiveModel::Naming
# Couch Model
define_model_callbacks :save
attribute :session_id
attribute :views, :default => 0
attribute :created_at, :default => lambda { Time.zone.now }
# iterate through attr keys and set instance vars
def initialize(attr = {})
#errors = ActiveModel::Errors.new(self)
unless attr.nil?
attr.each do |name, value|
setter = "#{name}="
next unless respond_to?(setter)
send(setter, value)
end
end
end
def addOrUpdate(key, value)
self[key] = value
end
def save
return false unless valid?
run_callbacks :save do
Couch.client.set(self.session_id, self)
end
true
end
def self.find(key)
return nil unless key
begin
doc = Couch.client.get(key)
self.new(doc)
rescue Couchbase::Error::NotFound => e
nil
end
end
end
Why don't you like to use find, save and create methods from couchbase-model gem?
class Couchbase::Error::RecordInvalid < Couchbase::Error::Base
attr_reader :record
def initialize(record)
#record = record
errors = #record.errors.full_messages.join(", ")
super("Record Invalid: #{errors}")
end
end
class SomeRequest < Couchbase::Model
include ActiveModel::Validations
attribute :session_id
attribute :views, :default => 0
attribute :created_at, :default => lambda { Time.zone.now }
validates_presence_of :session_id
before_save do |doc|
if doc.valid?
doc
else
raise Couchbase::Error::RecordInvalid.new(doc)
end
end
def initialize(*args)
#errors = ActiveModel::Errors.new(self)
super
end
end
And you might be right, it worth to add validation hooks by default, I think I will do it in next release. The example above is valid for release 0.3.0
What considering updateOrAdd I recommend you just use method #save and it will check if the key is persisted (currently by checking id attribute) and if the record doesn't have key yet, it will generate key and update it.
Update
In version 0.4.0 I added validation hooks into the gem, so the example above could be rewritten simpler.
class SomeRequest < Couchbase::Model
attribute :session_id
attribute :views, :default => 0
attribute :created_at, :default => lambda { Time.zone.now }
validates_presence_of :session_id
end

Show only exceptions messages to user

I've created this code to show specific errors messages to user:
application_controller.rb
class ApplicationController < ActionController::Base
rescue_from Exception do |exception|
message = exception.message
message = "default error message" if exception.message.nil?
render :text => message
end
end
room_controller.rb
class RoomController < ApplicationController
def show
#room = Room.find(params[:room_id]) # Can throw 'ActiveRecord::RecordNotFound'
end
def business_method
# something
raise ValidationErros::BusinessException("You cant do this") if something #message "You cant do this" should be shown for user
#...
end
def business_method_2
Room.find(params[:room_id]).do_something
end
end
room.rb
class Room < ActiveRecord::Base
def do_something
#...
raise ValidationErrors::BusinessException("Invalid state for room") if something #message "Invalid state for room" should be shown for user
#...
end
end
app/models/erros/validation_errors.rb
module ValidationErrors
class BusinessException < RuntimeError
attr :message
def initialize(message = nil)
#message = message
end
end
end
example.js
$.ajax({
url: '/room/show/' + roomId,
success: function(data){
//... do something with data
},
error: function(data){
App.notifyError(data) //show dialog with message
}
});
But I can not use the class BusinessException. When BusinessException should be raised,
the message
uninitialized constant Room::ValidationErrors
is shown to user.
if I change this code:
raise ValidationErrors::BusinessException("Invalid state for room") if something
by this:
raise "Invalid state for room" if something
It works.
What change to this code works with BusinessException with messages. I need this
to create specifics rescue_from methods in ApplicationController.
Edit:
Thank you for comments!
My error is it doesn't know ValidationErrors Module. How to import this Module to my class?
I've tested add theses lines to lines:
require 'app/models/errors/validation_errors.rb'
require 'app/models/errors/validation_errors'
But then raise the error:
cannot load such file -- app/models/errors/validation_errors
Solution:
https://stackoverflow.com/a/3356843/740394
config.autoload_paths += %W(#{config.root}/app/models/errors)
raise ::ValidationErrors::BusinessException("Invalid state for room")

Resources