I want to listening SerialPort and when message occurs then get or create Log model with id received from my device.
How to load once automatically SerialPort and keep established connection and if key_detected? in listener deal with Log model?
This is my autoloaded module in lib:
module Serialport
class Connection
def initialize(port = "/dev/tty0")
port_str = port
baud_rate = 9600
data_bits = 8
stop_bits = 1
parity = SerialPort::NONE
#sp = SerialPort.new(port_str, baud_rate, data_bits, stop_bits, parity)
#key_parts = []
#key_limit = 16 # number of slots in the RFID card.
while true do
listener
end
#sp.close
end
def key_detected?
#key_parts << #sp.getc
if #key_parts.size >= #key_limit
self.key = #key_parts.join()
#key_parts = []
true
else
false
end
end
def listener
if key_detected?
puts self.key
# log = Log.find(rfid: self.key).first_or_create(rfid: self.key)
end
end
end
end
Model:
class Log < ActiveRecord::Base
end
I would have written this in a comment, but it's a bit long... But I wonder if you could clarify your question, and I will update my answer as we go:
With all due respect to the Rails ability to "autoload", why not initialize a connection in an initialization file or while setting up the environment?
i.e., within a file in you_app/config/initializers called serial_port.rb:
SERIAL_PORT_CONNECTION = Serialport::Connection.new
Implementing an infinite loop within your Rails application will, in all probability, hang the Rails app and prevent it from being used as a web service.
What are you trying to accomplish?
If you just want to use active_record or active_support, why not just include these two gems in a separate script?
Alternatively, consider creating a separate thread for the infinite loop (or better yet, use a reactor (They are not that difficult to write, but there are plenty pre-written in the wild, such as Iodine which I wrote for implementing web services)...
Here's an example for an updated listener method, using a separate thread so you call it only once:
def listener
Thread.new do
loop { self.key while key_detected? }
# this will never be called - same as in your code.
#sp.close
end
end
Related
I'm trying to make a Rails 4 application that makes a lot of http requests to some API handle more traffic, originally the code in the controller looks like this:
def index
#var1 = api_call some_params1
#var2 = api_call some_params2
#var3 = api_call some_params3
#var4 = api_call some_params4
#var5 = api_call some_params5
end
I did some googling around and ended up refactoring it as so:
def index
#var1 = Thread.new { api_call some_params1 }.value
#var2 = Thread.new { api_call some_params2 }.value
#var3 = Thread.new { api_call some_params3 }.value
#var4 = Thread.new { api_call some_params4 }.value
#var5 = Thread.new { api_call some_params5 }.value
end
Am I doing this right? Or am I instead supposed to call join on those threads somewhere?
Is this safe for production or is there something I should be tweaking, maybe in the Nginx or passenger configs?
Am I doing this right?
There are no issues in your code but I don't think that using threads makes a lot of sense in your code example since you're executing requests one after another anyway.
If you want to make parallel requests then you should do it like this instead:
threads = [params1, params2, ...].map { |p| Thread.new { api_call(p) } }
values = threads.map(&:value)
Am I doing this right? Or am I instead supposed to call join on those threads somewhere?
Both join and value calls will wait for a thread to finish but value is more convenient for you there if you want to retrieve a value returned from a thread. value is using join under the hood.
Is this safe for production or is there something I should be tweaking, maybe in the Nginx or passenger configs?
You don't need to tweak anything to use threads and it is generally safe to use them in production (if you're using MRI then GIL prevents deadlocks). You just need to be aware that if you're using a lot of threads then you'll be using a lot of extra memory. And using threads don't always improve performance of a program. For example, due to GIL there is not much point in using threads for executing CPU-intensive code even on a multicore machine.
I have a Rails 5 application using raven-ruby to send exceptions to Sentry which then sends alerts to our Slack.
Raven.configure do |config|
config.dsn = ENV['SENTRY_DSN']
config.environments = %w[ production development ]
config.excluded_exceptions += []
config.async = lambda { |event|
SentryWorker.perform_async(event.to_hash)
}
end
class SentryWorker < ApplicationWorker
sidekiq_options queue: :default
def perform(event)
Raven.send_event(event)
end
end
It's normal for our Sidekiq jobs to throw exceptions and be retried. These are mostly intermittent API errors and timeouts which clear up on their own in a few minutes. Sentry is dutifully sending these false alarms to our Slack.
I've already added the retry_count to the jobs. How can I prevent Sentry from sending exceptions with a retry_count < N to Slack while still alerting for other exceptions? An example that should not be alerted will have extra context like this:
sidekiq: {
context: Job raised exception,
job: {
args: [{...}],
class: SomeWorker,
created_at: 1540590745.3296254,
enqueued_at: 1540607026.4979043,
error_class: HTTP::TimeoutError,
error_message: Timed out after using the allocated 13 seconds,
failed_at: 1540590758.4266324,
jid: b4c7a68c45b7aebcf7c2f577,
queue: default,
retried_at: 1540600397.5804272,
retry: True,
retry_count: 2
},
}
What are the pros and cons of not sending them to Sentry at all vs sending them to Sentry but not being alerted?
Summary
An option that has worked well for me is by configuring Sentry's should_capture alongside Sidekiq's sidekiq_retries_exhausted with a custom attribute on the exception.
Details
1a. Add the custom attribute
You can add a custom attribute to an exception. You can define this on any error class with attr_accessor:
class SomeError
attr_accessor :ignore
alias ignore? ignore
end
1b. Rescue the error, set the custom attribute, & re-raise
def perform
# do something
rescue SomeError => e
e.ignore = true
raise e
end
Configure should_capture
should_capture allows you to capture exceptions when they meet a defined criteria. The exception is passed to it, on which you can access the custom attribute.
config.should_capture { |e| !e.ignore? }
Flip the custom attribute when retries are exhausted
There are 2 ways to define the behaviour you want to happen when a job dies, depending on the version of Sidekiq being used. If you want to apply globally & have sidekiq v5.1+, you can use a death handler. If you want to apply to a particular worker or have less than v5.1, you can use sidekiq_retries_exhausted.
sidekiq_retries_exhausted { |_job, ex| ex.ignore = false }
You can filter out the entire event if the retry_count is < N (can be done inside that sidekiq worker you posted). You will loose the data on how often this happens without alerting, but the alerts themselves will not be too noisy.
class SentryWorker < ApplicationWorker
sidekiq_options queue: :default
def perform(event)
retry_count = event.dig(:extra, :sidekiq, :job, retry_count)
if retry_count.nil? || retry_count > N
Raven.send_event(event)
end
end
end
Another idea is to set a different fingerprint depending on whether this is a retry or not. Like this:
class MyJobProcessor < Raven::Processor
def process(data)
retry_count = event.dig(:extra, :sidekiq, :job, retry_count)
if (retry_count || 0) < N
data["fingerprint"] = ["will-retry-again", "{{default}}"]
end
end
end
See https://docs.sentry.io/learn/rollups/?platform=javascript#custom-grouping
I didn't test this, but this should split up your issues into two, depending on whether sidekiq will retry them. You can then ignore one group but can still look at it whenever you need the data.
A much cleaner approach if you are trying to ignore exceptions belonging to a certain class is to add them to your config file
config.excluded_exceptions += ['ActionController::RoutingError', 'ActiveRecord::RecordNotFound']
In the above example, the exceptions Rails uses to generate 404 responses will be suppressed.
See the docs for more configuration options
From my point of view, the best option is Sentry holds all the exceptions and you could modify Sentry and set alerts to send or not the exceptions to the Slack.
In order to configure the Alerts in Sentry: In the sentry account, you could go to the ALerts option in the main menu.
In the following picture I configure an alert to only send to slack a notification if occurs an Exception of type ControllerException more than 10 times
Using this alert we only receive the notification in Slack when all conditions are accomplished
I want to implement an action that calls remote service with RabbitMQ and presents returned data. I implemented this (more as a proof of concept so far) in similar way to example taken from here: https://github.com/baowen/RailsRabbit and it looks like this:
controller:
def rpc
text = params[:text]
c = RpcClient.new('RPC server route key')
response = c.call text
render text: response
end
RabbitMQ RPC client:
class RpcClient < MQ
attr_reader :reply_queue
attr_accessor :response, :call_id
attr_reader :lock, :condition
def initialize()
# initialize exchange:
conn = Bunny.new(:automatically_recover => false)
conn.start
ch = conn.create_channel
#x = ch.default_exchange
#reply_queue = ch.queue("", :exclusive => true)
#server_queue = 'rpc_queue'
#lock = Mutex.new
#condition = ConditionVariable.new
that = self
#reply_queue.subscribe do |_delivery_info, properties, payload|
if properties[:correlation_id] == that.call_id
that.response = payload.to_s
that.lock.synchronize { that.condition.signal }
end
end
end
def call(message)
self.call_id = generate_uuid
#x.publish(message.to_s,
routing_key: #server_queue,
correlation_id: call_id,
reply_to: #reply_queue.name)
lock.synchronize { condition.wait(lock) }
response
end
private
def generate_uuid
# very naive but good enough for code
# examples
"#{rand}#{rand}#{rand}"
end
end
A few tests indicate that this approach works. On the other hand, this approach assumes creating a client (and subscribing to the queue) for every request on this action, which is inefficient according to the RabbitMQ tutorial. So I've got two questions:
Is it possible to avoid creating a queue for every Rails request?
How will this approach (with threads and mutex) interfere with my whole Rails environment? Is it safe to implement things this way in Rails? I'm using Puma as my web server, if it's relevant.
Is it possible to avoid creating a queue for every Rails request?
Yes - there is no need for every single request to have it's own reply queue.
You can use the built-in direct-reply queue. See the documentation here.
If you don't want to use the direct-reply feature, you can create a single reply queue per rails instance. You can use a single reply queue, and have the correlation id help you figure out where the reply needs to go within that rails instance.
How will this approach (with threads and mutex) interfere with my whole Rails environment? Is it safe to implement things this way in Rails?
what's the purpose of the lock / mutex in this code? doesn't seem necessary to me, but i'm probably missing something since i haven't done ruby in about 5 years :)
I've got a question about how to share rspec-mocks' double between examples. I'm writing a new rails app with rspec-mocks 3.1.3. I'm used to using the old (< 2.14 and and trying to update my knowledge if current rspec usage.
I have a model method:
def self.from_strava(activity_id, race_id, user)
#client ||= Strava::Api::V3::Client.new(access_token: 'abc123')
activity = #client.retrieve_an_activity(activity_id)
result_details = {race_id: race_id, user: user}
result_details[:duration] = activity['moving_time']
result_details[:date] = Date.parse(activity['start_date'])
result_details[:comment] = activity['description']
result_details[:strava_url] = "http://www.strava.com/activities/#{activity_id}"
Result.create!(result_details)
end
And here is the spec:
describe ".from_strava" do
let(:user) { FactoryGirl.build(:user) }
let(:client) { double(:client) }
let(:json_response) { JSON.parse(File.read('spec/support/strava_response.json')) }
before(:each) do
allow(Strava::Api::V3::Client).to receive(:new) { client }
allow(client).to receive(:retrieve_an_activity) { json_response }
allow(Result).to receive(:create!)
end
it "sets the duration" do
expect(Result).to receive(:create!).with(hash_including(duration: 3635))
Result.from_strava('123', 456, user)
end
it "sets the date" do
expect(Result).to receive(:create!).with(hash_including(date: Date.parse("2014-11-14")))
Result.from_strava('123', 456, user)
end
end
When I run a single test on it's own it's fine, but when I run the whole describe ".from_strava" block it fails with the message
Double :client was originally created in one example but has leaked into another example and can no longer be used. rspec-mocks' doubles are designed to only last for one example, and you need to create a new one in each example you wish to use it for.
I understand what it's saying, but surely this is an appropriate use of a double being used in 2 examples. After all, the client double isn't important to the example, it's just a way for me to load the canned response. I guess I could use WebMock but that seems very low-level and doesn't translate well to the actual code written. We should only be asserting one thing per example after all.
I had thought about replacing the client double with a call to
allow(Strava::Api::V3::Client).to receive_message_chain(:new, :retrieve_an_activity) { json_response }
but that doesn't seem to be the right approach either, given that the documentation states that receive_message_chain should be a code smell.
So if I shouldn't use receive_message_chain, shared client double and also follow the standard DRY principle then how should I fix this?
I would love some feedback on this.
Thanks,
Dave
Caching clients for external components can often be really desired (keeping alive connections/any SSL setup that you might need, etc.) and removing that for the sake of fixing an issue with tests is not a desirable solution.
In order to fix your test (without refactoring your code), you can do the following to clear the instance variable after each of your tests:
after { Result.instance_variable_set("#client", nil) }
While admittedly, this is not the cleanest solution, it seems to be the simplest and achieves both, lets you have a clear setup with no state shared in between tests, and keep your client cached in "normal" operation mode.
surely this is an appropriate use of a double being used in 2 examples.
No, it's not. :) You're trying to use a class variable; do not do that because the variable doesn't span examples. The solution is to set the client each time i.e. in each example.
Bad:
#client ||= Strava::Api::V3::Client.new(access_token: 'abc123')
Good:
#client = Strava::Api::V3::Client.new(access_token: 'abc123')
I had the same use case in an app of mine, and we solved it by extracting the cacheing into a private method and then stubbing that method to return the double (instead of stubbing the new method directly).
For example, in the class under test:
def self.from_strava(activity_id, race_id, user)
activity = strava_client.retrieve_an_activity(activity_id)
...
end
private
def self.strava_client
#client ||= Strava::Api::V3::Client.new(access_token: 'abc123')
end
And in the spec:
let(:client) { double(:client) }
before { allow(described_class).to receive(:strava_client).and_return(client) }
...
TLDR: Add after { order.vendor_service = nil } to balance the before block. Or read on...
I ran into this, and it was not obvious where it was coming from. In order_spec.rb model tests, I had this:
describe 'order history' do
before do
service = double('VendorAPI')
allow(service).to receive(:order_count).and_return(5)
order.vendor_service = service
end
# tests here ..
end
And in my Order model:
def too_many_orders?
##vendor_service ||= VendorAPI.new(key: 'abc', account: '123')
return ##vendor_service.order_count > 10
end
This worked fine when I only ran rspec on order_spec.rb
I was mocking something completely different in order_controller_spec.rb a little differently, using allow_any_instance_of() instead of double and allow:
allow_any_instance_of(Order).to receive(:too_many_orders?).and_return(true)
This, too, tested out fine.
The confounding trouble is that when I ran the full suite of tests, I got the OP's error on the controller mock -- the one using allow_any_instance. This was very hard to track down, as the problem (or at least my solution) lay in the model tests where I use double/allow.
To fix this, I added an after block clearing the class variable ##vendor_service, balancing the before block's action:
describe 'order history' do
before do
service = double('VendorAPI')
allow(service).to receive(:order_count).and_return(5)
order.vendor_service = service
end
after do
order.vendor_service = nil
end
# tests here ..
end
This forced the ||= VendorAPI.new() to use the real new function in later unrelated tests, not the mock object.
My Rails web app has dozens of methods from making calls to an API and processing query result. These methods have the following structure:
def method_one
batch_query_API
process_data
end
..........
def method_nth
batch_query_API
process_data
end
def summary
method_one
......
method_nth
collect_results
end
How can I run all query methods at the same time instead of sequential in Rails (without firing up multiple workers, of course)?
Edit: all of the methods are called from a single instance variable. I think this limits the use of Sidekiq or Delay in submitting jobs simultaneously.
Ruby has the excellent promise gem. Your example would look like:
require 'future'
def method_one
...
def method_nth
def summary
result1 = future { method_one }
......
resultn = future { method_nth }
collect_results result1, ..., resultn
end
Simple, isn't it? But let's get to more details. This is a future object:
result1 = future { method_one }
It means, the result1 is getting evaluated in the background. You can pass it around to other methods. But result1 doesn't have any result yet, it is still processing in the background. Think of passing around a Thread. But the major difference is - the moment you try to read it, instead of passing it around, it blocks and waits for the result at that point. So in the above example, all the result1 .. resultn variables will keep getting evaluated in the background, but when the time comes to collect the results, and when you try to actually read these values, the reads will wait for the queries to finish at that point.
Install the promise gem and try the below in Ruby console:
require 'future'
x = future { sleep 20; puts 'x calculated'; 10 }; nil
# adding a nil to the end so that x is not immediately tried to print in the console
y = future { sleep 25; puts 'y calculated'; 20 }; nil
# At this point, you'll still be using the console!
# The sleeps are happening in the background
# Now do:
x + y
# At this point, the program actually waits for the x & y future blocks to complete
Edit: Typo in result, should have been result1, change echo to puts
You can take a look at a new option in town: The futoroscope gem.
As you can see by the announcing blog post it tries to solve the same problem you are facing, making simultaneous API query's. It seems to have pretty good support and good test coverage.
Assuming that your problem is a slow external API, a solution could be the use of either threaded programming or asynchronous programming. By default when doing IO, your code will block. This basically means that if you have a method that does an HTTP request to retrieve some JSON your method will tell your operating system that you're going to sleep and you don't want to be woken up until the operating system has a response to that request. Since that can take several seconds, your application will just idly have to wait.
This behavior is not specific to just HTTP requests. Reading from a file or a device such as a webcam has the same implications. Software does this to prevent hogging up the CPU when it obviously has no use of it.
So the question in your case is: Do we really have to wait for one method to finish before we can call another? In the event that the behavior of method_two is dependent on the outcome of method_one, then yes. But in your case, it seems that they are individual units of work without co-dependence. So there is a potential for concurrency execution.
You can start new threads by initializing an instance of the Thread class with a block that contains the code you'd like to run. Think of a thread as a program inside your program. Your Ruby interpreter will automatically alternate between the thread and your main program. You can start as many threads as you'd like, but the more threads you create, the longer turns your main program will have to wait before returning to execution. However, we are probably talking microseconds or less. Let's look at an example of threaded execution.
def main_method
Thread.new { method_one }
Thread.new { method_two }
Thread.new { method_three }
end
def method_one
# something_slow_that_does_an_http_request
end
def method_two
# something_slow_that_does_an_http_request
end
def method_three
# something_slow_that_does_an_http_request
end
Calling main_method will cause all three methods to be executed in what appears to be parallel. In reality they are still being sequentually processed, but instead of going to sleep when method_one blocks, Ruby will just return to the main thread and switch back to method_one thread, when the OS has the input ready.
Assuming each method takes two 2 ms to execute minus the wait for the response, that means all three methods are running after just 6 ms - practically instantly.
If we assume that a response takes 500 ms to complete, that means you can cut down your total execution time from 2 + 500 + 2 + 500 + 2 + 500 to just 2 + 2 + 2 + 500 - in other words from 1506 ms to just 506 ms.
It will feel like the methods are running simultanously, but in fact they are just sleeping simultanously.
In your case however you have a challenge because you have an operation that is dependent on the completion of a set of previous operations. In other words, if you have task A, B, C, D, E and F, then A, B, C, D and E can be performed simultanously, but F cannot be performed until A, B, C, D and E are all complete.
There are different ways to solve this. Let's look at a simple solution which is creating a sleepy loop in the main thread that periodically examines a list of return values to make sure some condition is fullfilled.
def task_1
# Something slow
return results
end
def task_2
# Something slow
return results
end
def task_3
# Something slow
return results
end
my_responses = {}
Thread.new { my_responses[:result_1] = task_1 }
Thread.new { my_responses[:result_2] = task_2 }
Thread.new { my_responses[:result_3] = task_3 }
while (my_responses.count < 3) # Prevents the main thread from continuing until the three spawned threads are done and have dumped their results in the hash.
sleep(0.1) # This will cause the main thread to sleep for 100 ms between each check. Without it, you will end up checking the response count thousands of times pr. second which is most likely unnecessary.
end
# Any code at this line will not execute until all three results are collected.
Keep in mind that multithreaded programming is a tricky subject with numerous pitfalls. With MRI it's not so bad, because while MRI will happily switch between blocked threads, MRI doesn't support executing two threads simultanously and that solves quite a few concurrency concerns.
If you want to get into multithreaded programming, I recommend this book:
http://www.amazon.com/Java-Concurrency-Practice-Brian-Goetz/dp/0321349601
It's centered around Java, but the pitfalls and concepts explained are universal.
You should check out Sidekiq.
RailsCasts episode about Sidekiq.