I use rails 4.2.5 and Sidekiq for background processing.
There is an API which an application can call.
I now have this code:
def start_item(name, init_query)
job_id = AzureBufferBase.delay.execute_in_transaction(name, init_query)
job_id
end
I get a job_id back like this: ef95bdd9cf5da0ef1273db6c
Now I want to expose this status through the API:
module Api
class BackgroundJobsController < BaseApiController
def show
result = Sidekiq::Status(params[:id])
render json: { 'status' => result.to_json }, status: 200
end
end
end
Sidekiq::Status: this doesn't work, but my question is, how can I get the status from Active Job of a job (queued, progress, completed, ...)?
It seems like you're looking for Active Job Status gem.
Related
In my Rails 6 API only app I've got FetchAllProductsWorker background job which takes around 1h30m.
module Imports
class FetchAllProductsWorker
include Sidekiq::Worker
sidekiq_options queue: 'imports_fetch_all'
def perform
# do some things
end
end
end
During this time the frontend app sends requests to the endpoint on BE which checks if the job is still running. I need to send true/false of that process. According to the docs there is a scan method - https://github.com/mperham/sidekiq/wiki/API#scan but none of these works for me even when worker is up and running:
# endpoint method to check sidekiq status
def status
ss = Sidekiq::ScheduledSet.new
render json: ss.scan('FetchAllProductsWorker') { |job| job.present? }
end
The console shows me:
> ss.scan("\"class\":\"FetchAllProductsWorker\"") {|job| job }
=> nil
> ss.scan("FetchAllProductsWorker") { |job| job }
=> nil
How to check if particular sidekiq process is not finished?
Maybe this will be useful for someone. Sidekiq provides programmatic access to the current active worker using Sidekiq::Workers https://github.com/mperham/sidekiq/wiki/API#workers
So based on that we could do something like:
active_workers = Sidekiq::Workers.new.map do |_process_id, _thread_id, work|
work
end
active_workers.select do |worker|
worker['queue'] == 'imports_fetch_all'
end.present?
I am building a web service that accepts a string, parses it and returns it as JSON. In my controller, I am calling an ActiveJob to run a service to parse the data. I would like to return the results of the ActiveJob back to my controller in order to return it as JSON and I am not sure how to do it. What is the simplest way to do this?
class GeocoderService
require 'ruby_postal/parser'
def initialize(address)
#address = address
end
def parse_address
address_group = {}
result = Postal::Parser.parse_address(#address)
result.each do |r|
address_group[r.values[0]] = r.values[1]
end
address_group
end
class ParseAddressJob < ApplicationJob
queue_as :default
def perform(address)
geo = GeocoderService.new(address)
result = geo.parse_address
end
end
class LocationsController < ApplicationController
def create
if geo_params[:address]
ParseAddressJob.perform_later(geo_params[:address])
render json: result
else
render json: { error: "Invalid address"}, status: 400
end
end
private
def geo_params
params.require(:geo).permit(:address)
end
end
ActiveJob is used to create tasks that run in the background, and asynchronously from the HTTP request/response flow. The controller hands off the task to ActiveJob then the controller returns while the job runs in the future. If you need the output from a job immediately because its something the user needs then you shouldn't be using a job - you should just be calling the code the job calls directly and blocking until the code is finished so you can get the output.
def create
if geo_params[:address]
geo = GeocoderService.new(geo_params[:address])
result = geo.parse_address
render json: result
else
render json: { error: "Invalid address"}, status: 400
end
end
If you're really concerned with the blocking that your controller action will do while it waits on the response from the geolocation API, you can institute a queueing system of your own, for your API consumers. The flow looks something like this:
A user makes a request to your endpoint
Your endpoint inserts a record in to a database table called GeoResults with a status of 'processing', and an empty response text. Get the ID of this record.
Your endpoint fires off the job as you're doing now, but now you also pass in the ID of the GeoResults record you created.
Your endpoint gives the user a URL to check this record in GeoResults.
Your consumer starts to poll this endpoint until they see the status of 'complete'
When your background job is completed, it updates its record in GeoResults (since it has the ID) with a status of 'complete', and assigns the geolocation response text.
Your consumer sees the update, and grabs the response.
I use rails with ActiveJob and sidekiq as backend. When user come on a page sidekiq create a long-term background task, how can I notice a user (by render partial on the web page) when a task would be completed?
Rails and sidekiq work as different processes. This fact confused me I don't understand how to handle completed status using background job.
ActiveJob provides an after_perform callback which according to docs work like this:
class VideoProcessJob < ActiveJob::Base
queue_as :default
after_perform do |job|
UserMailer.notify_video_processed(job.arguments.first)
end
def perform(video_id)
Video.find(video_id).process
end
end
So, you don't have to worry to integrate directly with Sidekiq or any other queuing backend, talk to ActiveJob :)
My approach in this situation is:
Add sidekiq-status so that background jobs can be tracked by ID.
In the client call that creates the background job, return the newly-created job's ID.
class MyController < ApplicationController
def create
# sidekiq-status lets us retrieve a unique job ID when
# creating a job
job_id = Workers::MyJob.perform_async(...)
# tell the client where to find the progress of this job
return :json => {
:next => "/my/progress?job_id={job_id}"
}
end
end
Poll a 'progress' endpoint on the server with that job ID. This endpoint fetches job progress information for the job and returns it to the client.
class MyController < ApplicationController
def progress
# fetch job status from sidekiq-status
status = Sidekiq::Status::get_all(params[:job_id])
# in practice, status can be nil if the info has expired from
# Redis; I'm ignoring that for the purpose of this example
if status["complete"]
# job is complete; notify the client in some way
# perhaps by sending it a rendered partial
payload = {
:html => render_to_string({
:partial => "my/job_finished",
:layout => nil
})
}
else
# tell client to check back again later
payload = {:next => "/my/progress?job_id={params[:job_id]}"}
end
render :json => payload
end
end
If the client sees that the job has completed, it can then display a message or take whatever next step is required.
var getProgress = function(progress_url, poll_interval) {
$.get(progress_url).done(function(progress) {
if(progress.html) {
// job is complete; show HTML returned by server
$('#my-container').html(progress.html);
} else {
// job is not yet complete, try again later at the URL
// provided by the server
setTimeout(function() {
getProgress(progress.next, poll_interval);
}, poll_interval);
}
});
};
$("#my-button").on('click', function(e) {
$.post("/my").done(function(data) {
getProgress(data.next, 5000);
});
e.preventDefault();
});
Caveat emptor: that code is meant to be illustrative, and is missing things you should take care of such as error handling, preventing duplicate submissions, and so forth.
In my Rails app, I'm trying to take my working API calls and have them handled by background workers.
I have the following in app/jobs/api_request_job.rb:
class ApiRequestJob
def self.perform(params)
Query.new(params).start
end
end
The Query class is where the HTTParty requests are being executed (there are lots of methods for different query types with the same basic format as the parks method:
require 'ostruct'
class Query
include FourSquare
attr_reader :results,
:first_address,
:second_address,
:queries,
:radius
def initialize(params)
#results = OpenStruct.new
#queries = params["query"]
#first_address = params["first_address"]
#second_address = params["second_address"]
#radius = params["radius"].to_f
end
def start
queries.keys.each do |query|
results[query] = self.send(query)
end
results
end
def parks
category_id = "4bf58dd8d48988d163941735"
first_address_results = FourSquare.send_request(#first_address, radius_to_meters, category_id)["response"]["venues"]
second_address_results = FourSquare.send_request(#second_address, radius_to_meters, category_id)["response"]["venues"]
response = [first_address_results, second_address_results]
end
And, finally, the controller. Before trying to farm this action out to background workers, this line was working fine: #results = Query.new(params).start
class ComparisonsController < ApplicationController
attr_reader :first_address, :second_address
def new
end
def show
#first_address = Address.new(params["first_address"])
#second_address = Address.new(params["second_address"])
if #first_address.invalid?
flash[:notice] = #first_address.errors.full_messages
redirect_to :back
elsif Query.new(params).queries.nil?
flash[:notice] = "You must choose at least one criteria for your comparison."
redirect_to comparisons_new_path(request.params)
else
#queries = params["query"].keys
#results = Resque.enqueue(ApiRequestJob, params) # <-- this is where I'm stuck
end
end
end
I'm running redis, have resque installed, and am running the task/starting the workers. The current value being returned for #results is true instead of the hash of results I was need to get back. Is there a way to have the results of the Resque job persist and return data instead of true? What am I missing about how to get background workers to return the same type of data my regular api calls were returning?
Many thanks in advance!
The true you are receiving means the job was scheduled enqueued successfully. The worker will pick it up and run it on the background asynchronously, which means, not at same time as the thread that enqueued it. So there's no way to retrieve the returned value from the job.
If you need the value from that process, you have to run it from the controller without the worker. Also, you wouldn't gain anything from just pushing the work to be done by another process as the web process would have to wait for the response to then keep going anyway.
If you need that returned value right away and are doing this for performance reasons, then you could look into other forms of concurrency, like having another thread doing the request and then only grabbing the result when you need it on the view like:
class AsyncValue
def initialize(&block)
#thr = Thread.new(&block)
end
def value
#thr.join
end
end
on the controller
#results = AsyncValue.new { Query.new(params).start }
and on the view
<%= #results.value.each .... %>
but you'd still have to work around error handling which can get pretty complicated, but is doable.
Personally, I'd just make the request in place, but you know your domain better than me.
I have a "Status" page. On which I am displaying the status of my local machines such as current upload/download speed, is recording going or not.
For getting above information I am ssh into that machine. Here is my sample code for it
Net::SSH.start('localhost','ubuntu', :password => 'ubuntu') do |session|
upload_speed = session.exec!("speedtest | grep Upload:").chomp.strip
return upload_speed
end
But it is taking time (about 3-4 minutes) for fetching those status. And it returns me "Connection time out error". So I am trying to add this process in the background. For this I am using delayed_job gem
Here is my code for it
My controller method
def unit_additional_status
#machine = MachineInfo.find(params[:unit_id])
stat = Delayed::Job.enqueue(LongerTask.new(#machine), 3, :run_at => 1.seconds.from_now)
end
Here is my longer_task.rb file
require 'rubygems'
require 'net/ssh'
class LongerTask < Struct.new(:machine)
def perform
port = #machine.port
#status = Hash.new
Net::SSH.start('localhost','ubuntu', :password => 'ubuntu', :port => port) do |session|
upload_speed = session.exec!("speedtest | grep Upload:").chomp.strip
status["upload_speed"].push(upload_speed)
end
#status
end
end
After execution I have to pass this #status to my controller action so that I can pass it to my status.html.erb view.
So I have a question how can I pass it to my controller method or how can get the output of execution of delayed job.
Also, if any one have better solution then let me know.
I am using rails 3.2.14 and ruby 1.8.7
You need to create some kind of additional status model, e.g. Job (status:string, message:string). Then you pass an instance of this model to your delayed job task when it is scheduled. When the task starts executing, you set the status to 'running'. When it finishes you update the message field with the desired result information and set status to 'finished'. This has several benefits like you have a good overview of your job queue and it can be extended to reflect execution time, errors etc.
To display the machine status in your example, you simply select the latest Job with status='finished' and show its timestamp and message.