I have a form through which I am uploading images. The workflow is, that I will upload and save an image and then I run a delayed job, that will take this image and will create from it 3 thumbs.
When I refresh the page, I usually see that the thumbs are not created yet, but after a while (10-15 seconds) are the thumbs ready.
But this is not very friendly - I would like to show to a user that his thumb is in progress of creating, but how to do that?
The structure of the delayed_jobs table is like this:
create_table "delayed_jobs", :force => true do |t|
t.integer "priority", :default => 0, :null => false
t.integer "attempts", :default => 0, :null => false
t.text "handler", :null => false
t.text "last_error"
t.datetime "run_at"
t.datetime "locked_at"
t.datetime "failed_at"
t.string "locked_by"
t.string "queue"
t.datetime "created_at", :null => false
t.datetime "updated_at", :null => false
end
I think the way to go could be to add to this table a column, like photo_id and simply if the respective row would be missing in this table, then I would know that the thumb is ready. Otherwise is in a queue/in progress of creating.
But how to save into this table this extra column? When I call the delay method, it's just like this:
#user.delay.activate!(#device)
How to pass there an ID of a photo?
Thanks
Rather than storing the photo_id in the delayed_jobs table, you can store the job id in the photos (or users) table. For this you'll need to write a Job class like so:
class PhotoThumbnailJob < Struct.new(:user_id)
def perform
# generate the thumbnail
end
end
Then get a reference to the job instance like this:
job = Delayed::Job.enqueue(PhotoThumbnailJob.new(#user.id))
#user.thumbnail_job_id = job.id
Now as long as you have the #user you can get the job. That way you can tell whether it failed or simply hasn't finished yet, and if it failed you can report on the reason, etc.
Note that if you do this, you might want to create a foreign key constraint on thumbnail_job_id and tell it to null that column when the job is deleted, because by default DJ will delete successful jobs from the delayed_jobs table.
I had a need for a similar capability in an application I built. The approach I took was to use WebSockets to send status updates to the user as my DelayedJob jobs progress. I used Pusher because the API is straightforward and it was free for us, but any implementation, including your own, would do.
Otherwise, I would stay very far way from altering the table and instead utilize the various hooks DelayedJob provides to enable a callback capability for you to take whatever action you prefer at various stages of the job.
Related
I haven't used webhooks before so I wanted to understand how it works before implementing the listener with a rails app.
SendGrid docs say the webhook can notify my URL when an email is delivered.
I assume I don't want to store each notification as a new row in a database, because if we're sending millions of emails, that can quickly build up.
Should I be implementing a listener that increments a counter? For example:
create_table "campaign", force: :cascade do |t|
t.text "name"
t.integer "delivered"
t.integer "failed"
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
end
Whenever a new notification is received via the webhook:
campaign.delivered += 1
campaign.save
Is a counter the correct way to handle webhooks?
Edit: I just read in the docs:
It is possible to see duplicate events in the data posted by the Event
Webhook.
We recommend that you use some form of deduplication when processing
or storing your Event Webhook data using the sg_event_id as a
differentiator, since this ID is unique for every event.
Does this mean I must save each event as a database row with a sg_event_id?
(I replicated an isolated example of my issue on github: https://github.com/diingo/jobs_emps. It contains just the problematic portion - no views, no controllers, just the models I describe below and the breaking test.)
My code is working on development and production but currently a portion is breaking only in the test environment leaving me unable to test it properly.
Here's what's happening:
I have two models, jobs and employees. Each job has employees that oversee and participate in it as a specific role (employee_type) - Manager or Exective. Employees can be promoted - a manager can be promoted to an executive. But their roles for previous jobs they participated in must remain the same.
A join between employees and jobs (JobsEmployee) keeps track of the employee's role through the employee_type attribute. The join will not update a user's position if it was previously set - this is done with a before_save, as seen here:
class JobsEmployee < ActiveRecord::Base
before_save :set_user_type
def set_user_type
self.user_type ||= self.user.type
end
end
This works fine in actual use on development and production. If a job is created, with a manager and exective, job.jobs_employees will show one manager and one executive. If a manager is promoted to executive and that job is then updated for whatever reason, job.jobs_employees will still show one manager and one executive.
However, during testing this changes. Upon updating a job, if an employee was promoted, job.jobs_employees shows two executives.
My test is shown below. You can see I abstracted controller create and update methods into models for convenience. You can reference them on the github link: https://github.com/diingo/jobs_emps/blob/master/app/models/job.rb
RSpec.describe JobsEmployee, :type => :model do
before do
#job_permitted_params = {
city: "Munich",
status: "in_progress"
}
#manager = Employee.create!(name: "Bob Bobber", type: 'Manager')
#executive = Employee.create!(name: "Alice Smith", type: 'Executive')
#job_raw_params = {
job: {
manager_id: #manager.id,
executive_id: #executive.id
}
}
end
it "creates and updates" do
job = Job.create_with_params(#job_permitted_params, #job_raw_params)
# This passes:
expect(job.jobs_employees.map &:employee_type).to include("Manager", "Executive")
#manager.type = 'Executive'
#manager.save!
Job.update_with_params(job, #job_permitted_params, #job_raw_params)
# This breaks in testing but works in production:
expect(job.jobs_employees.map &:employee_type).to include("Manager", "Executive")
end
end
I put break points (pry debugger) in JobsEmployee#set_user_type to see what might be happening. It appears like the record in JobsEmployee are deleted before or during a Job update. So instead of seeing that self.user_type is already set in self.user_type ||= self.user.type, it just runs self.user.type again.
Here is the schema. You can also see it in the github link.
ActiveRecord::Schema.define(version: 20150301232938) do
create_table "employees", force: true do |t|
t.string "name"
t.string "type"
t.datetime "created_at"
t.datetime "updated_at"
end
create_table "jobs", force: true do |t|
t.string "city"
t.string "status"
t.datetime "created_at"
t.datetime "updated_at"
end
create_table "jobs_employees", force: true do |t|
t.string "employee_type"
t.integer "employee_id"
t.integer "job_id"
t.datetime "created_at"
t.datetime "updated_at"
end
end
I'm not sure why this is happening. Thanks so much for checking out the problem.
The "actual use" you are describing may not correlate with your test. Updating a job and changing an employee's role would not necessarily cause the wholesale reassignment of jobs_employees association that you are doing in your test. I suspect that when you are doing that, Rails is comparing the set of JobsEmployee records represented by the ids you are assigning to the currently associated records. Since the currently associated records have the user_type field set, they aren't equivalent to the records that would be created upon a fresh assignment, so they are deleted and new ones regenerated.
I have an application that has tens of thousands of snapshot records. A very small number of these 'snapshots' (say 1 in 1000) will have one or more 'positions' through a :has_many association.
How can I efficiently discover if a snapshot has a position without firing an active record query for each snapshot? My current solution is to add a boolean field to snapshots - if a snapshot has a position, 'has_position' is set to true. This seems a little messy since it means I have to modify the associated snapshot every time I create a position. Is there a cleaner way to handle this scenario?
create_table "snapshots", :force => true do |t|
t.datetime "created_at",
t.datetime "updated_at",
t.boolean "has_position",
end
create_table "positions", :force => true do |t|
t.integer "snapshot_id"
t.datetime "created_at",
t.datetime "updated_at",
end
What will happen if you generate the migration for positions with the reference to snapshots, the migration file will be generated with a
add_index :positions, :snapshot_id
appended to the end of it.
With an index on snapshot_id the DB will take log(n) queries to figure out whether or not a position has at least one associated record. Not as good as constant time with the boolean, but with mere tens of thousands of records it shouldn't take noticeably longer (unless you're doing this very, very frequently).
Additionally, a simple has_position boolean might be harder than you think to maintain without an index. You can set it to true on creation of an associated position, but you can't set it to false on the deletion because there might exist another one, and you'd have to do a table scan again.
If for some reason using an index is undesirable (or you really need constant time lookup), then I'd recommend using a :counter_cache column.
I have just changed a column (called time) from t.string to t.datetime and then dropped and re-created the database and run the migration.
I have a script set to run every minute that scrapes information from a remote website and then adds the information to new records based on the time column that I adjusted to be a datetime rather than string.
# Add each row to a new call record
page = agent.page.search("table tbody tr").each do |row|
next if (!row.at('td'))
time, source, destination, duration = row.search('td').map{ |td| td.text.strip }
call = Call.find_or_create_by_time(time)
call.update_attributes({:time => time, :source => source, :destination => destination, :duration => duration})
end
Since changing the time column to integer the script doesn't seem to be importing any information at all. I wondered if there is an extra step that I need to do to make this work again?
My schema looks like this:
create_table "calls", :force => true do |t|
t.string "source"
t.string "duration"
t.datetime "time"
t.string "destination"
t.string "recording"
t.string "cost"
t.datetime "created_at"
t.datetime "updated_at"
end
In this part
time, source, destination, duration = row.search('td').map{ |td| td.text.strip }
call = Call.find_or_create_by_time(time)
you get time variable as a string, and trying to find_by it. I think smth like
call = Call.find_or_create_by_time(Time.parse(time))
should do the trick
This might seem like a duplicate question, but I can't find any information on this. I want to show the results from a remotely acquired json array excluding certain results by comparing them to a local table. I have a gallery model with:
t.integer :smugmug_id
t.string :smugmug_key
t.integer :category_id
t.string :category_name
t.string :description
t.integer :highlight_id
t.string :highlight_key
t.string :highlight_type
t.string :keywords
t.string :nicename
t.integer :subcategory_id
t.string :subcategory_name
t.string :title
t.string :url
The data for this model gets populated by a rake task that connects to the smugmug api (json) and stores the data locally. I'm trying to create a view that shows all the smugmug galleries that are not stored locally.
Here's what I've tried so far, but it's not excluding the locally stored galleries like I thought it would.
def self.not_stored
smugmug_list = Smug::Client.new.albums(heavy = true)
gallery_list = Gallery.select(:smugmug_id)
smugmug_list.each do |smugmug|
smugmug unless gallery_list.include? smugmug.id
end
end
Hopefully this makes sense. I'm getting a json array of galleries, and I want to display that array excluding results where the album id matches the smugmug_id of any of my locally stored records.
Quick edit: I'm using an adaptation of this gem to connect to the smugmug api.
Just use the difference operator.
General Example:
ruby-1.9.2-p136 :001 > [3,2,1] - [2,1]
=> [3]
So you would have:
smugmug_list.collect{|e| e.id} - gallery_list
Enumerable#collect will turn the smugmug_list into a list of id's. From there, you can do the difference operator, which will return all the id's of all the smugmug galleries that are not stored locally.
Another option to maintain the list of galleries:
smugmug_list.select{|e|!gallery_list.include?(e.id)}