I have a stream created in snowflake and a task which moves stream data into another table. I want the task to execute automatically every time there is new data in stream. How to automatically trigger the task when there is new data in stream?
Tasks can only triggered on a schedule but you can have them run as often as every minute
If a run takes more than 1 minute, then the next task is delayed until 1 min after the previous task finishes
While it is not "on demand" a task running every minute should suffice for most situations. Of course it means a warehouse constantly running but the same would be true for an on demand service assuming new data arrives all the time. If data is irregular, you can add a stream has data check so the task does not run if there is no data: https://docs.snowflake.com/en/sql-reference/functions/system_stream_has_data.html
Related
Debouncing is a common method to postpone a function/job from executing until after certain time has passed.
Use-case: A conversation with active chatting from multiple users, they should not receive an email notification for each message typed. But more than likely after a few minutes of silence, if the messages are unread, the user should see a notification.
Delayed_Job
Has no solution, has related issues: https://github.com/collectiveidea/delayed_job/issues/72
Sidekiq
https://github.com/hummingbird-me/sidekiq-debounce
Doing yourself is not so bad.
class AdminJob
def self.debounce(job, args={})
handler = YAML.dump(job)
count = Delayed::Job.where(handler: handler).where('locked_at IS NULL').delete_all
Rails.logger.info("deleted: #{count} jobs")
Delayed::Job.enqueue(job, args)
end
end
Instead of writing:
Delayed::Job.enqueue(YourJobName.new(account_id), {run_at: 10.minutes.from_now})
You now write:
AdminJob.debounce(YourJobName.new(account_id), {run_at: 10.minutes.from_now})
Delayed job serializes your job params in YAML and then saves it to the database as handler. So if you call AdminJob.debounce(...) 10 times in a row, it will delete before each.
Make sure to give yourself time (5.minutes, etc) to give users to keep taking actions. If you run your job after 1 second, its likely they'll keep taking actions and trigger again.
Yes i'm answering my own question 3 years later...
What I would do is schedule a periodic task, lets say every 5 minutes that checks if there is someone who has to be notified. Yes it seems an expensive operation, but for your use case I don't see (for now) others solutions. So lets say you have 10 users that uses a chat. Every 5 minutes you could check if there are users that didn't see some messages, and if so you notify them only if they are inactive from N minutes.
To schedule such task you could use crono gem. Check this answer.
Crono lets you do thing like that:
Crono.perform(CheckUsersToBeNotifiedJob).every 5.minutes
If you are using Delayed Job as the implementation for Active Job then take a look at the 'activejob-trackable' gem.
https://github.com/ignatiusreza/activejob-trackable
It uses another table that tracks the jobs and can throttle and debounce.
Currently I am working with queue job on the ruby on rail with the Sidekiq. I have 2 jobs that are depend to each other and I want 1st job to finish first before starting the 2nd job, so is there any way to make it with Sidekiq.
Yes, you can use the YourSidekiqJob.new.perform(parameters_to_the_job) pattern. This will run your jobs in order, synchronously.
However, there are 2 things to consider here:
What happens if the first job fails?
How long does the each job run?
For #2, the pattern blocks execution for the length of time each job takes to run. If the jobs are extremely short in runtime, why use the jobs in the first place? If they're long, are you expecting the user to wait until they're done?
Alternatively, you can schedule the running of the second job as the last line in the body of the first one. You still need to account for the failure mode of job #1 or #2. Also, you need to consider that the job won't necessarily run when it's scheduled to run, due to the state of the queue at schedule time. How does this affect your business logic?
Hope this helps
--edit according to last comment
class SecondJob < SidekiqJob
def perform(params)
data = SomeData.find
return unless data.ready?
# do whatever you need to do with the ready data
end
end
In my process, i need to allow a user to execute the same task ( with different params ) multiple times.
Showing TaskService class i observe that there is just complete method.
The complete method set that task to complete and user cannot execute it again.
Is there a solution to that?
Thanks!
A task can only be executed once. However, your process could allow the same task (node) being triggered multiple times, for example use an intermediate signal start event leading to your task, where depending on the data sent alongside the signal, the task will be triggered. The process could allow retriggering the same task multiple times (during a certain phase of the process for example).
See the updated question below.
Original question:
In my current Rails project, I need to parse large xml/csv data file and save it into mongodb.
Right now I use this steps:
Receive uploaded file from user, store the data into mongodb
Use sidekiq to perform async process of the data in mongodb.
After process finished, delete the raw data.
For small and medium data in localhost, the steps above run well. But in heroku, I use hirefire to dynamically scale worker dyno up and down. When worker still processing the large data, hirefire see empty queue and scale down worker dyno. This send kill signal to the process, and leave the process in incomplete state.
I'm searching a better way to do the parsing, allow the parsing process got killed anytime (saving the current state when receiving kill signal), and allow the process got re-queued.
Right now I'm using Model.delay.parse_file and it don't get re-queued.
UPDATE
After reading sidekiq wiki, I found article about job control. Can anyone explain the code, how it works, and how it preserve it's state when receiving SIGTERM signal and the worker get re-queued?
Is there any alternative way to handle job termination, save current state, and continue right from the last position?
Thanks,
Might be easier to explain the process and the high level steps, give a sample implementation (a stripped down version of one that I use), and then talk about throw and catch:
Insert the raw csv rows with an incrementing index (to be able to resume from a specific row/index later)
Process the CSV stopping every 'chunk' to check if the job is done by checking if Sidekiq::Fetcher.done? returns true
When the fetcher is done?, store the index of the currently processed item on the user and return so that the job completes and control is returned to sidekiq.
Note that if a job is still running after a short timeout (default 20s) the job will be killed.
Then when the job runs again simply, start where you left off last time (or at 0)
Example:
class UserCSVImportWorker
include Sidekiq::Worker
def perform(user_id)
user = User.find(user_id)
items = user.raw_csv_items.where(:index => {'$gte' => user.last_csv_index.to_i})
items.each_with_index do |item, i|
if (i+1 % 100) == 0 && Sidekiq::Fetcher.done?
user.update(last_csv_index: item.index)
return
end
# Process the item as normal
end
end
end
The above class makes sure that each 100 items we check that the fetcher is not done (a proxy for if shutdown has been started), and ends execution of the job. Before the execution ends however we update the user with the last index that has been processed so that we can start where we left off next time.
throw catch is a way to implement this above functionality a little cleaner (maybe) but is a little like using Fibers, nice concept but hard to wrap your head around. Technically throw catch is more like goto than most people are generally comfortable with.
edit
Also you could not make call to Sidekiq::Fetcher.done? and record the last_csv_index on each row or on each chunk of rows processed, that way if your worker is killed without having the opportunity to record the last_csv_index you can still resume 'close' to where you left off.
You are trying to address the concept of idempotency, the idea that processing a thing multiple times with potential incomplete cycles does not cause problems. (https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-jobs-idempotent-and-transactional)
Possible steps forward
Split the file up into parts and process those parts with a job per part.
Lift the threshold for hirefire so that it will scale when jobs are likely to have fully completed (10 minutes)
Don't allow hirefire to scale down while a job is working (set a redis key on start and clear on completion)
Track progress of the job as it is processing and pick up where you left off if the job is killed.
I'm looking for some good library for processing tasks (or 'operations' as we call them in our domain model) for Java or .NET. We save each operation to perform in db and then we need some mechanism for fetching unprocessed tasks from db, process them and update db record with proper status ('processed OK' / 'process error').
The trick is that operation can depend one on another. For example when 'Operation Payment' is being processed the system might discover that we need to perform 'Operation Check Payment Data' before - so it should create new operation row in db, pause performing 'Operation Payment', process 'Operation Check Payment Data' in next turn and after it completes go back to processing 'Operation Payment'.
I'll show you how we manage this at the moment.
We've got db table 'operations'. Cron-like mechanism runs each minute and fetches first 100 unprocessed operations from db and process it. If (while processing) system finds that some other operation (B) is needed to perform current operation (A), then new operation (B) record is created and performing current operation (A) is halted. Next minute cron fetches operations A and B. Operation A is fetches as it is not processed but system sees that dependent operation B is already created so it does not create it once again. Operation B is processed and status 'processed OK' is saved in proper row in db. Next minute cron fetches operation A from db and can finally perform it because dependent task is completed.
We are looking for ways to make it simpler, better and more elegant.
There is a list of open-source Java workflow engines.