In my application, I'm using Resque to resize images. If an image is in the resizing queue, I want to show a "resizing image" icon.
This means that I need to be able to find all current jobs relating to a specific model ID in the queue. Right now I do it like this:
Resque.peek(:resize, 0, 100).find_all { |job| /#{model.id}/.match(job["args"][0]) }
This is stupid. But is there any way to query the Resque queue to find all jobs where the first argument is equal to [id]?
Thanks in advance.
Give resque-status a try. It is an extension to Resque that adds job tracking.
resque-status provides a set of simple classes that extend resque’s default functionality (with 0% monkey patching) to give apps a way to track specific job instances and their status. It achieves this by giving job instances UUID’s and allowing the job instances to report their status from within their iterations.
Note: d11wtq mentioned this above as a comment, but is actually the best answer so far.
Instead of querying resque queue, you should store image meta-data along with your model.
Lets assume you are storing product images. You are likely using a Redis hash to store product details. Just add another flag like this -
hset product:123 is_resizing true
You can them perform a simple lookup to show the resizing image icon. At the end of your resque job, delete the is_resizing key, and add the resized_image_url key.
I think the easiest way might be to use a redis set to cache this information.
When you add an image to the 'resize' queue, also add the image id to the 'resize_in_progress' set using SADD. (I assume you have some kind of unique key or name to refer to the image, even if not stored in the db. Maybe the full path to the filename.)
In the 'resize' process, as one of the final actions after successfully resizing the image, remove it from the set using the SREM command.
When you want a list of all images, you can fetch that with SMEMBERS. If you want just the members for a specific model id, you might need to store a separate set for each model, named something like 'resize_in_progress_3451' where 3451 is the id of the model that has the images being resized.
See http://redis.io/commands#set for more set commands.
Related
Context
I want to create a Power-Automate flow that automatically creates a sub-task in Azure DevOps when the Effort of a PBI is set.
When the Effort field goes from blank to a positive value, the task should be added (using the newly set Effort value as the task's Original Estimate and Remaining Work).
I managed to create a flow that does that using When a work item is update trigger.
Problem
The flow runs too often (whenever the work item changes, as long as the Effort is > 0).
Question
What would be the best way to ensure this flow runs only once per PBI?
Thoughts
Perhaps check for the presence of child tasks?
Perhaps set a hidden property when adding the task the first time and check that property afterwards?
You could add a trigger condition to the settings of your trigger action. Use the following expression:
#greater(triggerOutputs()?['body/fields/Microsoft_VSTS_Scheduling_Effort'], 0)
Add a Send an HTTP request action directly after the trigger. Use the following URI:
YourProjectName/_apis/wit/workItems/#{triggerOutputs()?['body/id']}/updates?api-version=6.0
Add a Filter Array. Use this expression for the From
body('Send_an_HTTP_request_to_Azure_DevOps')['value']
In the where of the Filter Array use the following expression which you add via the advanced mode:
#greater(length(string(item()?['fields']?['Microsoft.VSTS.Scheduling.Effort']?['newValue'])), 0)
In a condition check if the Filter Array returns no results. If it does, the value of Effort has not been changed in the past and you can safely create your new task
length(body('Filter_array'))
is equal to 1
I have created a scenario where I iterate through multiple modules with an array of data. This works fine.
After this completes, I want to run a module once before the scenario completes.
How do I add a module that won't get called in the loop?
There are few ways to achieve this,
Use Router to Create a new Route that will be triggered after the
first route is complete
Trigger new Scenario via Webhooks after you are done with the
scenario
If you are working with array, then using Array Aggregator or other
Aggregators will allow you to first complete the iteration and then
trigger the module you want to use
I am not sure exactly what you want to do after the iteration is complete, but setting the scenarios as displayed in the screenshot below should help you get started on this,
Using Router
For this you can create a router, the upper hand of the router is always executed first, so the iterator and other operations will be done there. After which, the next hand/route will be executed which will be the module you want to trigger at last.
However, If you want to pass some values from the first hand/route to the last one then you will need to set a variable and fetch it on the second route. See details here : https://www.integromat.com/en/help/converger
Using Aggregator Module
You can either use Array, Text or Numeric Aggregator to aggregate all the iteration operations and then trigger the module that you want to use at last.
As far as my knowledge goes, there is no Integromat default modules that can be configured before the scenario ends. We can leverage the Integromat API in future that is currently in development to do so.
I found a filter to be the most easy way of doing this. Essentially chekcing if this bundle position is equal to the total number of bundles!
If you're interested in doing something on the last iteration only, you can use a filter to check if the current bundle is equal to the total number of bundles
last bundle filter
They won't let me paste pics sigh
I have a user submission form that includes images. Originally I was using Carrierwave, but with that the image is sent to my server for processing first before being saved to Google Cloud Services, and if the image/s is/are too large, the request times out and the user just gets a server error.
So what I need is a way to upload directly to GCS. Active Storage seemed like the perfect solution, but I'm getting really confused about how hard compression seems to be.
An ideal solution would be to resize the image automatically upon upload, but there doesn't seem to be a way to do that.
A next-best solution would be to create a resized variant upon upload using something like #record.images.first.variant(resize_to_limit [xxx,xxx]) #using image_processing gem, but the docs seem to imply that a variant can only be created upon page load, which would obviously be extremely detrimental to load time, especially if there are many images. More evidence for this is that when I create a variant, it's not in my GCS bucket, so it clearly only exists in my server's memory. If I try
#record.images.first.variant(resize_to_limit [xxx,xxx]).service_url
I get a url back, but it's invalid. I get a failed image when I try to display the image on my site, and when I visit the url, I get these errors from GCS:
The specified key does not exist.
No such object.
so apparently I can't create a permanent url.
A third best solution would be to write a Google Cloud Function that automatically resizes the images inside Google Cloud, but reading through the docs, it appears that I would have to create a new resized file with a new url, and I'm not sure how I could replace the original url with the new one in my database.
To summarize, what I'd like to accomplish is to allow direct upload to GCS, but control the size of the files before they are downloaded by the user. My problems with Active Storage are that (1) I can't control the size of the files on my GCS bucket, leading to arbitrary storage costs, and (2) I apparently have to choose between users having to download arbitrarily large files, or having to process images while their page loads, both of which will be very expensive in server costs and load time.
It seems extremely strange that Active Storage would be set up this way and I can't help but think I'm missing something. Does anyone know of a way to solve either problem?
Here's what I did to fix this:
1- I upload the attachment that the user added directly to my service provider ( I use S3 ).
2- I add an after_commit job that calls a Sidekiq worker to generate the thumbs
3- My sidekiq worker ( AttachmentWorker ) calls my model's generate_thumbs method
4- generate_thumbs will loop through the different sizes that I want to generate for this file
Now, here's the tricky part:
def generate_thumbs
[
{ resize: '300x300^', extent: '300x300', gravity: :center },
{ resize: '600>' }
].each do |size|
self.file_url(size, true)
end
end
def file_url(size, process = false)
value = self.file # where file is my has_one_attached
if size.nil?
url = value
else
url = value.variant(size)
if process
url = url.processed
end
end
return url.service_url
end
In the file_url method, we will only call .processed if we pass process = true. I've experimented a lot with this method to have the best possible performance outcome out of it.
The .processed will check with your bucket if the file exists or not, and if not, it will generate your new file and upload it.
Also, here's another question that I have previously asked concerning ActiveStorage that can also help you: ActiveStorage & S3: Make files public
I absolutely don't know Active Storage. However, a good pattern for your use case is to resize the image when it come in. For this
Let the user store the image in Bucket1
When the file is created in Bucket1, an event is triggered. Plug a function on this event
The Cloud Functions resizes the image and store it into Bucket2
You can delete the image in Bucket1 at the end of the Cloud Function, or keep it few days or move it to cheaper storage (to keep the original image in case of issue). For this last 2 actions, you can use Life Cycle to delete of change the storage class of files.
Note: You can use the same Bucket (instead of Bucket1 and Bucket2), but an event to resize the image will be sent every time that a file is create in the bucket. You can use PubSub as middleware and add filter on it to trigger your function only with the file is created in the correct folder. I wrote an article on this
This is a bit of a complex, abstract question, so forgive me if it's not specific enough.
I've encountered a specific type of problem numerous times: That on one hand, a data source is used to update a certain data structure in an automated fashion at regular intervals, but on the other hand, stakeholders want to be able to manually override the automated entries.
Example:
You have a list of products, which are kept up-to-date (title, description, etc.) by some automated script which uses external data sources (product databases, etc.).
Let's say that in your data source you have a toaster "Freshtoast XYZ 300" and if its name changes to "FreshToast! XYZ-300", you want to propagate that update into your own (differently structured) product model.
At the same time, if a co-worker doesn't like the name "Freshtoast XYZ 300" and wants to change it to "Toaster XYZ 300 by Freshtoast" (manually), you don't want to override that change automatically (he would get angry), but you also don't want to simply ignore the updated name, since if the co-worker knew about the change, he'd adjust the name to "Toaster XYZ-300 by FreshToast!".
What's the best method to "consider" updated data sources - even for overridden data - while still allowing manual override?
PS: I'm using mostly Ruby / Rails, but I guess the question is very general. Also, to be clear, automated updates are the rule, while manual overrides are the exception in this scenario. So let's say 200,000 products get updated every single day, only 20 of which have manually overridden titles. So, for example, having to approve every single update is not an option.
Here goes nothing...
Hands off approach: Add a string column to products table that contains a serialized list of user-touched columns. Anytime a user touches a column in the products table, put it in the serialized list. When the automatic updater hits that record it checks the list for columns it should ignore.
Hand-wringing micro-manager approach: Use a versioning library (e.g. vestal_versions gem) and add a user_id column to the products table. Anytime a user-touched record is automatically updated, send them a notification and allow them to view a before/after which they can approve or reject.
I'm building a site that shows changes in deals that we have in our db. For example, if a deals status changes from pending to win, I want to show it, and if the value goes up or down, I want to show it, that kind of thing. Also, if you open the overview page, I want it to show the history of changes. So I need some kind of change logging, to be able to look in the past. How do I do this?
It is a rails project, but I think that's irrelevant.
I doubt there is any generic solution to this problem.
You can roll out your own. Start by considering all objects that need change logging. How many types are there? How often do you expect changes to occur? This will help you estimate the potential number of changes throughput you'll need to be dealing with. If there aren't too many, just stick them into database. If you are generating a lot, try storing to comma-separated-value file.
I have implemented a similar system before. I had 3 types of changes: 1) property value change, 2) adding of a value to a list, 3) removing value from a list.
I used the following format, stored in a log file:
//For type 1)
1,2011/01/01 00:00:00,MyObject,myProperty,oldValue,newValue
//For type 2)
2,2011/01/01 00:00:00,MyObject,myListProperty,addedValue
//For type 3)
3,2011/01/01 00:00:00,MyObject,myListProperty,removedValue
This captured most information I needed. The value parts were just some user-readable summary of the changed/added/removed property value.
Paper Trail Gem
Since you're on Rails, take a look at the PaperTrail gem. It does exactly what you're looking for and is beautifully built. You'll just need to add in a callback so that your overview page knows that a change occurred. But for the history of a model, just use the built-in PaperTrail functionality.