Uploading Multiple Images to Imgur API with Ruby is Slow - ruby-on-rails

So I have a form that can submit multiple images at once. I also wrote a wrapper for the Imgur API in Ruby. My problem is that since this happens completely synchronously, it takes forever and times out for even 10 images. I'm wondering if there is a better way that would be able to handle more images.
All I can think of is asynchronously submitting forms with one image each, but I'm not sure if that's a good idea or if it will just hold up other requests.
class MissionsController < ApplicationController
def add_images
image_ids = params[:images].collect do |image|
Image.with_imgur(
title: "#{#mission.trip.name} - #{current_user.username}",
image: image,
album_id: #mission.album.imgur_id,
user_id: current_user.id,
trip_id: #mission.trip_id
).imgur_id
end
end
end
class Image < ActiveRecord::Base
def self.with_imgur(options)
trip_id = options.delete(:trip_id)
user_id = options.delete(:user_id)
image = Imgur::Image.create(options)
create(
imgur_id: image["id"],
link: image["link"],
trip_id: trip_id,
user_id: user_id
)
end
end
https://github.com/tomprats/toms-missions/blob/master/app/models/imgur.rb#L116
class Imgur::Image < Base
def self.create(options)
url = "https://api.imgur.com/3/image"
params = {
image: options[:image],
album: options[:album_id],
type: options[:type] || "file", # "file" || "base64" || "URL"
title: options[:title],
description: options[:description]
}
api_post(url, params).body["data"]
end
end

I'd take a look at Typhoeus https://github.com/typhoeus/typhoeus - I've used this in the past to handle uploads etc to Amazon Glacier. I found that it could be much faster - Typhoeus gives you a future object and will handle the upload of you in the background without blocking the application.
I hope this helps!

Related

How can do a performance friendly version of this Ruby code?

Hey guys i'm new to RoR and ruby in general.
In this project i am working at the moment i'm showing 4 images of a vehicle that were previously uploaded by users.
In my class 'Vehicle' i have a singleton method - that returns an array with urls for attached images of a vehicle object - which i call like this Vehicle.vehicle_images(#vehicle.id). I'd like to make this method look more performatic and less hard-coded.
my vehicle class:
class Vehicle < ApplicationRecord
has_many :vehicles_attachment, dependent: :destroy
has_many :vehicles_attachment_imagem, -> { where({ type_attachment: 'imagem' }) }, class_name: 'VehiclesAttachment', dependent: :destroy
def self.vehicle_images(vehicle_id)
if vehicle_id
images = VehiclesAttachment.where({ vehicle_id: vehicle_id, type_attachment: :imagem })
urls = []
if !images.nil? && !images.empty?
count = 0
images.each do |img|
urls << if img.attachment.file?
"https:#{img.attachment.url(:regular)}"
else
4.times do
urls << ActionController::Base.helpers.image_url('viatura.jpg', digest: false).to_s
end
end
count += 1
end
if count < 4
(4 - count).times do
urls << ActionController::Base.helpers.image_url('viatura.jpg', digest: false).to_s
end
end
else
4.times do
urls << ActionController::Base.helpers.image_url('viatura.jpg', digest: false).to_s
end
end
else
4.times do
urls << ActionController::Base.helpers.image_url('viatura.jpg', digest: false).to_s
end
end
urls
end
end
Can you help me?
As a first step it could be simplified to this:
def self.vehicle_images(vehicle_id)
blank_image_url = ActionController::Base.helpers.image_url('viatura.jpg', digest: false).to_s
urls = []
if vehicle_id
VehiclesAttachment.where(vehicle_id: vehicle_id, type_attachment: :imagem).each do |image|
urls << img.attachment.file? ? "https:#{img.attachment.url(:regular)}" : blank_image_url
end
end
urls << blank_image_url while urls.size < 4
urls
end
But this is just a first step. The next steps could be:
Determine if you have to deal with blank vehicle_id. If not then an instance method on a vehicle would be better than a class method. And you would be able to use the defined association instead of the where on the class.
I would investigate if it is really possible that you sometimes get only some images returned and really have to fill in the dummy image between them. Or if it would be enough to only fill in dummy images at the end if there are less than 4. That would simplify the method further.
I wonder if you sometimes have to deal with more than 4 images? Your code currently doesn't handle that.
Perhaps you could add variations to ensure that there are always exactly the four images?

Updating Lots of Records at Once in Rails

I've got a background job that I run about 5,000 of them every 10 minutes. Each job makes a request to an external API and then either adds new or updates existing records in my database. Each API request returns around 100 items, so every 10 minutes I am making 50,000 CREATE or UPDATE sql queries.
The way I handle this now is, each API item returned has a unique ID. I search my database for a post that has this id, and if it exists, it updates the model. If it doesn't exist, it creates a new one.
Imagine the api response looks like this:
[
{
external_id: '123',
text: 'blah blah',
count: 450
},
{
external_id: 'abc',
text: 'something else',
count: 393
}
]
which is set to the variable collection
Then I run this code in my parent model:
class ParentModel < ApplicationRecord
def update
collection.each do |attrs|
child = ChildModel.find_or_initialize_by(external_id: attrs[:external_id], parent_model_id: self.id)
child.assign_attributes attrs
child.save if child.changed?
end
end
end
Each of these individual calls is extremely quick, but when I am doing 50,000 in a short period of time it really adds up and can slow things down.
I'm wondering if there's a more efficient way I can handle this, I was thinking of doing something instead like:
class ParentModel < ApplicationRecord
def update
eager_loaded_children = ChildModel.where(parent_model_id: self.id).limit(100)
collection.each do |attrs|
cached_child = eager_loaded_children.select {|child| child.external_id == attrs[:external_id] }.first
if cached_child
cached_child.update_attributes attrs
else
ChildModel.create attrs
end
end
end
end
Essentially I would be saving the lookups and instead doing a bigger query up front (this is also quite fast) but making a tradeoff in memory. But this doesn't seem like it would be that much time, maybe slightly speeding up the lookup part, but I'd still have to do 100 updates and creates.
Is there some kind of way I can do batch updates that I'm not thinking of? Anything else obvious that could make this go faster, or reduce the amount of queries I am doing?
You can do something like this:
collection2 = collection.map { |c| [c[:external_id], c.except(:external_id)]}.to_h
def update
ChildModel.where(external_id: collection2.keys).each |cm| do
ext_id = cm.external_id
cm.assign_attributes collection2[ext_id]
cm.save if cm.changed?
collection2.delete(ext_id)
end
if collection2.present?
new_ids = collection2.keys
new = collection.select { |c| new_ids.include? c[:external_id] }
ChildModel.create(new)
end
end
Better because
fetches all required records all at once
creates all new records at once
You can use update_columns if you don't need callbacks/validations
Only drawback, more ruby code manipulation which I think is a good tradeoff for db queries..

How to refactor method to lower RuboCop's ABCsize

On my journey to learn ruby & rails I went on and installed Rubocop. So far it's been a great help in refactoring my code the ruby-way, but now I think I've hit the wall with this helpless case. Given the following method for creating a new entity, I'm looking for a way to refactor it to make Rubocop stop yelling at me about:
Line length
Assignment Branch Condition size (currently 26.02/15)
the only thing that I can think of for the moment, except of disabling those cops ofc, is actually splitting up the model into two smaller ones (say basic info and financial) and set them up accordingly, but I get the impression that this would move the complexity out of the creation method and put it elsewhere, as I would need to remember to create both related entities etc. Any hints are more than welcome.
def create_store_information(store, meta)
user = #datasource.user
user.store_informations.create!(
name: store['name'],
description: store['description'],
status: 1,
url: store['URL'].downcase,
store_version: store['version'],
api_version: store['wc_version'],
timezone: meta['timezone'],
currency: meta['currency'],
currency_format: meta['currency_format'],
currency_position: meta['currency_position'],
thousand_separator: meta['thousand_separator'],
decimal_separator: meta['decimal_separator'],
price_num_decimals: meta['price_num_decimals'],
tax_included: cast_to_bool(meta['tax_included']),
weight_unit: meta['weight_unit'],
dimension_unit: meta['dimension_unit'],
ssl_enabled: cast_to_bool(meta['ssl_enabled']),
permalinks_enabled: cast_to_bool(meta['permalinks_enabled']),
generate_password: cast_to_bool(meta['generate_password']),
user: user
)
end
Edit:
As per request, I'm attaching a second sample of creating store_information from a different class.
def create_store_information(store, meta)
user = #datasource.user
user.store_informations.create!(
name: store['id'],
description: store['name'],
status: 1,
url: store['domain'].downcase,
store_version: '1.0',
api_version: '1.0',
timezone: meta['timezone'],
currency: meta['currency'],
currency_format: meta['money_format'],
currency_position: '', # not applicable
thousand_separator: '', # not applicable, take from user's locale
decimal_separator: '', # not applicable, take from user's locale
price_num_decimals: '', # not applicable, take from user's locale
tax_included: cast_to_bool(meta['taxes_included']),
weight_unit: nil, # not applicable
dimension_unit: nil, # not applicable
ssl_enabled: cast_to_bool(meta['force_ssl']),
permalinks_enabled: true,
generate_password: false,
user: user
)
end
This is just 1 suggestion out of many possibilities.
You can use Ruby's meta programming abilities to dynamically send methods.
The meta object's fields are easy to assign the user.store_informations because the fields match 1 for 1.
It is also possible for the store object but it wouldn't be as straightforward.
You can move the fields to an array inside your class definition:
CAST_TO_BOOL = %w(
tax_included
ssl_enabled
permalinks_enabled
generate_password
).freeze
META_FIELDS = %w(
timezone
currency
currency_format
currency_position
thousand_separator
decimal_separator
price_num_decimals
tax_included
weight_unit
dimension_unit
ssl_enabled
permalinks_enabled
generate_password
).freeze
then you could define a private method which dynamically sets the meta fields of the user.store_informations
private
def set_meta_fields_to_store_information(user)
META_FIELDS.each do |field|
if CAST_TO_BOOL.include? field
user.store_informations.__send__ "#{f}=" { cast_to_bool( meta[field] ) }
next
end
user.store_informations.__send__ "#{f}=" { meta[field] }
end
end
then you could call that method instead:
def create_store_information(store, meta)
user = #datasource.user
user.store_informations.new(
name: store['name'],
description: store['description'],
status: 1,
url: store['URL'].downcase,
store_version: store['version'],
api_version: store['wc_version'],
user: user
)
set_meta_fields_to_store_information(user)
user.save!
end
Edit#2
Regarding populating the fields with objects of different classes;
One way to go about it would be to define a method which assign's the fields for you depending on the class of the store.
But then again, if you have thousands of different stores, this probably wouldn't be optimal.
class StoreA; end
class StoreB; end
class StoreC; end
then:
# you could also use dynamic method dispatching here instead:
def set_store_information_to_user(store, user)
case store
when StoreA
assign_store_a_method(store, user)
when StoreB
assign_store_b_method(store, user)
when StoreC
assign_store_c_method(store, user)
end
end
private
def assign_store_a_method(store, user); end
def assign_store_b_method(store, user); end
def assign_store_c_method(store, user); end

Rails Paperclip copy set of images from model A to model B

Currently I have a model A (ItemRequest) that has many images.
Once this item request has been approved, it creates a model B (Item) that also has many images.
I'm using paperclip to handle image upload to S3, and what I would like is that after the approve process has been completed, copy the images from model A to model B, duplicating them.
Following some similar questions (but none using models with many images) I've written the following in my ItemRequest model approve method.
if images.length > 0
item.images = images.map { |img| img.dup }
end
And this apparently creates a new set of Image models, but when trying to access the url, it does not open any image and S3 returns 403.
I've checked and no image is created in S3. I can access other images so the S3 is properly configured.
Any ideas on how to solve this?
def resolve! user_id
if status != RESOLVED
ActiveRecord::Base.transaction do
self.resolved_by_id = user_id
self.resolved_date = DateTime.now
self.status = RESOLVED
save!
catalog_item = Item.create(
name: item_name,
oem_name: oem_name,
description: description,
category_id: category_id,
make: make,
part_number: part_number,
uom: uom,
cost_in_cents: cost_in_cents,
weight_in_grams: weight_in_grams
)
if images.length > 0
catalog_item.images = images.map { |img| img.dup }
catalog_item.save
end
end
end
true
end
Something is worth mentioning that doesn't simplify things, is that my Image model is polymorphic.
I've managed to write a solution but not happy with it yet.
Add this to the Image model
def file_remote_url=(url_value)
self.file = URI.parse url_value
#file_remote_url = url_value
end
And then
catalog_item.images = images.map { |img|
new_image = Image.new()
new_image.file_remote_url = img.url
new_image
}
catalog_item.save
Also, this solution does not work locally if uploading to the dev machine.
After a few days of trying different approaches, here is what solved the issue for me:
catalog_item.images = images.map do |img|
new_image = Image.new
new_image.imageable_type = 'CatalogItem'
new_image.imageable_id = catalog_item.id
new_image.file = img.file
new_image
end
This will create a copy of the image instead of changing the reference only. As far as I understood, if you're using S3 and paperclip, it won't simply copy the image using the S3 api but rather make a copy and upload it again, which might be a bit inefficient, but that is a whole other unrelated issue.

Get large images from Page Feed? using Koala Gem & Rails 3.2

Does anyone know how to pull different size images from the Page Feed?
I was trying to use the Type hash that works great for friends and profile pictures.
#page-feed = #graph.get_connections("somepage", "feed", {"type" => "large"})
but for some reason I'm always getting the same picture size for all posts.
Thanks !
Reading the code here: https://github.com/arsduo/koala/blob/81e66f459df840d9d5e122c0d498e2fb9d146655/lib/koala/api/graph_api.rb (line 178, def get_picture) you can see that the method accepts options hash:
Gem source:
# Fetches a photo.
# (Facebook returns the src of the photo as a response header; this method parses that properly,
# unlike using get_connections("photo").)
#
# #param options options for Facebook (see #get_object).
# To get a different size photo, pass :type => size (small, normal, large, square).
# #param block (see Koala::Facebook::API#api)
#
# #note to delete photos or videos, use delete_object(id)
#
# #return the URL to the image
def get_picture(object, args = {}, options = {}, &block)
# Gets a picture object, returning the URL (which Facebook sends as a header)
resolved_result = graph_call("#{object}/picture", args, "get", options.merge(:http_component => :headers)) do |result|
result ? result["Location"] : nil
end
block ? block.call(resolved_result) : resolved_result
end
So you can call it like .get_picture(id, type: :large). Like this:
graph = Koala::Facebook::API.new(token)
profile = graph.get_object('me')
graph.get_picture(profile['id'], type: :large)
Just in case anyone else comes across this, this is what I had to do in order to retrieve the large images. Note that I'm only grabbing the first post in the feed.
In my controller:
#fb_post = #facebook.get_connections(page_id, 'posts').first
#photo = #facebook.get_connections(#fb_post['id'], 'attachments').first
Then, to grab the proper link in the view inside of an img tag, I used:
#photo["media"]["image"]["src"]
For anyone still struggling with this, I was able to use the 'full_picture' field in my Koala object to retrieve a full-resolution URLs of the images:
fields = ['id','picture','full_picture']
graphObj = Koala::Facebook::API.new(accessToken)
hashes = graphObj.get_connection(groupId, 'feed', { limit: 10, fields: fields })
hashes.each do |hash|
mash = Hashie::Mash.new(hash)
fullPicture = mash.full_picture
end

Resources