I have two APIs with different resources:
www.api-A.com**/consumers,
which returns: {consumers: ['mike', 'Anna', 'Danilo']}
www.api-B.com**/clients,
which returns: {clients: ['Jack', 'Bruce', 'Mary']}
I would like to use these two results in one controller. I want to treat them like if there were just one.
Do I have to create a wrapper for each api like:
module ApiAWrapper
#code here
end
module ApiBWrapper
#code here
end
and call the following inside my controller?
MyController
def index
#clients << ApiAWrapper.most_recent
#clients << ApiBWrapper.most_recent
#clients
end
end
Doing this, #clients will be:
['mike', 'Anna', 'Danilo', 'Jack', 'Bruce', 'Mary']
Is this the right way to use these different APIs with similar responses? Is there a design pattern that I can use or I should read about to guide me?
When I need external services to respond in a common way, I implement a parser. In other languages, you could use interfaces to enforce a method signature contract, but Ruby doesn't have this feature because of the duck typing.
This parser could be a function or a module. For example:
module GitHub
class Service
BASE_URI = 'https://api.github.com'
def self.fetch
response = HTTP.get("#{BASE_URI}/clients")
raise GitHub::ApiError unless response.ok?
Parser.new(response).to_common
end
end
class Parser
def initialize(response)
#response = response
end
def to_common
json_response = JSON.parse(#response)
json_response[:customers] = json_response.delete :clients
# more rules
# ...
json_response
end
end
end
Ok, there you go. Now you've got a Service, for fetching and handling the HTTP part, and the Parser, that handles the response body from the HTTP request. Now, let's suppose that you want to use another API, the BitBucket API, for instance:
module BitBucket
class Service
BASE_URI = 'https://bitbucket.com/api'
def self.fetch
response = HTTP.get("#{BASE_URI}/customers")
raise BitBucket::ApiError unless response.ok?
Parser.new(response).to_common
end
end
class Parser
def initialize(response)
#response = response
end
def to_common
json_response = JSON.parse(#response)
json_response[:clients] = (json_response.delete(:data).delete(:clients))
# more rules
# ...
json_response
end
end
end
This way, you'll have both services returning using the same interface. To join the results, you could do:
data = [GitHub::Service.fetch, BitBucket::Service.fetch, ...]
names = data.map { |customer_list| customer_list[:name] }
names.uniq
You should have wrappers for your API calls anyway because the controller should have as little logic as possible.
Regardless, I would create a class Client with a method to deserialize an array of client jsons into an array of clients. That way, in both wrappers you would call this method and return the array of clients ready to concat in the controller.
Something like:
class Client
attr_accessor :name
def initialize(client_json)
#name = client_json['name']
end
def self.deserialize_clients(clients_json)
clients_json.map{ |c| Client.new(c) }
end
end
Then for the wrappers:
module ApiAWrapper
def self.most_recent
response = #api call here
Client.deserialize_clients(JSON.parse(response.body))
end
end
What do you think?
Related
How can I refactor this ruby code using the Open/Closed principle or Strategy pattern ?
I know that the main thought is 'software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification' but how can I use this in practice?
class PaymentService
def initialize(payment, payment_type)
#payment = payment
#payment_type = payment_type
end
def process
result = case payment_type
when 'first'
process_first
when 'second'
process_second
end
payment.save(result)
end
def process_first
'process_first'
end
def process_second
'process_second'
end
end
In this example, instead of passing a payment_type you can build an object with a class that processes a payment:
class FirstPayment
def process
'process_first'
end
end
class SecondPayment
def process
'process_second'
end
end
class PaymentService
def initialize(payment, payment_strategy)
#payment = payment
#payment_strategy = payment_strategy
end
def process
result = #payment_stategy.process
payment.save(result)
end
end
PaymentService.new(payment, FirstPayment.new)
As a result, PaymentService behaviour can be extended by passing a new strategy (for example, ThirdPayment), but the class doesn't need to be modified, if the logic of processing the first or the second payments is changed.
So I'm wanting to parse a table on about 10 websites, so I want to create a new thread for each site. However, I'm not exactly sure how to return the data from this type of request.
Here's one class:
class TestRequest
def initialize
end
def start
urls = ['site1','site2','site3']
existing_data = Data.pluck(:symbol, :page)
data = GetData.pool(size: 10)
urls.each do |url|
data.async.perform_requests(url, existing_data)
end
end
end
and then GetData class looks like this:
require 'celluloid/current'
class GetData
include Celluloid
def perform_requests(url, existing_data)
# perform HTTP request
# parse HTTP response
# return returned data ???
end
end
What I'd ultimately like to do is have an instance variable in TestRequest class and simply add the returned value from GetData into that instance variable from the TestRequest class. After the threads are finished, I want to perform another action using the data in the instance variable.
I tried playing around with attr_reader, but it doesn't seem to play in my favor.
I tried this:
class TestRequest
def initialize
end
def start
#returned_data = []
urls = ['site1','site2','site3']
existing_data = Data.pluck(:symbol, :page)
data = GetData.pool(size: 10)
urls.each do |url|
data.async.perform_requests(url, existing_data)
end
end
attr_reader :returned_data
end
and then
require 'celluloid/current'
class GetData
include Celluloid
def perform_requests(tr, existing_data)
# perform HTTP request
# parse HTTP response
t = TestData.new
t.returned_data << "value"
end
end
but this doesn't work either.
Multi-threading and Ruby on Rails don't mix very well.
However, you should consider using the ActiveJob documentation (http://guides.rubyonrails.org/active_job_basics.html).
With ActiveJob, you can enqueue jobs and have them executed in the background. There are hooks method defined as well to notify you when a job is about to start, is running or have finished.
I use a gem to manage certain attributes of a gmail api integration, and I'm pretty happy with the way it works.
I want to add some local methods to act on the Gmail::Message class that is used in that gem.
i.e. I want to do something like this.
models/GmailMessage.rb
class GmailMessage < Gmail::Message
def initialize(gmail)
#create a Gmail::Message instance as a GmailMessage instance
self = gmail
end
def something_clever
#do something clever utilising the Gmail::Message methods
end
end
I don't want to persist it. But obviously I can't define self in that way.
To clarify, I want to take an instance of Gmail::Message and create a GmailMessage instance which is a straight copy of that other message.
I can then run methods like #gmail.subject and #gmail.html, but also run #gmail.something_clever... and save local attributes if necessary.
Am I completely crazy?
You can use concept of mixin, wherein you include a Module in another class to enhance it with additional functions.
Here is how to do it. To create a complete working example, I have created modules that resemble what you may have in your code base.
# Assumed to be present in 3rd party gem, dummy implementation used for demonstration
module Gmail
class Message
def initialize
#some_var = "there"
end
def subject
"Hi"
end
end
end
# Your code
module GmailMessage
# You can code this method assuming as if it is an instance method
# of Gmail::Message. Once we include this module in that class, it
# will be able to call instance methods and access instance variables.
def something_clever
puts "Subject is #{subject} and #some_var = #{#some_var}"
end
end
# Enhance 3rd party class with your code by including your module
Gmail::Message.include(GmailMessage)
# Below gmail object will actually be obtained by reading the user inbox
# Lets create it explicitly for demonstration purposes.
gmail = Gmail::Message.new
# Method can access methods and instance variables of gmail object
p gmail.something_clever
#=> Subject is Hi and #some_var = there
# You can call the methods of original class as well on same object
p gmail.subject
#=> "Hi"
Following should work:
class GmailMessage < Gmail::Message
def initialize(extra)
super
# some additional stuff
#extra = extra
end
def something_clever
#do something clever utilising the Gmail::Message methods
end
end
GmailMessage.new # => will call first the initializer of Gmail::Message class..
Building upon what the other posters have said, you can use built-in class SimpleDelegator in ruby to wrap an existing message:
require 'delegate'
class MyMessage < SimpleDelegator
def my_clever_method
some_method_on_the_original_message + "woohoo"
end
end
class OriginalMessage
def some_method_on_the_original_message
"hey"
end
def another_original_method
"zoink"
end
end
original = OriginalMessage.new
wrapper = MyMessage.new(original)
puts wrapper.my_clever_method
# => "heywoohoo"
puts wrapper.another_original_method
# => "zoink"
As you can see, the wrapper automatically forwards method calls to the wrapped object.
I'm not sure why you can't just have a simple wrapper class...
class GmailMessage
def initialize(message)
#message = message
end
def something_clever
# do something clever here
end
def method_missing(m, *args, &block)
if #message.class.instance_methods.include?(m)
#message.send(m, *args, &block)
else
super
end
end
end
Then you can do...
#my_message = GmailMessage.new(#original_message)
#my_message will correctly respond to all the methods that were supported with #original_message and you can add your own methods to the class.
EDIT - changed thanks to #jeeper's observations in the comments
It's not the prettiest, but it works...
class GmailMessage < Gmail::Message
def initialize(message)
message.instance_variables.each do |variable|
self.instance_variable_set(
variable,
message.instance_variable_get(variable)
)
end
end
def something_clever
# do something clever here
end
end
Thanks for all your help guys.
A user can import his data from other websites. All he needs to do is type in his username on the foreign website and we'll grab all pictures and save it into his own gallery. Some of the pictures needs to be transformed with rMagick (rotating,watermarking), that depends on the importer (depends on which website the user chooses to import data from)
We are discussing the sexiest and most flexible way to do so. We are using carrierwave, but we will change to paperclip in case it fits us more.
Importer Structure
The current structure does looks like (its roughly pseudocode)
module Importer
class Website1
def grab_pictures
end
end
class Website2
def grab_pictures
end
end
end
class ImporterJob
def perform(user, type, foreign_username)
pictures = Importer::type.grab_pictures(foreign_username)
pictures.each do |picture|
user.pictures.create picture
end
end
end
We struggle with the decision, whats the best return of the importer.
Solution1:
The Importer is returning an array of strings with URLs ["http://...", "http://...", "http://..."].
That array we can easily loop and tell carrierwave/paperclip to remote_download the images. After that, we'll run a processor to transform the pictures, if we need to.
def get_picture_urls username
pictures = []
page = get_html(username)
page.scan(/\/p\/\d{4}-\d{2}\/#{username}\/[\w\d]{32}-thumb.jpg/).each do |path|
pictures << path
end
pictures.uniq.collect{|x| "http://www.somewebsite.com/#{x.gsub(/medium|thumb/, "big")}"}
end
this actually returns an array ["url_to_image", "url_to_image", "url_to_image"]
Then in the Picture.after_create, we call something to remove the Watermark on that Image.
Solution2:
grab_pictures is downloading each picture to an tempfile and transform it. it will return an array of tempfiles [tempfile, tempfile, tempfile]
code for that is:
def read_pictures username
pictures = []
page = get_html(username)
page.scan(/\/p\/\d{4}-\d{2}\/#{username}\/[a-z0-9]{32}-thumb.jpg/).each do |path|
pictures << path
end
pictures.uniq.map { |pic_url| remove_logo(pic_url) }
end
def remove_logo pic_url
big = Magick::Image.from_blob(#agent.get(pic_url.gsub(/medium.jpg|thumb.jpg/, 'big.jpg')).body).first
# ... do some transformation and watermarking
file = Tempfile.new(['tempfile', '.jpg'])
result.write(file.path)
file
end
This actually returns an array of [Tempfile, Tempfile, Tempfile]
Summary
The result will be the same for the user - but internally we are discovering 2 different ways of data handling.
We want to keep logic where it belongs and work as generic as possible.
Can you guys help us with choosing the right way? Longterm we want to have around 15 differnt Importers.
I've had a similar situation to this recently - I recommend an array of strings for several reasons:
Familiarity: How often are you working with tempfiles? What about the other developers on your team? How easy is it to manipulate strings vs manipulating tempfiles?
Flexibility: Now you want to just process the picture, but maybe in the future you'll need to keep track of the picture id for each picture from the external site. That's trivial with an array of strings. With an array of tempfiles, it's more difficult (just how much depends, but the fact is it will be more difficult). That of course goes for other as-yet-unknown objectives as well.
Speed: It's faster and uses less disk space to process an array of strings than a group of files. That's perhaps a small issue, but if you get flooded with a lot of photos at the same time, it could be a consideration depending on your environment.
Ultimately, the best thing I can say is start with strings, make a few importers, and then see how it looks and feels. Pretend you're a project manager or a client - start making strange, potentially unreasonable demands of the data you've collected. How easy will it be for you to meet those demands with your current implementation? Would it be easier if you were using tempfiles?
I am doing this for a similar project, where I have to browse and get information on different websites. On each of those websites I have to reach for same goal by performing roughly the same actions, and they are off-course all structured differently.
The solution is inspired from the basic principles of OOP:
Main class: handle the high level operations, handle database operations, handle images operation, manage errors
class MainClass
def import
# Main method, prepare the download and loop through each images
log_in
go_to_images_page
images = get_list_of_images
images.each do |url|
begin
image_record = download_image url
transform_image image_record
rescue
manage_error
end
end
display_logs
send_emails
end
def download_image(url)
# Once the specific class returned the images url, this common method
# Is responsible for downloading and creating database record
record = Image.new picture: url
record.save!
record
end
def transform_image(record)
# Transformation is common so this method sits in the main class
record.watermark!
end
# ... the same for all commom methods (manage_error, display_logs, ...)
end
Specific classes (one per targeted website) : handle low lovel operations and return data to the main class. The only interraction this class must have is with the website, meaning no database access and no error management as much as possible (don't get stuck by your design ;))
Note: In my design I simply inherit from the MainClass, but you can use module inclusion if you prefer.
class Target1Site < MainClass
def log_in
# Perform specific action in website to log the use in
visit '/log_in'
fill_in :user_name, with: ENV['user_name']
...
end
def go_to_images_page
# Go to specific url
visit '/account/gallery'
end
def get_list_of_images
# Use specific css paths
images = all :css, 'div#image-listing img'
images.collect{|i| i['src']}
end
# ...
end
I solved a similar problem... I had to import from a xls file, different resource types using:
The Importer class (ResourcesGroupsImporter).
A base mapper class (ResourceMapper) It acts as interface for specific mappers. It has common methods for all resources and raises NotImplementedError encouraging you to implement those methods when you adds a new resource type.
One mapper by resource type (DetentionsPollMapper, FrontCycleMapper). Each one, implements specific logic for an specific resource.
Implementation example:
The importer...
class ResourcesGroupsImporter
attr_reader :group
attr_reader :mappers
def initialize(_source, _resources_group)
#group = _resources_group
#source = _source
#xls = Roo::Spreadsheet.open(#source.path, extension: :xlsx)
#mappers = Resource::RESOURCEABLE_CLASSES.map { |klass| resource_mapper(klass) }
end
def import
ActiveRecord::Base.transaction do
self.mappers.each { |mapper| create_resource(mapper) }
relate_source_with_group unless self.has_errors?
raise ActiveRecord::Rollback if self.has_errors?
end
end
def has_errors?
!self.mappers.select { |mapper| mapper.has_errors? }.empty?
end
private
def resource_mapper(_class)
"#{_class}Mapper".constantize.new(#xls, #group)
end
def create_resource(_mapper)
return unless _mapper.resource
_mapper.load_resource_attributes
_mapper.resource.complete
_mapper.resource.force_validation = true
if _mapper.resource.save
create_resource_items(_mapper)
else
_mapper.load_general_errors
end
end
def create_resource_items(_mapper)
_mapper.set_items_sheet
columns = _mapper.get_items_columns
#xls.each_with_index(columns) do |data, index|
next if data == columns
break if data.values.compact.size.zero?
item = _mapper.build_resource_item(data)
_mapper.add_detail_errors(index, item.errors.messages) unless item.save
end
end
def relate_source_with_group
#group.reload
#group.source = #source
#group.save!
end
end
The interface...
class ResourceMapper
attr_reader :general_errors
attr_reader :detailed_errors
attr_reader :resource
def initialize(_xls, _resource_group)
#xls = _xls
#resource = _resource_group.resourceable_by_class_type(resource_class)
end
def resource_class
raise_implementation_error
end
def items_sheet_number
raise_implementation_error
end
def load_resource_attributes
raise_implementation_error
end
def get_items_columns
raise_implementation_error
end
def build_resource_item(_xls_item_data)
resource_items.build(_xls_item_data)
end
def raise_implementation_error
raise NotImplementedError.new("#{caller[0]} method not implemented on inherited class")
end
def has_errors?
!self.general_errors.nil? || !self.detailed_errors.nil?
end
def resource_items
self.resource.items
end
def human_resource_name
resource_class.model_name.human
end
def human_resource_attr(_attr)
resource_class.human_attribute_name(_attr)
end
def human_resource_item_attr(_attr)
"#{resource_class}Item".constantize.human_attribute_name(_attr)
end
def load_general_errors
#general_errors = self.resource.errors.messages
end
def add_detail_errors(_xls_row_idx, _error)
#detailed_errors ||= []
#detailed_errors << [ _xls_row_idx+1, _error ]
end
def set_items_sheet
#xls.default_sheet = items_sheet
end
def general_sheet
sheet(0)
end
def items_sheet
sheet(self.items_sheet_number)
end
def sheet(_idx)
#xls.sheets[_idx]
end
def general_cell(_col, _row)
#xls.cell(_col, _row, general_sheet)
end
end
Specific mapper types...
class DetentionsPollMapper < ResourceMapper
def items_sheet_number
6
end
def resource_class
DetentionsPoll
end
def load_resource_attributes
self.resource.crew = general_cell("N", 3)
self.resource.supervisor = general_cell("N", 4)
end
def get_items_columns
{
issue: "Problema identificado",
creation_date: "Fecha",
workers_count: "N° Trabajadores esperando",
detention_hours_string: "HH Detención",
lost_hours: "HH perdidas",
observations: "Observación"
}
end
def build_resource_item(_xls_item_data)
activity = self.resource.activity_by_name(_xls_item_data[:issue])
data = {
creation_date: _xls_item_data[:creation_date],
workers_count: _xls_item_data[:workers_count],
detention_hours_string: _xls_item_data[:detention_hours_string],
lost_hours: _xls_item_data[:lost_hours],
observations: _xls_item_data[:observations],
activity_id: !!activity ? activity.id : nil
}
resource_items.build(data)
end
end
class FrontCycleMapper < ResourceMapper
def items_sheet_number
8
end
def resource_class
FrontCycle
end
def load_resource_attributes
self.resource.front = general_cell("S", 3)
end
def get_items_columns
{
task: "Tarea",
start_time_string: "Hora",
task_type: "Tipo de Tarea",
description: "Descripción"
}
end
def build_resource_item(_xls_item_data)
activity = self.resource.activity_by_name_and_category(
_xls_item_data[:task], _xls_item_data[:task_type])
data = {
description: _xls_item_data[:description],
start_time_string: _xls_item_data[:start_time_string],
activity_id: !!activity ? activity.id : nil
}
resource_items.build(data)
end
end
A helper have to provide a way to access pict as you prefer.
However saving "http://...", "http://...", "http://..." this kind of strings, is a lack of security.
I 'd preferd hash like this: domain_name = {"name_on_url.jpg" =>path_on_disk, ...}
To ensure flexibility of access.
I am trying to make a bit of a custom Rails logger which ultimately will log to a database. However, I don't have access to things like the request object, which I very much would like to have.
I'm currently trying to use the LogSubscriber (notification) interface to do the bulk of this; perhaps this is not the right approach. I do know I could abuse Thread.current[] but I was hoping to avoid doing that.
Here's the code I have which is as basic as I can get it for an example. This is loaded in an initializer.
module RequestLogging
class LogSubscriber < ActiveSupport::LogSubscriber
def process_action(event)
pp request # <--- does not work
pp event
end
end
RequestLogging::LogSubscriber.attach_to :action_controller
Probably you need to override process_action in ActionController::Instrumentation and then request object will be accessible like event.payload[:request]. I think you can put code somewhere in config/initializers, code example:
ActionController::Instrumentation.class_eval do
def process_action(*args)
raw_payload = {
controller: self.class.name,
action: self.action_name,
params: request.filtered_parameters,
format: request.format.try(:ref),
method: request.method,
path: (request.fullpath rescue "unknown"),
request: request,
session: session
}
ActiveSupport::Notifications.instrument("start_processing.action_controller", raw_payload.dup)
ActiveSupport::Notifications.instrument("process_action.action_controller", raw_payload) do |payload|
result = super
payload[:status] = response.status
append_info_to_payload(payload)
result
end
end
end
you can get the even.payload then pass it your own CustomLogger(formatted_log(even.payload) and then there you can define a module and save it.
You may want to customise your formatted_log function to beautify the payload accordingly.
def process_action(event)
CustomLogger.application(formattedLog(event.payload))
end
def formattedLog(payload)
# some restructuring of data.
end