I'm developing a learning application for a nonprofit using rails that WILL NOT have access to the internet. All assets must be housed locally. The application contains thousands of images, which are actually just URLs. I'm rather new to the dev world but have found it very difficult to find a solution to this problem.
A majority of the database content is seeded using a custom rake and cron task. We're using the Dribbble API to pull in most of this content, including the images. Looking at the urls, it looks like Dribbble is using S3 to house their images. Perhaps that is part of the solution. (example url: https://d13yacurqjgara.cloudfront.net/users/4521/screenshots/2742352/nest_notifications.jpg).
I know this topic must be easy for someone out there, but I literally have no experience or success finding a solution. Help and suggestions would be greatly appreciated!
By request, here is a copy of the rake task:
namespace :dribbble do
desc "TODO"
task get_recent: :environment do
url="https://api.dribbble.com/v1/shots/?access_token=XXX"
response = HTTParty.get(url)
recent_shots = JSON.parse(response.body)
recent_shots.each do |s|
users_dribbbleid = s['user']['id']
shots_dribbbleid = s['user']['id']
existing_user = User.where(designer_id: users_dribbbleid)
#IS THERE AN EXISTING USER IN THE DATABASE? IF NO, THEN...
if existing_user.empty?
newuser = User.create(
designer_id: s['user']['id'],
designer_full_name: s['user']['name'],
designer_username: s['user']['username'],
designer_home_url: s['user']['html_url'],
designer_avatar_url: s['user']['avatar_url'],
designer_bio: s['user']['bio'],
designer_location: s['user']['location'],
designer_bk_count: s['user']['buckets_count'],
designer_comments_received_count: s['user']['comments_received_count'],
designer_follower_count: s['user']['followers_count'],
designer_is_following_count: s['user']['followings_count'],
designer_made_likes_count: s['user']['likes_count'],
designer_received_likes_count: s['user']['likes_received_count'],
designer_project_count: s['user']['projects_count'],
designer_rebounds_received_count: s['user']['rebounds_received_count'],
designer_added_shots_count: s['user']['shots_count'],
designer_list_of_followers_url: s['user']['followers_url'],
designer_following_list_url: s['user']['following_url'],
designer_list_of_shots_url: s['shots_url']
)
newshot = Shot.create(
dribbble_id: s["id"],
title: s["title"],
description: s["description"],
width: s["width"],
height: s["height"],
images_hidpi: s["images"]["hidpi"],
images_normal: s["images"]["normal"],
images_teaser: s["images"]["teaser"],
viewcount: s["views_count"],
likes_count: s['likes_count'],
comments_count: s['comments_count'],
attachments_count: s['attachments_count'],
rebounds_count: s['rebounds_count'],
buckets_count: s['buckets_count'],
html_url: s['html_url'],
attachments_url: s["attatchments_url"],
buckets_url: s['buckets_url'],
comments_url: s['comments_url'],
likes_url: s['likes_url'],
projects_url: s['projects_url'],
animated: s['animated'],
tags: s['tags'],
user_id: newuser.id
)
commentresponse = HTTParty.get(s['comments_url']+"?access_token=XXX")
comments = JSON.parse(commentresponse.body)
comments.each do |c|
comments_dribbbleid = c["id"]
existing_comment = Comment.where(comment_id: comments_dribbbleid)
if existing_comment.empty?
newcomment = Comment.create(
comment_id: c["id"].to_s,
comment_created_at: c["created_at"],
body: c["body"],
user_avatar_url: c["user"]["avatar_url"],
user_id: c["user"]["id"],
user_name: c['user']['name'],
shot_id: newshot.id
)
end
end
#IF THERE IS AN EXISTING USER ALREADY, THEN CHECK IF THERE IS AN EXISTING SHOT. IF THERE IS NOT AN EXISTING SHOT, THEN...
else
existing_user = User.where(designer_id: users_dribbbleid)
existing_shot = Shot.where(user_id: existing_user[0].id)
if existing_shot.empty?
newshot = Shot.create(
dribbble_id: s["id"],
title: s["title"],
description: s["description"],
width: s["width"],
height: s["height"],
images_hidpi: s["images"]["hidpi"],
images_normal: s["images"]["normal"],
images_teaser: s["images"]["teaser"],
viewcount: s["views_count"],
likes_count: s['likes_count'],
comments_count: s['comments_count'],
attachments_count: s['attachments_count'],
rebounds_count: s['rebounds_count'],
buckets_count: s['buckets_count'],
html_url: s['html_url'],
attachments_url: s["attatchments_url"],
buckets_url: s['buckets_url'],
comments_url: s['comments_url'],
likes_url: s['likes_url'],
projects_url: s['projects_url'],
animated: s['animated'],
tags: s['tags'],
user_id: existing_user[0].id
)
commentresponse = HTTParty.get(s['comments_url']+"?access_token=XXX")
comments = JSON.parse(commentresponse.body)
comments.each do |c|
comments_dribbbleid = c["id"]
existing_comment = Comment.where(comment_id: comments_dribbbleid)
if existing_comment.empty?
newcomment = Comment.create(
comment_id: c["id"].to_s,
comment_created_at: c["created_at"],
body: c["body"],
user_avatar_url: c["user"]["avatar_url"],
user_id: c["user"]["id"],
user_name: c['user']['name'],
shot_id: newshot.id
)
end
end
end
end
end
end
end
`
Related
So, I have this controller, it gets the data from the uploaded file and save it to the right models. I made it work, but it is gigantic. I would like some directions of how to make it smaller. Should I create a class to extract the data and save to the db?
class UploadsController < ApplicationController
require 'csv'
def save_info_to_db
file = params[:file]
purchases_made = []
CSV.open(file.path, "r", { col_sep: "\t", headers: true }).each do |row|
data = Hash.new
data = {
purchaser: { name: row[0] },
item: { description: row[1], price: row[2], quantity: row[3] },
merchant: { name: row[4], address: row[5] }
}
merchant = Merchant.create_with(address: data[:merchant_address]).find_or_create_by(name: data[:merchant][:name])
item = merchant.items.create_with(price: data[:item][:price]).find_or_create_by(description: data[:item][:description])
purchaser = Purchaser.create(name: data[:purchaser][:name])
purchase = purchaser.purchases.create
purchase_items = PurchaseItem.create(item_id: item.id, quantity: data[:item][:quantity])
purchase.add_purchase_items(purchase_items)
purchases_made << purchase.total_price
end
session[:total_gross_income] = Purchase.total_gross_income(purchases_made)
redirect_to display_incomes_path
end
end
Thank you!
Lots of ways to do this, depending on what you prefer, but one idea is to use a Service class. In the service, you could handle the file processing.
Your controller would then end up looking something like:
class UploadsController < ApplicationController
require 'csv'
def save_info_to_db
file_service = FileProcessingService.new(file)
file_service.save_info!
session[:total_gross_income] = Purchase.total_gross_income(file_service.purchases_made)
redirect_to display_incomes_path
rescue
# perhaps some rescue logic in the event processing fails
end
All your file processing logic would go in the Service. You could even split that logic up -- logic for getting the data from the file, then separate logic for saving the data.
Lots of options, but the general idea is you can use a service to handle logic that may span multiple models.
I have the following query hooked up to a rails app. It only seems to be returning 30 projects at a time, even with an increased per_page count. However, if I make the same request via the browser, I get the expected number of projects.
https://api.github.com/search/repositories?q=license:gpl+license:lgpl+license:gpl+license:mit+forks:0+stars:1..2000+pushed:%3E2019-12-11+language:ruby+language:javascript&per_page=100
controller:
def request_projects
api_key = Rails.application.credentials.dig(:github)
query = "#{Project::LICENSES}+#{Project::FORKS}+#{Project::STARS}+#{Project::DATE}+#{Project::LANGUAGES}"
uri = URI.parse("#{Project::GITHUB_BASE_URL}?q=#{query}&client_secret=#{api_key[:secret]}&per_page=100")
res = Net::HTTP.get_response(uri)
Project.create_from_request(res.body)
end
Model:
def self.create_from_request(request_body)
data = JSON.parse(request_body, symbolize_names: true)
data[:items].each do |item|
name = item[:name]
owner = item[:owner][:login]
url = item[:html_url]
stars = item[:stargazers_count]
Project.create(name: name, owner: owner, url: url, stars: stars) if !Project.exists?(name: name, owner: owner)
end
end
In the User model I have an archive! method that is called when a User is destroyed. This action creates a new ArchivedUser in separate table.
The ArchivedUser is successfully created, but the way I am manually setting each value is pretty dirty; if a new column is added to the User table it must be added here as well.
I tried to select and slice the attributes, but got undefined local variable or methoduser'`
ArchivedUser.create(user.attributes.select{ |key, _| ArchivedUser.attribute_names.include? key })
ArchivedUser.create(user.attributes.slice(ArchivedUser.attribute_names))
How can I iterate through each attribute in the User table when creating an ArchivedUser with self?
def archive!
if ArchivedUser.create(
user_id: self.id,
company_id: self.company_id,
first_name: self.first_name,
last_name: self.last_name,
email: self.email,
encrypted_password: self.encrypted_password,
password_salt: self.password_salt,
session_token: self.session_token,
perishable_token: self.perishable_token,
role: self.role,
score: self.score,
created_at: self.created_at,
updated_at: self.updated_at,
api_key: self.api_key,
device_id: self.device_id,
time_zone: self.time_zone,
device_type: self.device_type,
verified_at: self.verified_at,
verification_key: self.verification_key,
uninstalled: self.uninstalled,
device_details: self.device_details,
is_archived: self.is_archived,
registered_at: self.registered_at,
logged_in_at: self.logged_in_at,
state: self.state,
creation_state: self.creation_state,
language_id: self.language_id,
offer_count: self.offer_count,
expired_device_id: self.expired_device_id,
unique_id: self.unique_id,
best_language_code: self.best_language_code,
offer_id: self.offer_id,
vetted_state: self.vetted_state,
photo_path: self.photo_path
)
self.is_archived = true
self.email = "#{self.email}.archived#{Time.now.to_i}"
self.encrypted_password = nil
self.password_salt = nil
self.session_token = nil
self.perishable_token = nil
self.device_id = nil
self.verification_key = nil
self.save!
self.update_column(:api_key, nil)
UserGroup.delete_all(:user_id => self.id)
else
# handle the ArchivedUser not being created properly
end
end
Thanks for viewing :)
Update:
We were able to figure out the reasons why ArchivedUser.create(self.attributes.slice!(ArchivedUser.attribute_names) wasn't working. The first reason is the create method requires "bang" to write the object. The second reason was that ArchivedUser has a user_id field, that User doesn't receive until after create. We have to set the user_id: manually with merge(user_id: self.id)
The final output looks like
ArchivedUser.create!(self.attributes.slice!(ArchivedUser.attribute_names).merge(user_id: self.id))
You were on the right track with the first implementation. You just have to use self instead of user.
ArchivedUser.create(self.attributes.slice(ArchivedUser.attribute_names))
If you just want to have a copy of the user that is being archived, I think an elegant way would be to do
archived_user = user_to_be_archived.dup
or you can take a look at the amoeba gem, this will do all the heavy lifting including associations if you want.
https://github.com/amoeba-rb/amoeba
I'm fetching rss data from a news API. However, sometimes the entries have certain fields such as image or summary, and sometimes they do not. How can I check if the object is empty before calling it?
url = "http://rss.cnn.com/rss/cnn_topstories.rss"
feed = Feedjira::Feed.fetch_and_parse url
api_json = JSON.parse(feed.to_json)
data_array = []
feed.entries.each_with_index do |entry,index|
data_array << { title: entry.title, link: entry.url, image_url: entry.image, summary: entry.summary }.as_json
end
In the above code, sometimes entry.image and entry.summary are empty and it returns an error such as:
undefined method `image' for #<Feedjira::Parser::ITunesRSSItem:0x007fb25c452688>
Current Attempt:
One obvious way is to check every object before saving it as a variable. But is this the best approach?
if entry.image.exists?
image = entry.image
else
image = ""
end
if entry.summary.exists?
summary = entry.summary
else
summary = ""
end
Use try(), it will return nil if the associated property is missing
entry.try(:image)
In your code, you can do like this
data_array << { title: entry.try(:title), link: entry.try(:url), image_url: entry.try(:image), summary: entry.try(:summary) }.as_json
Hope this helps!
If you are using Ruby 2.3.0 or higher you can make use of the brand new Safe Navigation Operator:
entry&.image # same thing as entry.try(:image) but shorter
You can even navigate as deep as you want, e.g.:
entry&.image&.urls&.small # Works as a charm even if "urls" is not present
The main thing I am looking to achieve from this question is understanding. With some assistance I have been looking at refactoring my controller code into more manageable modules/classes so that I can test them effectively. I have an example here that I would like to work on, my question is how would I test the class Sale:
class TransactionsController < ApplicationController
def create
payment = BraintreeTransaction::VerifyPayment.new(params, #user_id, #transaction_total)
payment.run(params)
if payment.success?
redirect_to thank_you_path
else
flash.now[:alert] = payment.error
flash.keep
redirect_to new_transaction_path
end
end
module BraintreeTransaction
class VerifyPayment
def initialize(params, user_id, total)
#transaction_total = total
#user_id = user_id
#params = params
#error_message = nil
end
def run(params)
#result = BraintreeTransaction::Sale.new.braintree_hash(params, #transaction_total)
if #result.success?
#cart_items = CartItem.where(user_id: #user_id).where.not(image_id: nil)
#cart_items.destroy_all
create_real_user
update_completed_transaction
guest_user.destroy
#success = true
else
update_transaction
#error_message = BraintreeErrors::Errors.new.error_message(#result)
end
end
def success?
#success
end
def error
#error_message
end
end
module BraintreeTransaction
class Sale
def braintree_hash(params, total)
Braintree::Transaction.sale(
amount: total,
payment_method_nonce: params[:payment_method_nonce],
device_data: params[:device_data],
customer: {
first_name: params[:first_name],
last_name: params[:last_name],
email: params[:email],
phone: params[:phone]
},
billing: {
first_name: params[:first_name],
last_name: params[:last_name],
company: params[:company],
street_address: params[:street_address],
locality: params[:locality],
region: params[:region],
postal_code: params[:postal_code]
},
shipping: {
first_name: params[:shipping_first_name].presence || params[:first_name].presence,
last_name: params[:shipping_last_name].presence || params[:last_name].presence,
company: params[:shipping_company].presence || params[:company].presence,
street_address: params[:shipping_street_address].presence || params[:street_address].presence,
locality: params[:shipping_locality].presence || params[:locality].presence,
region: params[:shipping_region].presence || params[:region].presence,
postal_code: params[:shipping_postal_code].presence || params[:postal_code].presence
},
options: {
submit_for_settlement: true,
store_in_vault_on_success: true
}
)
end
end
end
I don't know if I am looking at this wrong but this piece of code here BraintreeTransaction::Sale.new.braintree_hash is what I want to test and I want to ensure that when called the class receives a hash ?
Update
So far I have come up with this (though I am not 100% confident it is the correct approach ?)
require 'rails_helper'
RSpec.describe BraintreeTransaction::Sale do
#transaction_total = 100
let(:params) { FactoryGirl.attributes_for(:braintree_transaction, amount: #transaction_total) }
it 'recieves a hash when creating a payment' do
expect_any_instance_of(BraintreeTransaction::Sale).to receive(:braintree_hash).with(params, #transaction_total).and_return(true)
end
end
I get an error returned which I don't understand
Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
Exactly one instance should have received the following message(s) but didn't: braintree_hash
I might not be spot on but I would answer the way I would have tackled the issue. There are three ways you can write a test that hits the code you want to test.
Write a unit test for braintree_hash for BraintreeTransaction::Sale object
Write a controller unit method for create method in TransactionsController controller
write an integration test for route for create method in TransactionsController.
These are the ways you can start exploring.
A couple of things here. All the suggestions for refactoring your code (from your other question Writing valuable controller tests - Rspec) apply here. I can make further suggestions on this code, if helpful.
In your test, I believe your problem is that you never actually call BraintreeTransaction.new.braintree_hash(params) (which should be called immediately following your expect_any_instance_of declaration). And so no instances ever receive the message(s).