In a rails application, I do some requests on an external database. In newrelic, when I look at the SQL requests I have this :
2.997 718 ms SHOW FULL FIELDS FROM `whale`
3.717 721 ms SHOW VARIABLES WHERE Variable_name = 'character_set_client'
4.440 728 ms SHOW TABLES LIKE 'whale'
5.169 668 ms SHOW CREATE TABLE `whale`
5.839 731 ms SELECT id, `whale`.`name` FROM `whale`
As you can see, all requests take a long time so I want to minimize them. I only need the last result.
This is my simple controller :
class AnimalsController < ApplicationController
def index
MicsPix.pluck(:id, :name)
render text: 'ok'
end
end
And my model :
class MicsPix < ActiveRecord::Base
establish_connection(:otherdb)
def self.table_name
"whale"
end
end
Is there a solution to skip queries than I don't use? I don't necessarily want to use ActiveRecord.
I'm not certain where the extra queries arise but i have a suggestion on how to remove them
c = ActiveRecord::Base.establish_connection(
:adapter => "mysql2",
:host => "localhost",
:username => "root",
:password => "",
:database => "mydatabase"
)
sql = "SELECT id, `whale`.`name` FROM `whale`"
res = c.connection.execute(sql)
then reset db connection to default.
this code works for me, and I can get results from an external db with only query being executed. I did this in a controller method when I tried it out but I think it would be more neat in the model. As far as I understand the establish_connection(:otherdb) does the same as I do when I produce my c. So I think you could write something like this in your MicsPix
def get_list
sql = "SELECT id, `whale`.`name` FROM `whale`"
connection.execute(sql)
end
this returns and mysql res object which is quite similar to arrays but not quite.
regards
/Albin
Related
I'm trying to download a CSV file from an FTP server and if the record exists I want to update that record and not create a duplicate. To give a little more context - I'm trying to upload a group of orders from an FTP folder into my Rails app. There is a new file every hour - sometimes the orders in a certain drop contain duplicates from the previous drop to prevent one from slipping through the tracks or on occasion the order has been updated by the customer (change in qty, change in address, etc.) w/ the next drop. So my question is if the order is purely a duplicate with no changes how can I skip over those orders and if a record has been changed how can I update that record?
Ruby on Rails 5.1.4 - Ruby 2.4.1
Thank you!
The code below is from my model:
class Geek < ApplicationRecord
require 'csv'
def self.download_walmart_orders(out)
out ||= "#{Rails.root}/test_orders.csv"
CSV.foreach(out, :headers => true,
:converters => :all,
:header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
) do |row|
geek = Geek.where(customer_order_id: row.to_h["customer_order_id"],
customer_name: row.to_h["customer_name"],
item_sku: row.to_h["item_sku"],
quantity_to_ship: row.to_h["quantity_to_ship"],
total_items_price: row.to_h["total_items_price"]).first_or_create
puts geek
end
end
end
I am assuming that customer_order_id is unique.
You could try something like this -
def self.update_or_create(attributes)
assign_or_new(attributes).save
end
Geek.where(customer_order_id: row.to_h["customer_order_id"]).update_or_create.(
customer_name: row.to_h["customer_name"],
item_sku: row.to_h["item_sku"],
quantity_to_ship: row.to_h["quantity_to_ship"],
total_items_price: row.to_h["total_items_price"])
^^^ Thank you, Michael, for the direction above. I ended up using this code and it worked perfectly. (For a slightly different project but exact same use case) my finalized model code is below:
class Wheel < ApplicationRecord
require 'csv'
def self.update_or_create(attributes)
obj = first || new
obj.assign_attributes(attributes)
obj.save!
end
def self.import(out)
out ||= "#{Rails.root}/public/300-RRW Daily Inv Report.csv"
CSV.foreach(out, :headers => true,
:converters => :all,
:header_converters => lambda { |h| h.downcase.gsub(' ', '_') }
) do |row|
Wheel.where(item: row.to_h["item"]).update_or_create(
item_desc: row.to_h["item_desc"],
total_quantity: row.to_h["total_quantity"])
end
end
end
This is only the second rails app I've ever created so I new at doing this. If there is a better way of doing things then I'm open to hearing your suggestions but keep in mind that I'm learning right now.
I have a method in my model called twilio_api_call. It uses an api to send a text message to a specified phone number pulled out of the database. I would like this method to be executed once per day at 12pm without any user interaction. I am working off my localhost and plan to deploy this app to Heroku. How can I achieve this goal? Maybe pull out this code and place it in some external file that can be called automatically somehow?
require 'twilio-ruby'
class User < ActiveRecord::Base
has_many :selections
has_many :movies, through: :selection
has_secure_password
def self.get_notifications
Movie.find_by_sql("SELECT u.phone, m.title FROM users AS u INNER JOIN movies AS m ON u.id = m.user_id WHERE m.release_date::date = current_date")
end
def self.twilio_api_call
# get user data to parse and send to Twilio api
data = get_notifications
# put your own credentials here
account_sid = 'insert sid here'
auth_token = 'token goes here'
# data is array of hashes. Parse data.
# loop through data hash, build Twilio code
data.each { | key |
phone_number = key["phone"]
message = "Message from MovieTextAlert - the movie: '" + key["title"] + "' has been released today in theaters."
# set up a client to talk to the Twilio REST API
#client = Twilio::REST::Client.new account_sid, auth_token
#client.account.messages.create({
:from => '+18563176098',
:to => phone_number,
:body => message,
})
}
end
end
You can use whenever gem for such periodic actions
every 1.day, :at => '12:00 pm' do
runner "User.twilio_api_call"
end
This code will execute your method everyday at 12:00 pm.
I am testing my Tire / ElasticSearch queries and am having a problem with a custom method I'm including in to_indexed_json. For some reason, it doesn't look like it's getting indexed properly - or at least I cannot filter with it.
In my development environment, my filters and facets work fine and I am get the expected results. However in my tests, I continuously see zero results.. I cannot figure out where I'm going wrong.
I have the following:
def to_indexed_json
to_json methods: [:user_tags, :location_users]
end
For which my user_tags method looks as follows:
def user_tags
tags.map(&:content) if tags.present?
end
Tags is a polymorphic relationship with my user model:
has_many :tags, :as => :tagable
My search block looks like this:
def self.online_sales(params)
s = Tire.search('users') { query { string '*' }}
filter = []
filter << { :range => { :created_at => { :from => params[:start], :to => params[:end] } } }
filter << { :terms => { :user_tags => ['online'] }}
s.facet('online_sales') do
date :created_at, interval: 'day'
facet_filter :and, filter
end
end
end
I have checked the user_tags are included using User.last.to_indexed_json:
{"id":2,"username":"testusername", ... "user_tags":["online"] }
In my development environment, if I run the following query, I get a per day list of online sales for my users:
#sales = User.online_sales(start_date: Date.today - 100.days).results.facets["online_sales"]
"_type"=>"date_histogram", "entries"=>[{"time"=>1350950400000, "count"=>1, "min"=>6.0, "max"=>6.0, "total"=>6.0, "total_count"=>1, "mean"=>6.0}, {"time"=>1361836800000, "count"=>7, "min"=>3.0, "max"=>9.0, "total"=>39.0, "total_count"=>7, "mean"=>#<BigDecimal:7fabc07348f8,'0.5571428571 428571E1',27(27)>}....
In my unit tests, I get zero results unless I remove the facet filter..
{"online_sales"=>{"_type"=>"date_histogram", "entries"=>[]}}
My test looks like this:
it "should test the online sales facets", focus: true do
User.index.delete
User.create_elasticsearch_index
user = User.create(username: 'testusername', value: 'pass', location_id: #location.id)
user.tags.create content: 'online'
user.tags.first.content.should eq 'online'
user.index.refresh
ws = User.online_sales(start: (Date.today - 10.days), :end => Date.today)
puts ws.results.facets["online_sales"]
end
Is there something I'm missing, doing wrong or have just misunderstood to get this to pass? Thanks in advance.
-- EDIT --
It appears to be something to do with the tags relationship. I have another method, ** location_users ** which is a has_many through relationship. This is updated on index using:
def location_users
location.users.map(&:id)
end
I can see an array of location_users in the results when searching. Doesn't make sense to me why the other polymorphic relationship wouldn't work..
-- EDIT 2 --
I have fixed this by putting this in my test:
User.index.import User.all
sleep 1
Which is silly. And, I don't really understand why this works. Why?!
Elastic search by default updates it's indexes once per second.
This is a performance thing because committing your changes to Lucene (which ES uses under the hood) can be quite an expensive operation.
If you need it to update immediately include refresh=true in the URL when inserting documents. You normally don't want this since committing every time when inserting lots of documents is expensive, but unit testing is one of those cases where you do want to use it.
From the documentation:
refresh
To refresh the index immediately after the operation occurs, so that the document appears in search results immediately, the refresh parameter can be set to true. Setting this option to true should ONLY be done after careful thought and verification that it does not lead to poor performance, both from an indexing and a search standpoint. Note, getting a document using the get API is completely realtime.
I am using Ruby on Rails 3.0.7 and I am trying to minimize database hitting. In order to do that I retrieve from the database all Article objects related to a User and then perform a search on those retrieved objects.
What I do is:
stored_objects = Article.where(:user_id => <id>) # => ActiveRecord::Relation
<some_iterative_function_1>.each { |...|
stored_object = stored_objects.where(:status => 'published').limit(1)
...
# perform operation on the current 'stored_object' considered
}
<some_iterative_function_2>.each { |...|
stored_object = stored_objects.where(:visibility => 'public').limit(1)
...
# perform operation on the current 'stored_object' considered
}
<some_iterative_function_n>.each { |...|
...
}
The stored_object = stored_objects.where(:status => 'published') code will really avoid to hitting the database (I ask this because in my log file it seams still run a database query for each iteration)? If no, how can I minimize database hitting?
P.S.: in few words, what I would like to do is to work on the ActiveRecord::Relation (an array of ) but the where method called on it seams to hit the database.
Rails has functionality to grab chunks of the database at one time, then iterate over the rows without having to hit the database again.
See "Retrieving Multiple Objects in Batches" for more information about find_each and find_in_batches.
Once you start iterating over stored_objects (if that's what you're doing), they'll be loaded from the database. If you want to load only the users's published articles, you could do this:
stored_objects = Article.where(:user_id => id, :status => 'published')
If you instead want to load published and unpublished articles and do something different with the published ones, you could do this:
stored_objects = Article.where(:user_id => id)
stored_objects.find_all { |a| a.status == 'published' }. each do |a|
# ... do something with a published article
end
Or perhaps:
Article.where(:user_id => id).each do |article|
case article.status
when 'published'
# ... do something with a published article
else
# ... do something with an article that's not published
end
end
Each of these examples performs only one database query. Choosing which one depends on which data you really want to work with.
Quick summary:
I have a Rails app that is a personal checklist / to-do list. Basically, you can log in and manage your to-do list.
My Question:
When a user creates a new account, I want to populate their checklist with 20-30 default to-do items. I know I could say:
wash_the_car = ChecklistItem.new
wash_the_car.name = 'Wash and wax the Ford F650.'
wash_the_car.user = #new_user
wash_the_car.save!
...repeat 20 times...
However, I have 20 ChecklistItem rows to populate, so that would be 60 lines of very damp (aka not DRY) code. There's gotta be a better way.
So I want to use seed the ChecklistItems table from a YAML file when the account is created. The YAML file can have all of my ChecklistItem objects to be populated. When a new user is created -- bam! -- the preset to-do items are in their list.
How do I do this?
Thanks!
(PS: For those of you wondering WHY I am doing this: I am making a client login for my web design company. I have a set of 20 steps (first meeting, design, validate, test, etc.) that I go through with each web client. These 20 steps are the 20 checklist items that I want to populate for each new client. However, while everyone starts with the same 20 items, I normally customize the steps I'll take based on the project (and hence my vanilla to-do list implementation and desire to populate the rows programatically). If you have questions, I can explain further.
Just write a function:
def add_data(data, user)
wash_the_car = ChecklistItem.new
wash_the_car.name = data
wash_the_car.user = user
wash_the_car.save!
end
add_data('Wash and wax the Ford F650.', #user)
I agree with the other answerers suggesting you just do it in code. But it doesn't have to be as verbose as suggested. It's already a one liner if you want it to be:
#new_user.checklist_items.create! :name => 'Wash and wax the Ford F650.'
Throw that in a loop of items that you read from a file, or store in your class, or wherever:
class ChecklistItem < AR::Base
DEFAULTS = ['do one thing', 'do another']
...
end
class User < AR::Base
after_create :create_default_checklist_items
protected
def create_default_checklist_items
ChecklistItem::DEFAULTS.each do |x|
#new_user.checklist_items.create! :name => x
end
end
end
or if your items increase in complexity, replace the array of strings with an array of hashes...
# ChecklistItem...
DEFAULTS = [
{ :name => 'do one thing', :other_thing => 'asdf' },
{ :name => 'do another', :other_thing => 'jkl' },
]
# User.rb in after_create hook:
ChecklistItem::DEFAULTS.each do |x|
#new_user.checklist_items.create! x
end
But I'm not really suggesting you throw all the defaults in a constant inside ChecklistItem. I just described it that way so that you could see the structure of the Ruby object. Instead, throw them in a YAML file that you read in once and cache:
class ChecklistItem < AR::Base
def self.defaults
##defaults ||= YAML.read ...
end
end
Or if you wand administrators to be able to manage the default options on the fly, put them in the database:
class ChecklistItem < AR::Base
named_scope :defaults, :conditions => { :is_default => true }
end
# User.rb in after_create hook:
ChecklistItem.defaults.each do |x|
#new_user.checklist_items.create! :name => x.name
end
Lots of options.
A Rails Fixture is used to populate test-data for unit tests ; Dont think it's meant to be used in the scenario you mentioned.
I'd say just Extract a new method add_checklist_item and be done with it.
def on_user_create
add_checklist_item 'Wash and wax the Ford F650.', #user
# 19 more invocations to go
end
If you want more flexibility
def on_user_create( new_user_template_filename )
#read each line from file and call add_checklist_item
end
The file can be a simple text file where each line corresponds to a task description like "Wash and wax the Ford F650.". Should be pretty easy to write in Ruby,