Does rake task limit functionality of my class to call external API? - ruby-on-rails

AllegroAPI is a class in the /models directory that calls an external API. It works as I wish when I test in somewhere else not by running rake task.
Example working code:
require "./AllegroAPI"
allegro = AllegroAPI.new(login: 'LOGIN',
password: File.read('XXXX.txt'),
webapikey: File.read('XXX.txt')
)
puts allegro.do_search({"search-string"=>"nokia",
"search-price-from"=>300.0,
"search-price-to"=>500.0,
"search-limit"=>50}).to_s
As I've said it works correctly. It calls the API and prints out the result.
File allegro.rb is also in the models directory and it's a file I'm executing by running this task:
namespace :data do
desc "Update auctions table in database"
task update_auctions: :environment do
Allegro.check_for_new_auctions
end
end
allegro.rb:
module Allegro
require 'AllegroAPI'
def self.check_for_new_auctions
allegro = AllegroAPI.new(login: 'LOGIN',
password: File.read('app/models/ignore/XXXX.txt'),
webapikey: File.read('app/models/ignore/XXX.txt')
)
looks = Look.all
looks.each do |l|
hash_to_ask = ActiveSupport::JSON.decode(l[:look_query]).symbolize_keys
hash_to_ask = hash_to_ask.each_with_object({}) do |(k,v), h|
if v.is_number?
h[k.to_s.split('_').join('-')] = v.to_f
else
h[k.to_s.split('_').join('-')] = v
end
end
results = allegro.do_search(hash_to_ask)
#do something with data
end
end
end
The problem is that it doesn't return anything. var result is not nil, but it does not hold anything.
When I'm trying to debug it and call API from the inside do_search function it's calling API, doesn't raise a error but response is nothing. AllegroAPI works correctly. There is no problem with var "hash_to_ask", it's exactly the same hash as in working example.
EDIT:
I've commented out check_for_new_auctions and used "puts", it works fine when I run it by executing rake task. Then I've used exactly the same code which I used in normal file which have ran properly:
class Allegro
def self.check_for_new_auctions
allegro = AllegroAPI.new(login: 'LOGIN',
password: File.read('app/models/ignore/XXXX.txt'),
webapikey: File.read('app/models/ignore/XXXX.txt')
)
hash_to_ask = {"search-string"=>"nokia",
"search-price-from"=>300.0,
"search-price-to"=>500.0,
"search-limit"=>50}
allegro.do_search(hash_to_ask).to_s
end
end
It have not worked;/ The returned value from allegro.do_search(hash_to_ask) is hash, not empty, not nil but when I try to print it, it's nothing, empty place.
EDIT:
Everything have worked properly, waste like 15 hours total debugging the problem which have not existed. I'm not sure why it have not worked but it couldn't print to the console after converting to string, so I tried writing it down to file blindly. What I have found in the text file? Data.
I don't know why it couldn't print out everything in the console.

In the IRB script that you show, you have some puts statement that is not in your rake task. So for debugging, I would add puts ... to your Rake task, e.g.:
namespace :data do
desc "Update auctions table in database"
task update_auctions: :environment do
puts "Start Auctions..."
results = Allegro.check_for_new_auctions
puts "Results: #{results}"
end
end
Now, when you run:
rake data:update_auctions
You should get some output. Otherwise rinse-and-repeat by adding puts statements in the method that you are calling.

Related

How to check status with a rake task

i'm trying to make a rake task to run it with scheduler on heroku, but first im testing locally so i have a method where i check the status of polls like this
def check_status
if Date.today.between?(self.start_date, self.expiration_date)
self.poll_active = true
else
self.poll_active = false
end
end
and its working great but now i want this exact method to run it with a task.
i create my task file
namespace :change_poll_status do
task :poll_status => :environment do
if Date.today.between?(Poll.start_date, Poll.expiration_date)
Poll.poll_active = true
puts "It works"
else
Poll.poll_active = false
puts "no"
end
end
end
but when i run rake change_poll_status:poll_status
nothing happens it just skip like there is nothing to run, no errors, nothing.
The error is in this line:
if Date.today.between?(Poll.start_date, Poll.expiration_date)
You're trying to compare today's date with two class methods, start_date and expiration_date. These methods don't exist on the Poll class.
To fix this, you need to first retrieve an instance of the Poll class, and then call the methods on that instance. For example:
poll = Poll.first
if Date.today.between?(poll.start_date, poll.expiration_date)
poll.poll_active = true
puts "It works"
else
poll.poll_active = false
puts "no"
end

Rspec Rake Task how to parse/simulate user input?

In my rake task I have:
namespace :example do
desc "this does something"
task :something, [:arg1] => :environment do |t, args|
(some_irrelevant_code)
print 'YES/ NO : '
choice = STDIN.gets.chomp.upcase
case choice
when 'YES'
do_something
break
when 'NO'
break
end
end
end
In my spec I have:
require "spec_helper"
require "rake"
feature "Example" do
before do
load File.expand_path("../../../lib/tasks/example.rake", __FILE__)
Rake::Task.define_task(:environment)
end
scenario "something" do
Rake.application.invoke_task("example:something[rake_args_here]")
end
All is working fine, although I am having troubles finding a way to avoid having to type the user input in the console when running the test.
Basically I want the test to run and assume that the user is going to type "YES".
Please let me know if you have a solution for this or point me in the right direction.
Thanks in advance.
If you use STDIN, you're stuck, that's a constant. It's worth noting that using STDIN is not recommended because of this limitation.
If you use $stdin, the global variable equivalent and modern replacement, you can reassign it:
require 'stringio'
$stdin = StringIO.new("fake input")
$stdin.gets.chomp.upcase
# => "FAKE INPUT"
That means you can, for testing purposes, rework $stdin. You'll want to put it back, though, which means you need a wrapper like this:
def with_stdin(input)
prev = $stdin
$stdin = StringIO.new(input)
yield
ensure
$stdin = prev
end
So in practice:
with_stdin("fake input") do
puts $stdin.gets.chomp.upcase
end
You should stub STDIN object like this STDIN.stub(gets: 'test')
or
allow(STDIN).to receive(:gets).and_return('test')
If both of them do not work then try:
allow(Kernel).to receive(:gets).and_return('test')

Logging raw SQL errors in Rake Tasks

I'm using raw sql bulk updates (for performance reasons) in the context of a rake task. Something like the following:
update_sql = Book.connection.execute("UPDATE books AS b SET
stock = vs.stock,
promotion = vs.promotion,
sales = vs.sales
FROM (values #{values_string}) AS vs
(stock, promotion, sales) WHERE b.id = vs.id;")
While everything is "transparent" in local development, if this SQL fails in production during the execution of the rails task (for example because the promotion column is nil and the statement becomes invalid), no error is logged.
I can manually log this with catching the exception, like below, however some option that would allow for automatic logging would be better.
begin
...
rescue ActiveRecord::StatementInvalid => e
Rails.logger.fatal "Books update: ActiveRecord::StatementInvalid: "+ e.to_s
end
You can make your own custom class in your model folder:
app/models/custom_sql_logger.rb :
class CustomSqlLogger
def self.debug(msg=nil)
#custom_log ||= Logger.new("#{Rails.root}/log/custom_sql.log")
#custom_log.debug(msg) unless msg.nil?
end
end
Then go to the rake task where you would like to debug updated fields for example lib/task/calculate_avarages.rake and call your custom debugger:
CustomSqlLogger.debug "The field was successfully updated into DB"
Example from my project:
require 'rake'
task :calculate_averages => :environment do
products = Product.all
products.each do |product|
puts "Calculating average rating for #{product.name}..."
product.update_attribute(:average_rating, product.reviews.average("rating"))
CustomSqlLogger.debug "#{product.name} was susscefully updated into DB"
end
end
Custom debugger will create the new file custom_sql.log into log folder: log/custom_sql.log and saved all information there. Beware of a log file size after a while.

How to continue indexing documents in elasticsearch(rails)?

So I ran this command rake environment elasticsearch:import:model CLASS='AutoPartsMapper' FORCE=true to index documents in elasticsearch.In my database I have 10 000 000 records=)...it takes (I think) one day to index this...When indexing was running my computer turned off...(I indexed 2 000 000 documents)Is it possible to continue indexing documents?
If you use rails 4.2+ you can use ActiveJob to schedule and leave it running. So, first generate it with this
bin/rails generate job elastic_search_index
This will give you class and method perform:
class ElasticSearchIndexJob < ApplicationJob
def perform
# impleement here indexing
AutoPartMapper.__elasticsearch__.create_index! force:true
AutoPartMapper.__elasticsearch__.import
end
end
Set the sidekiq as your active job provider and from console initiate this with:
ElasticSearchIndexJob.perform_later
This will set the active job and execute it on next free job but it will free your console. You can leave it running and check the process in bash later:
ps aux | grep side
this will give you something like: sidekiq 4.1.2 app[1 of 12 busy]
Have a look at this post that explains them
http://ruby-journal.com/how-to-integrate-sidekiq-with-activejob/
Hope it helps
There is no such functionality in elasicsearch-rails afaik but you could write a simple task to do that.
namespace :es do
task :populate, [:start_id] => :environment do |_, args|
start_id = args[:start_id].to_i
AutoPartsMapper.where('id > ?', start_id).order(:id).find_each do |record|
puts "Processing record ##{record.id}"
record.__elasticsearch__.index_document
end
end
end
Start it with bundle exec rake es:populate[<start_id>] passing the id of the record from which to start the next batch.
Note that this is a simplistic solution which will be much slower than batch indexing.
UPDATE
Here is a batch indexing task. It is much faster and automatically detects the record from which to continue. It does make an assumption that previously imported records were processed in increasing id order and without gaps. I haven't tested it but most of the code is from a production system.
namespace :es do
task :populate_auto => :environment do |_, args|
start_id = get_max_indexed_id
AutoPartsMapper.find_in_batches(batch_size: 1000).where('id > ?', start_id).order(:id) do |records|
elasticsearch_bulk_index(records)
end
end
def get_max_indexed_id
AutoPartsMapper.search(aggs: {max_id: {max: {field: :id }}}, size: 0).response[:aggregations][:max_id][:value].to_i
end
def elasticsearch_bulk_index(records)
return if records.empty?
klass = records.first.class
klass.__elasticsearch__.client.bulk({
index: klass.__elasticsearch__.index_name,
type: klass.__elasticsearch__.document_type,
body: elasticsearch_records_to_index(records)
})
end
def self.elasticsearch_records_to_index(records)
records.map do |record|
payload = { _id: record.id, data: record.as_indexed_json }
{ index: payload }
end
end
end

Rails - Help with rake task

I have a rake task I need to run in order to sanitize (remove forward slashes) some data in the database. Here's the task:
namespace :db do
desc "Remove slashes from old-style URLs"
task :substitute_slashes => :environment do
puts "Starting"
contents = Content.all
contents.each do |c|
if c.permalink != nil
c.permalink.gsub!("/","")
c.save!
end
end
puts "Finished"
end
end
Which allows me to run rake db:substitute_slashes --trace
If I do puts c.permalink after the gsub! I can see it's setting the attribute properly. However the save! doesn't seem to be working because the data is not changed. Can someone spot what the issue may be?
Another thing, I have paperclip installed and this task is triggering [paperclip] Saving attachments. which I would rather avoid.
try this:
namespace :db do
desc "Remove slashes from old-style URLs"
task :substitute_slashes => :environment do
puts "Starting"
contents = Content.all
contents.each do |c|
unless c.permalink.nil?
c.permalink = c.permalink.gsub(/\//,'')
c.save!
end
end
puts "Finished"
end
end
1.) Change != nil to unless record.item.nil? (I don't know if it makes a different, but I've never used != nil. You may want to use .blank? also judging by your code)
2.) Your gsub was malformed. The pattern must be between two / (/ stuff /). The \ is necessary because you need to escape the /.
3.) Bang (!) updates the object in place. I think your biggest issue may be that you are overusing !.
4.) You're also making this very inefficient... You are looking at every record and updating every record. Rails isn't always the best option. Learn SQL and do this in one line:
"UPDATE contents SET permalink = replace(permalink, '/', '');"
If you MUST use Rails:
ActiveRecord::Base.connection.execute "UPDATE contents SET permalink = replace(permalink, '/', '');"
Wow! One query. Amazing! :)
The next thing I would try would be
c.permalink = c.permalink.gsub("/","")
As for saving without callbacks, this stackoverflow page has some suggestions.

Resources