I am trying to get series from the movie database in multiple languages.
My goal is to update the object returned by the API to have the following keys :
name_en, name_fr, overview_en, overview_fr
I have this piece of code :
require 'httparty'
class GetSeriesJob < ApplicationJob
queue_as :default
API_URL = 'https://api.themoviedb.org/3/'
LANGS = {'fr' => '&language=fr-FR', 'en' => '&language=en-EN'}
def perform
get_series
end
private
def get_series
seriesArray = []
LANGS.each do |lang|
series = HTTParty.get(API_URL + 'tv/popular?api_key=' + ENV['API_KEY'] + lang[1])
tmp = JSON.parse(series.body)['results']
tmp.each do |t|
if seriesArray.detect{ |s| s['id'] == t['id'] }
t['name_' + lang[0]] = t['name']
t['overview_' + lang[0]] = t['overview']
else
t['name_' + lang[0]] = t.delete t['name']
t['overview_' + lang[0]] = t.delete t['overview']
end
end
seriesArray = tmp
end
puts seriesArray
return seriesArray
end
end
Which return (for one series) :
[["original_name", "The Expanse"], ["name_en", "The Expanse"], ["popularity", "66.679301"], ["origin_country", "[\"US\"]"], ["vote_count", 564], ["first_air_date", "2015-12-14"], ["backdrop_path", "/beIjmWr3OBOtcWK4tKMObOIDJ1C.jpg"], ["original_language", "en"], ["vote_average", 7.5], ["overview_en", "A thriller set two hundred years in the future following the case of a missing young woman who brings a hardened detective and a rogue ship's captain together in a race across the solar system to expose the greatest conspiracy in human history."], ["poster_path", "/prJFWxJ0x8tBPTliMjj51wgYnSK.jpg"], ["episode_run_time", "[43]"], ["number_of_seasons", 3], ["external_id", 63639], ["created_at", "2018-06-24 13:40:16.143952"], ["updated_at", "2018-06-24 13:40:16.143952"]]
And I would expect the following result :
[["original_name", "The Expanse"], ["name_en", "The Expanse"], ["name_fr", "The Expanse"]["popularity", "66.679301"], ["origin_country", "[\"US\"]"], ["vote_count", 564], ["first_air_date", "2015-12-14"], ["backdrop_path", "/beIjmWr3OBOtcWK4tKMObOIDJ1C.jpg"], ["original_language", "en"], ["vote_average", 7.5], ["overview_en", "A thriller set two hundred years in the future following the case of a missing young woman who brings a hardened detective and a rogue ship's captain together in a race across the solar system to expose the greatest conspiracy in human history."], ["overview_fr", "overview in French."] ["poster_path", "/prJFWxJ0x8tBPTliMjj51wgYnSK.jpg"], ["episode_run_time", "[43]"], ["number_of_seasons", 3], ["external_id", 63639], ["created_at", "2018-06-24 13:40:16.143952"], ["updated_at", "2018-06-24 13:40:16.143952"]]
I am using ruby on rails 5.2
So, I found myself a way to solve my issue, here are the relevant modifications
def get_series
seriesArray = []
LANGS.each do |lang|
series = HTTParty.get(API_URL + 'tv/popular?api_key=' + ENV['API_KEY'] + lang[1])
tmp = JSON.parse(series.body)['results']
tmp.each do |t|
if serie = seriesArray.detect{ |s| s['id'] == t['id'] }
serie['name_' + lang[0]] = t['name']
serie['overview_' + lang[0]] = t['overview']
serie['genre_' + lang[0]] = t['genre_ids']
serie.delete('name')
serie.delete('overview')
serie.delete('genre_ids')
else
t['name_' + lang[0]] = t['name']
t['overview_' + lang[0]] = t['overview']
t['genre_' + lang[0]] = t['genre_ids']
end
end
if seriesArray.count == 0
seriesArray = tmp
end
end
return seriesArray
end
Related
My Class:
class TtsGoogle
def initialize
#google_client = Google::Cloud::TextToSpeech.text_to_speech
end
def write_file(topic)
text = topic.content
spaces = text.enum_for(:scan, /(?=\s)/).map { Regexp.last_match.offset(0).first }
start = finish = limit = 0
result = ''
while true do
#Google TTS limit 5000
limit += 4800
if limit < text.length
finish = spaces.select {|n| n > limit }.first
result += get_audio_content(text[start..finish])
start = finish
else
result += get_audio_content(text[start..text.length])
break
end
end
File.open "#{topic.name.gsub('/', ' ')}.wav", "wb" do |file|
file.write result
end
end
def delete_file(name)
File.delete("#{name.gsub('/', ' ')}.wav")
end
private
def get_audio_content(text)
synthesis_input = { text: text }
response = #google_client.synthesize_speech(
input: synthesis_input,
voice: voice_config,
audio_config: audio_config
)
response.audio_content
end
def voice_config
#voice_config ||= {
language_code: "en-US",
ssml_gender: "NEUTRAL",
name: "en-US-Wavenet-B"
}
end
def audio_config
#audio_config = { audio_encoding: "LINEAR16", speaking_rate: 0.90 }
end
end
And when I try to create audio from big text, I have this trouble:
Google::Cloud::ResourceExhaustedError 8:Received message larger than max (5250205 vs. 4194304)
With small text it work, when audio on the out less then 4MB. How I can increase or disable this 4MB limit? I found some solution for PHP, where used next way:
-$client = new TextToSpeechClient();
+$client = new TextToSpeechClient([
'transportConfig' => [
'grpc' => [
'stubOpts' => [
'grpc.max_receive_message_length' => -1,
'grpc.max_send_message_length' => -1
]
]
]
]);
But I don't know, how to fix this issue with Rails. Help please.
Good afternoon!
I have a function where she does the bank reconciliation with a .txt of the bank with the system, where she shows the number of the launch whose value is exactly equal to the one launched in the system, the problem is that there is 1 debit in the .txt that corresponds to ha several launches in the system of the same client id, in .txt I do not have the client id but the sum of the values launched corresponds exactly to the debit, is there any way to do this query? ... I am trying to use the in the variable sum_lancamentos having for me returns the ids but is generating synthase error. and I believe that all this function could be improved, I just don’t know how, I’m a new RoR and I’m still getting used to good practices.
Any help is very life and thank you in advance!
def conciliacao
#conciliacao = session[:conciliacao_file]
comparacao = {}
#conciliacao.each do |key, line|
data = line[:data]
valor = line[:valor].to_f.round(2)
if line[:mov] == "D"
despesa = true
else
despesa = false
end
lancamentos = Lancamento.ativos.pagos.where(conta_id: params[:conta_id], despesa: despesa).where("lancamentos.data_pagamento BETWEEN '#{data.to_date.beginning_of_day.to_s(:db)}' AND '#{data.to_date.end_of_day.to_s(:db)}'").where(["cast(lancamentos.valor AS NUMERIC(15,2) ) = :value or
cast(lancamentos.valor_pago AS NUMERIC(15,2)) = :value ",
{ value: line[:valor] }])
unless lancamentos.blank?
lancamentos.each do |lancamento|
#puts line
#conciliacao[key][:lancamentos] = "#{lancamento.id}"
#conciliacao[key][:status] = 1
#conciliacao[key][:color] = "green lighten-4"
end
else
sum_lancamentos = Lancamento.ativos.pagos.group(:cliente_id).where(conta_id: params[:conta_id], despesa: despesa).where("lancamentos.data_pagamento BETWEEN '#{data.to_date.beginning_of_day.to_s(:db)}' AND '#{data.to_date.end_of_day.to_s(:db)}'").having(["sum(cast(lancamentos.valor AS NUMERIC(15,2) )) = :value or sum(cast(lancamentos.valor_pago AS NUMERIC(15,2))) = :value ", { value: line[:valor] }])
unless sumlancamentos.blank?
#conciliacao[key][:lancamentos] = "#{sum_lancamentos.ids}"
#conciliacao[key][:status] = 1
#conciliacao[key][:color] = "green lighten-4"
end
end
end
session.delete(:conciliacao_file)
end
I'm completely new to Ruby on Rails but I think I might be missing something obvious. I'm currently working on a webapp that scrapes auction websites. The bones of the app was created by someone else. I'm currently trying to add new website scrapes but they don't seem to be working.
I have read through some of the Nokogiri documentation, checked that the scraped information is indeed not being written to the database (the seeded URLs that are being targeted have been when I check via the rails console) and used the chrome extension CSS Selector Tester to check that I am targeting the correct CSS selectors. The record ids are correct when I check via the rails console.
I have put what I think are the important sections of code below, but I might be missing something that I don't realise is important.
The websites I'm having issues with are Lot-art.com & Lot-Tissimo.com
Any help will be much appreciated.
Seeded URLs
Source.create(name: "Auction.fr", query_template: "https://www.auction.fr/_en/lot/search/?contexte=futures&tri=date_debut%20ASC&query={query}&page={page}")
Source.create(name: "Invaluable.co.uk", query_template: "https://www.invaluable.co.uk/search/api/search-results?keyword={query}&size=1000")
Source.create(name: "Interencheres.com", query_template: "http://www.interencheres.com/en/recherche/lot?search%5Bkeyword%5D={query}&page={page}")
Source.create(name: "Gazette-drouot.com", query_template: "http://catalogue.gazette-drouot.com/html/g/recherche.jsp?numPage={page}&filterDate=1&query={query}&npp=100")
Source.create(name: "Lot-art.com", query_template: "http://www.lot-art.com/auction-search/?form_id=lot_search_form&page=1&mq=&q={query}&ord=recent")
Source.create(name: "Lot-tissimo.com", query_template: "https://lot-tissimo.com/en/cmd=s&lwr=&ww={query}&xw=&srt=SN&wg=EUR&page={page}")
Scheduler code
require 'rufus-scheduler'
require 'nokogiri'
require 'mechanize'
require 'open-uri'
require "net/https"
s = Rufus::Scheduler.singleton
s.interval '1m' do
setting = Setting.find(1)
agent = Mechanize.new
agent.user_agent_alias = 'Windows Chrome'
agent.cookie_jar.load(File.join(Rails.root, 'tmp/cookies.yaml'))
List.all.each do |list|
number_of_new_items = 0
list.actions.each do |action|
url = action.source.query_template.gsub('{query}', action.list.query)
case action.source.id
when 1 # Auction.fr
20.downto(1) do |page|
doc = Nokogiri::HTML(open(url.gsub('{page}', page.to_s)))
doc.css("div.list-products > ul > li").reverse.each do |item_data|
price = 0
if item_data.at_css("h3.h4.adjucation.ft-blue") && /Selling price : ([\d\s]+) €/.match(item_data.at_css("h3.h4.adjucation.ft-blue").text)
price = /Selling price : ([\d\s]+) €/.match(item_data.at_css("h3.h4.adjucation.ft-blue").text)[1].gsub(" ", "")
end
item = action.items.new(
title: item_data.at_css("h2").text.strip,
url: item_data.at_css("h2 a")["href"],
picture: item_data.at_css("div.image-wrap.lazy div.image img")["src"],
price: price,
currency: "€"
)
ActiveRecord::Base.logger.silence do # This disable writing logs
if item.save
number_of_new_items = number_of_new_items + 1
end
end
end
end
when 97 # Lot-Tissimo.com
5.downto(1) do |page|
doc = Nokogiri::HTML(open(url.gsub('{page}', page.to_s)))
doc.css("#inhalt > .objektliste").reverse.each do |item_data|
# price = 0
# if item_data.at_css("h3.h4.adjucation.ft-blue") && /Selling price : ([\d\s]+) €/.match(item_data.at_css("h3.h4.adjucation.ft-blue").text)
# price = /Selling price : ([\d\s]+) €/.match(item_data.at_css("h3.h4.adjucation.ft-blue").text)[1].gsub(" ", "")
# end
item = action.items.new(
title: item_data.at_css("div.objli-desc").text.strip,
url: item_data.at_css("td.objektliste-foto a")["href"],
picture: item_data.at_css("td.objektliste-foto a#lot_link img")["src"],
price: price,
currency: "€"
)
ActiveRecord::Base.logger.silence do # This disable writing logs
if item.save
number_of_new_items = number_of_new_items + 1
end
end
end
end
when 2 # Invaluable.co.uk
doc = JSON.parse(open(url).read)
doc["itemViewList"].reverse.each do |item_data|
puts item_data["itemView"]["photos"]
item = action.items.new(
title: item_data["itemView"]["title"],
url: "https://www.invaluable.co.uk/buy-now/" + item_data["itemView"]["title"].parameterize + "-" + item_data["itemView"]["ref"],
picture: item_data["itemView"]["photos"] != nil ? item_data["itemView"]["photos"].first["_links"]["medium"]["href"] : nil,
price: item_data["itemView"]["price"],
currency: item_data["itemView"]["currencySymbol"]
)
ActiveRecord::Base.logger.silence do # This disable writing logs
if item.save
number_of_new_items = number_of_new_items + 1
end
end
end
when 3 # Interencheres.com
# doc = Nokogiri::HTML(open(url))
5.downto(1) do |page|
doc = Nokogiri::HTML(open(url.gsub('{page}', page.to_s)))
doc.css("div#lots_0 div.ligne_vente").reverse.each do |item_data|
price = 0
item = action.items.new(
title: item_data.at_css("div.ph_vente div.des_vente p a").text.strip,
url: "http://www.interencheres.com" + item_data.at_css("div.ph_vente div.des_vente p a")["href"],
picture: item_data.at_css("div.ph_vente div.gd_ph_vente img")["src"],
price: price,
currency: "€"
)
ActiveRecord::Base.logger.silence do # This disable writing logs
if item.save
number_of_new_items = number_of_new_items + 1
end
end
end
end
when 4 # Gazette-drouot.com
5.downto(1) do |page|
# doc = Nokogiri::HTML(open(url.gsub('{page}', page.to_s)))
doc = agent.get(url.gsub('{page}', page.to_s))
# doc = agent.get(url)
doc.css("div#recherche_resultats div.lot_recherche").reverse.each do |item_data|
price = 0
picture = item_data.at_css("img.image_thumb_recherche") ? item_data.at_css("img.image_thumb_recherche")["src"] : nil
item = action.items.new(
title: item_data.at_css("#des_recherche").text.strip.truncate(140),
url: "http://catalogue.gazette-drouot.com/html/g/" + item_data.at_css("a.lien_under")["href"],
picture: picture,
price: price,
currency: "€"
)
ActiveRecord::Base.logger.silence do # This disable writing logs
if item.save
number_of_new_items = number_of_new_items + 1
end
end
end
end
when 69 # Lot-art.com
doc = agent.get(url)
doc.css("div.lot_list_holder").reverse.each do |item_data|
price = 0
item = action.items.new(
title: item_data.at_css("div.lot_list_body a")[0].text.strip.truncate(140),
url: item_data.at_css("div.lot_list_body")["href"],
picture: item_data.at_css("a.lot_list_thumb img") ["src"],
price: price,
currency: "€"
)
ActiveRecord::Base.logger.silence do # This disable writing logs
if item.save
number_of_new_items = number_of_new_items + 1
end
end
end
end
end
if number_of_new_items > 0 && setting.notifications_per_hour > setting.notifications_this_hour && setting.pushover_app_token.present? && setting.pushover_user_key.present?
url = URI.parse("https://api.pushover.net/1/messages.json")
req = Net::HTTP::Post.new(url.path)
req.set_form_data({
:token => setting.pushover_app_token,
:user => setting.pushover_user_key,
:message => "#{number_of_new_items} new items on #{list.name}!",
:url_title => "Check the list",
:url => "http://spottheauction.com/lists/#{list.id}"
})
res = Net::HTTP.new(url.host, url.port)
res.use_ssl = true
res.verify_mode = OpenSSL::SSL::VERIFY_PEER
res.start {|http| http.request(req) }
end
end
agent.cookie_jar.save(File.join(Rails.root, 'tmp/cookies.yaml'))
end
s.cron '0 * * * *' do
setting = Setting.find(1)
setting.notifications_this_hour = 0
setting.save
end
new just initializes an instance but doesn't save the instance. Do you actually call save somewhere?
You have two options:
Call save on the item:
item = action.items.new(
# ...
)
item.save
Or use create instead of new:
item = action.items.create(
# ...
)
In case someone else comes across this. I got the scraping of lot-art.com to work. It seemed that I was lacking specificity in the css selector for nokogiri to pull the correct data.
I am still having continuing issues with lot-tissimo although that appears to be from something else as other scrapers have issues such as scraping-hub's portia spiders.
I have an application in Rails that I run on Heroku (apprx 1 000 page views a day). I have been experiencing frequent crashes of the application since the launch last week.
Looking into New Relic it seems like the Dynos memory usage is constantly increasing without ever going down in memory usage. Basically, it builds up during a couple of hours and then end in request timeouts, which seems likely.
Thus, I believe the problem with the app crashing is due to a memory leak.
My app (presenttips . com) is a gift website where I have features like "random gift", "gift of the day" and "banners". These I load in the application controller like this:
before_filter :global_setup
def global_setup
# Create random gift
rand_gift = []
rand_gift << Gift.where(:gift_status_id => 1) #=> Accepted
#random_gift = rand_gift[0][rand(rand_gift[0].size) - 1]
rand_gift = nil
#nbr_of_active_gifts = (Gift.where(:gift_status_id => 1).count / 100 ).round * 100
#toplist = Gift.where(:gift_status_id => 1).order("week_click DESC").limit(20)
#banners = Banner.where("first_date <= '" + Time.now.to_date.to_s + "'").where("last_date >= '" + Time.now.to_date.to_s + "'").order("first_date ASC")
advertise_here = []
(#banners.count..4).each do |i|
advertise_here[i] = Banner.new(:advertiser => "Presenttips.com", :banner_image => "annons.jpg", :url => advertise_path)
end
#banners << advertise_here.compact
#banners = #banners.flatten
#page_categories = PageCategory.order(:prio_rank)
if Rails.env.production?
#random_sql = "RANDOM()"
#meta_robots_block = false
#analytics_block = false
else
#meta_robots_block = true
#analytics_block = true
#random_sql = "RAND()"
end
gift_from_daily = DailyGift.where(:publish_date => Time.now.to_date).first
gift_from_daily = DailyGift.create(:publish_date => Time.now.to_date, :gift_id => #random_gift.id) if gift_from_daily.blank?
#daily_gift = Gift.find(gift_from_daily.gift_id)
#head_categories = Category.order(:name).where(:parent_id => nil)
todays_date = Time.now.to_date.to_s
#season = Season.where("'" + todays_date + "' >= date_start ", "'" + todays_date + "' <= date_end" ).first
#season_theme = #season.css
#logo = 'logo.png'
#logo = 'seasons/logo_christmas.png' if #season.css.eql?('theme_christmas.css')
end
so that I can use them in the app globabally (gift of the day, for example, is always presenet in the right column).
I guess this is not great considering memory usage though.
My questions:
Is this likely to cause the memory build-up?
What would be a smarter way to do this, in that case?
I remove almost all of these variables and it still didn't help. I will assume the application controller was not causing the memory problem.
I have a some code like this users_controller.rb, I need to print line no 30 in development.log
line#29 def selectrole
line#30 #userrole = RolesUser.find(:all, :conditions =>["p.user_id = ? and p.status = ? ",session[:user_id], params['status']])
line#31 logger.debug print_line(30)
line#32 end
Can I see 30th line in my development.log like this
#userrole = RoleUser.find(:all, :conditions => ["p.user_id = ? and p.status = ? ", 1234, 'Active'])
What is the approach to write "print_line" function? Here is the my print_line code?
def print_line(file_name, line)
counter = 1
printline = "-------- NO SUCH LINE --------"
File.open(file_name, "r") do |infile|
while (line_text = infile.gets)
if counter == line
printline = "#{counter} :: " + line_text
break
end
counter += 1
end
end
printline
end
from this function I am getting like this
#userrole = RolesUser.find(:all, :conditions =>["p.user_id = ? and p.status = ? ",session[:user_id], params['status']])
Is their any way to find and replace the variables with their respective values?
Assuming that you're primarily interested in knowing the contents of your :conditions, why not just do something like this:
def selectrole
conditions = ["p.user_id = ? and p.status = ? ",session[:user_id], params['status']]
logger.debug(conditions)
#userrole = RolesUser.find(:all, :conditions => conditions)
end