I am interacting with openAI using ruby-openAI gem, but I get timeout error, is there a way I can exceed the timeout limit?
response = #client.completions(
parameters: {
model: "text-davinci-003",
prompt: "In the style of #{#as_written_by}, write a longer article in HTML of at least 750 words using the #{article} as the primary source and basis for the new article, and include interesting facts from the #{secondary_sources}, with a tags around the source of the information pointing to the original URLs",
max_tokens: 3000
})
Related
I am trying to get some basic information from the Steam Community via the steam-condenser gem and so far the Steam.new seems to work just fine with all the players information.
however when I do this (example)
player = SteamId.new("tiger")
stats = player.fetch_games
I get the following error message
Traceback (most recent call last):
1: from lib/assets/ruby/test.rb:15:in `<main>'
/home/zigs/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/steam-condenser-1.3.11/lib/steam/community/steam_id.rb:326:in `fetch_games': undefined method `[]' for nil:NilClass (NoMethodError)
A lot of the information I need seems to be connected to the fetch_games (for example the method total_playtime(id))
Not sure why this is not working. I am lost. Any help or ideas are highly appreciated! Thank you!
TLDR; it looks like this gem no longer works.
the particular module that you're having trouble with is:
def fetch_games
games_data = parse "#{base_url}/games?xml=1"
#games = {}
#recent_playtimes = {}
#total_playtimes = {}
games_data['games']['game'].each do |game_data|
app_id = game_data['appID'].to_i
#games[app_id] = SteamGame.new app_id, game_data
recent = game_data['hoursLast2Weeks'].to_f
total = (game_data['hoursOnRecord'] || '').delete(',').to_f
#recent_playtimes[app_id] = (recent * 60).to_i
#total_playtimes[app_id] = (total * 60).to_i
end
true
end
with the particular problem statement being games_data['games']['game'].each
If we were looking to get information for a particular user, it downloads an XML document about the user from a URL looking like:
http://steamcommunity.com/id/demomenz?xml=1
and this file does not seem to contain any games objects in it.
Having looked at the codebase for the steam-condenser gem; it hasn't really been updated in about 6 years. I can only assume that the XML format has been modified since this time and that the gem will no longer work.
Valve has added more privacy options to Steam Community profiles which are not reflected in the old XML APIs.
Apparently, the profile in question (tiger) has it‘s game details set to “Friends Only” or ”Private” as games are also unavailable in the browser.
The code from the released 1.x versions is no longer guaranteed to work when it comes to Steam Community. Valve deprecated the old XML APIs several years ago. Sadly, the modern Web API hasn‘t gotten much attention from Valve‘s side either. So development of Steam Condenser has mostly come to halt, too.
You might have more luck using the code from the master branch of the GitHub repository which uses Web API for most of the Community features.
You will have to register for a Steam Web API key, though: https://steamcommunity.com/dev/apikey
I inherited a rails app that is deployed using Heroku (I think). I edit it on AWS's Cloud9 IDE and, for now, just do everything in development mode. The app's purpose is to process large amounts of survey data and spit it out onto a PDF report. This works for small reports with like 10 rows of data, but when I load a report that is querying a data upload of 5000+ rows to create an HTML page which gets converted to a PDF, it takes around 105 seconds, much longer than Heroku's 30 seconds allotted for HTTP requests.
Heroku says this on their website, which gave me some hope:
"Heroku supports HTTP 1.1 features such as long-polling and streaming responses. An application has an initial 30 second window to respond with a single byte back to the client. However, each byte transmitted thereafter (either received from the client or sent by your application) resets a rolling 55 second window. If no data is sent during the 55 second window, the connection will be terminated." (Source: https://devcenter.heroku.com/articles/request-timeout#long-polling-and-streaming-responses)
This sounds excellent to me - I can just send a request to the client every second or so in a loop until we're done creating the large PDF report. However, I don't know how to send or receive a byte or so to "reset the rolling 55 second window" they're talking about.
Here's the part of my controller that is sending the request.
return render pdf: pdf_name + " " + pdf_year.to_s,
disposition: 'attachment',
page_height: 1300,
encoding: 'utf8',
page_size: 'A4',
footer: {html: {template: 'recent_grad/footer.html.erb'}, spacing: 0 },
margin: { top: 10, # default 10 (mm)
bottom: 20,
left: 10,
right: 10 },
template: "recent_grad/report.html.erb",
locals: {start: #start, survey: #survey, years: #years, college: #college, department: #department, program: #program, emphasis: #emphasis, questions: #questions}
I'm making other requests to get to this point, but I believe the part that is causing the issue is here where the template is being rendered. My template queries the database in a finite loop that stops when it runs out of survey questions to query from.
My question is this: how can I "send or receive a byte to the client" to tell Heroku "I'm still trying to create this massive PDF so please reset the timer and give me my 55 seconds!" Is it in the form of a query? Because, if so, I am querying the MySql database over and over again in my report.html.erb file.
Also, it used to work without issues and does work on small reports, but now I get the error "504 Gateway Timeout" before the request is complete on the actual page, but my puma console continues to query the database like a mad man. I assume it's a Heroku problem because the 504 error happens exactly every 35 seconds (5 seconds to process the other parts and 30 seconds to try to finish the loop in the template so it can render correctly).
If you need more information or code, please ask! Thanks in advance
EDIT:
Both of the comments below suggest possible duplicates, but neither of them have a real answer with real code, they simply refer to the docs that I am quoting here. I'm looking for a code example (or at least a way to get my foot in the door), not just a link to the docs. Thanks!
EDIT 2:
I tried what #Sergio said and installed SideKiq. I think I'm really close, but still having some issues with the worker. The worker doesn't have access to ActionView::Base which is required for the render method in rails, so it's not working. I can access the worker method which means my sidekiq and redis servers are running correctly, but it gets caught on the ActionView line with this error:
WARN: NameError: uninitialized constant HardWorker::ActionView
Here's the worker code:
require 'sidekiq'
Sidekiq.configure_client do |config|
# config.redis = { db: 1 }
config.redis = { url: 'redis://172.31.6.51:6379/0' }
end
Sidekiq.configure_server do |config|
# config.redis = { db: 1 }
config.redis = { url: 'redis://172.31.6.51:6379/0' }
end
class HardWorker
include Sidekiq::Worker
def perform(pdf_name, pdf_year)
av = ActionView::Base.new()
av.view_paths = ActionController::Base.view_paths
av.class_eval do
include Rails.application.routes.url_helpers
include ApplicationHelper
end
puts "inside hardworker"
puts pdf_name, pdf_year
av.render pdf: pdf_name + " " + pdf_year.to_s,
disposition: 'attachment',
page_height: 1300,
encoding: 'utf8',
page_size: 'A4',
footer: {html: {template: 'recent_grad/footer.html.erb'}, spacing: 0 },
margin: { top: 10, # default 10 (mm)
bottom: 20,
left: 10,
right: 10 },
template: "recent_grad/report.html.erb",
locals: {start: #start, survey: #survey, years: #years, college: #college, department: #department, program: #program, emphasis: #emphasis, questions: #questions}
end
end
Any suggestions?
EDIT 3:
I did what #Sergio said and attempted to make a PDF from an html.erb file directly and save it to a file. Here's my code:
# /app/controllers/recentgrad_controller.rb
pdf = WickedPdf.new.pdf_from_html_file('home/ec2-user/environment/gradSurvey/gradSurvey/app/views/recent_grad/report.html.erb')
save_path = Rails.root.join('pdfs', pdf_name + pdf_year.to_s + '.pdf')
File.open(save_path, 'wb') do |file|
file << pdf
end
And the error output:
RuntimeError (Failed to execute:
["/usr/local/rvm/gems/ruby-2.4.1#gradSurvey/bin/wkhtmltopdf", "file:///home/ec2-user/environment/gradSurvey/gradSurvey/app/views/recent_grad/report.html.erb", "/tmp/wicked_pdf_generated_file20190523-15416-hvb3zg.pdf"]
Error: PDF could not be generated!
Command Error: Loading pages (1/6)
Error: Failed loading page file:///home/ec2-user/environment/gradSurvey/gradSurvey/app/views/recent_grad/report.html.erb (sometimes it will work just to ignore this error with --load-error-handling ignore)
Exit with code 1 due to network error: ContentNotFoundError
):
I have no idea what it means when it says "sometimes it will work just to ignore this error with --load-error-handling ignore". The file definitely exists and I've tried maybe 5 variations of the file path.
I've had to do something like this several times. In all cases, I ended up writing a background job that does all the heavy lifting generation. And because it's not a web request, it's not affected by the 30 seconds timeout. It goes something like this:
client (your javascript code) requests a new report.
server generates job description and enqueues it for your worker to pick up.
worker picks the job from the queue and starts working (querying database, etc.)
in the meanwhile, client periodically asks the server "is my report done yet?". Server responds with "not yet, try again later"
worker is finished generating the report. It uploads the file to some storage (S3, for example), sets job status to "completed" and job result to the download link for the uploaded report file.
server, seeing that job is completed, can now respond to client status update requests "yes, it's done now. Here's the url. Have a good day."
Everybody's happy. And nobody had to do any streaming or playing with heroku's rolling response timeouts.
The scenario above uses short-polling. I find it the easiest to implement. But it is, of course, a bit wasteful with regard to resources. You can use long-polling or websockets or other fancy things.
Check my response here just in case it works for you. I didn´t wanted to change the user workflow adding a bg job and then a place/notification to get the result.
I use Rails controller streaming support with Live module and set the right reponse headers. I fetch the data from some Enumerable object.
It looks like Base CRM has upgraded their API and replaced all of their endpoints/parameters.
Previously I was able to retrieve "Won" deals using this call:
session = BaseCrm::Session.new("<LEGACY_ACCESS_TOKEN>")
session.deals.all(stage: :won, sort_by: :last_activity, sort_order: :desc, page: 1)
This query recently started ignoring my parameters, yet it continued to respond with unfiltered data (that was fun when I realized that was happening).
The new syntax is:
client = BaseCRM::Client.new(access_token: "<YOUR_PERSONAL_ACCESS_TOKEN>")
client.deals.where(organization_id: google.id, hot: true)
yet this does not work:
client.deals.where(stage_name: :won)
client.deals.where(stage_name: "Won")
client.deals.where(stage_id: 8) # specified ID found in Base Docs for "Won"
etc.
I've looked into the most recent updates to the Base CRM Gem as well as the Base CRM API Docs but have not found a solution to searching by specific deal stage.
Has anyone had any luck with the new API and this kind of query?
Is there a way to use the legacy API?
I've left message with Base but I really need to fix this, you know, yesterday.
Thanks for your help!
ADDITIONAL INFO
The legacy API/gem responded with JSON where the v2 API/gem responds with a BaseCRM::Deal object:
$ session.deals.find(123456)
# <BaseCRM::Deal
dropbox_email="dropbox#67890.deals.futuresimple.com",
name="Cool Deal Name",
owner_id=54321,
creator_id=65432,
value=2500,
estimated_close_date=nil,
last_activity_at="2016-04-21T02:29:43Z",
tags=[],
stage_id=84588,
contact_id=098765432,
custom_fields={:"Event Location"=>"New York, NY", :Source=>"Friend"},
last_stage_change_at="2016-04-21T02:08:20Z",
last_stage_change_by_id=559951,
created_at="2016-04-18T22:16:35Z",
id=123456,
updated_at="2016-04-21T02:08:20Z",
organization_id=nil,
hot=false,
currency="USD",
source_id=1466480,
loss_reason_id=nil
>
Checkout stage_id. Is this a bug? According to the Docs stage_id should return an integer between 1 and 10.
After upgrading to Ruby-1.9.3-p392 today, REXML throws a Runtime Error when attempting to retrieve an XML response over a certain size - everything works fine and no error is thrown when receiving under 25 XML records, but once a certain XML response length threshold is reached, I get this error:
Error occurred while parsing request parameters.
Contents:
RuntimeError (entity expansion has grown too large):
/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/rexml/text.rb:387:in `block in unnormalize'
I realize this was changed in the most recent Ruby version:
http://www.ruby-lang.org/en/news/2013/02/22/rexml-dos-2013-02-22/
As a quick fix, I've changed the size of REXML::Document.entity_expansion_text_limit to a larger number and the error goes away.
Is there a less risky solution?
This issue is generated when you send too much content as XML response.
To fix this issue : You need to restrict the data(< 10k) in the individual node (Instead of sending the whole data, show truncated data and provide a seperate link to view full content)
The error is being raised from the below file :
ruby-2.1.2/lib/ruby/2.1.0/rexml/text.rb
# Unescapes all possible entities
def Text::unnormalize( string, doctype=nil, filter=nil, illegal=nil )
sum = 0
string.gsub( /\r\n?/, "\n" ).gsub( REFERENCE ) {
s = Text.expand($&, doctype, filter)
if sum + s.bytesize > Security.entity_expansion_text_limit
raise "entity expansion has grown too large"
else
sum += s.bytesize
end
s
}
end
The limit ruby-2.1.2/lib/ruby/2.1.0/rexml/text.rb defaults to 10240 which means 10k data per node.
REXML already defaults to only allow 10000 entity substitutions per document, so the maximum amount of text that can be generated by entity substitution will be around 98 megabytes. (Refer https://www.ruby-lang.org/en/news/2013/02/22/rexml-dos-2013-02-22/ )
That sounds like a LOT of XML. Do you really need to get all of it? Maybe you can just request certain fields from the remote server? One option might be to try another XML parser (Nokogiri for example). Another option to maybe use something other than XML as a transport (JSON? Binary?).
I have a Rails 3 application. One of the controller method is for parsing large number of Twitter search results and storing them to the database. All works fine if the number of URLs to parse is small. But once the URLs reach 1000 or more, I have the following error after a few seconds (copied from the log file):
Address: http://search.twitter.com/search?q=+Chas%20Salon+near:%22Baltimore%22+within:15mi
Completed in 111436ms
OpenURI::HTTPError (420 unused):
app/controllers/twitter_reviews_controller.rb:41:in block in new'
app/controllers/twitter_reviews_controller.rb:20:ineach'
app/controllers/twitter_reviews_controller.rb:20:in `new'
I am using Hpricot to parse the Twitter search results.
The line number 41 in the above message is this:
doc = Hpricot(open(address))
Does anyone knows where is the problem?
Thank you very much for any help.
Cheers,
Tony.
You are being rate limited by Twitter server. read more here: http://dev.twitter.com/pages/rate-limiting
Manage your code so it should not exceed the rate limit in an hour. and look at this response code return by http://apiwiki.twitter.com/w/page/22554652/HTTP-Response-Codes-and-Errors