I have a page that needs parameters received by a request from a third-party service. Unfortunately, the request takes a long time and the server crashes with a 504 error.
def show
start_time = Time.now
#project = Project.find(params[:id])
file = File.new(project.rvt_schema, 'rb')
rvt_params = ForgeHandler.instance.get_urn_token_params(file, "#{#project.id.to_s}.rvt")
#urn = rvt_params[:urn]
#token = rvt_params[:token]
end_time = Time.now
end
The most time inside the method is taken by request:
# Translate previously uploaded file to SVF format
def translate_to_svf(object_id,access_token)
base_64_urn = Base64.strict_encode64(object_id)
response = RestClient.post("#{API_URL}/modelderivative/v2/designdata/job",
{
input: {
urn: base_64_urn
},
output: {
formats: [
{
type: "svf",
views: [
"3d"
]
}
]
}
}.to_json,
{ Authorization: "Bearer #{access_token}", content_type:'application/json' })
return response
end
Which status is checked in cycle by another method:
def verify_job_complete(base_64_urn,access_token)
is_complete = false
while(!is_complete)
response = RestClient.get("#{API_URL}/modelderivative/v2/designdata/#{base_64_urn}/manifest",
{ Authorization: "Bearer #{access_token}"} )
json = JSON.parse(response.body)
if(json["progress"]=="complete")
is_complete = true
puts("***** Finished translating your file to SVF - status: #{json['status']}, progress: #{json['progress']} ")
else
puts("***** Haven't finished translating your file to SVF - status: #{json['status']}, progress: #{json['progress']} ")
sleep 5
end
end
I would like to implement asynchronous parameter loading. So I want to load data after losing control of the controller but but initializing the beginning of data loading from remote request in it. Tell me how best to implement this.
Or another way that would remove the error "Gateway timeout".
While this might be more of a question for the ruby-on-rails community, let me answer from the Autodesk Forge standpoint:
First of all, you should never wait for the Model Derivative job to complete when handling a request to your server. If the design file is complex enough, the translation could take up to hours, so this should definitely be handled asynchronously.
One option is to poll the status of the translation by requesting "derivative manifest" using the GET :urn/manifest endpoint.
Another option is to setup a Forge Webhook to get notified when the extraction.finished event is triggered.
It's probably easier to offload asynchronous stuff to a worker and save a reference to the user that needs to know about it. If you couple it with something like StimulusReflex you can render the result once it's finished. Another option might be the Render Async gem.
Related
In my rails app controller I am posting to the api of the app on the same machine. I have build this out to handle the posting the data to the url:
url = "http://172.16.155.165:3000/api/jobs"
params = {
:input => "original/video.h264",
:output => "new/video.mp4",
:preset => 'h264'
}
jobResults = Net::HTTP.post_form(URI.parse(url), params)
This works great when I run this code through rails console but when I use it in my controller it gives me this error after loading for a minute or so:
Timeout::Error in SeminarsController#create
Timeout::Error
Once the timeout happens the data is actually posted and the api does what it should. It is like it is hanging until it times out then posts the data. The controller never goes beyond this step though. It should write the response body to a file with jobResults.body which would work fine if it didn't time out. If I write this into rails console it outputs the response immediately. The api will never take a whole minute to respond.
Am I doing something to cause this to happen? How can I make it work right?
edit:
This is the code for create in app/controllers/api/jobs_controller.rb:
def create
job = Job.from_api(params, :callback_url => lambda { |job| api_job_url(job) })
if job.valid?
response.headers["X-State-Changes-Location"] = api_state_changes_url(job)
response.headers["X-Notifications-Location"] = api_notifications_url(job)
respond_with job, :location => api_job_url(job) do |format|
format.html { redirect_to jobs_path }
end
else
respond_with job do |format|
format.html { #job = job; render "/jobs/new"}
end
end
end
Yes. Ideally you should remove the long running process (yes this is long running process) into background job. Remember that when many users start updating the videos, this process will show down for many reasons (like bandwidth, API acceptance rate etc). Rake::Timeout always pops out if the process passes the threshold. It is actually designed to abort requests that are taking too long to respond. And, it is not raised in console.
How can I make it work right?
Move it to the background job. Or you can explictly increase the rake timeout interval by doing something like this
# config/initializers/timeout.rb
Rack::Timeout.timeout = 30 # seconds
But i suggest not to do this. This rake-timeout helps in debugging. Mainly people use in heroku with newrelic.
I'm running an application that uses mechanize to fetch some data every so often from an RSS feed.
It runs as a heroku worker and after a day or so I'm receiving the following error:
Errno::EMFILE: Too many open files - socket(2)
I wasn't able to find a "close" method within mechanize, is there anything special I need to be doing in order to close out my browser sessions?
Here is how I create the browser + read information:
def mechanize_browser
#mechanize_browser ||= begin
agent = Mechanize.new
agent.redirect_ok = true
agent.request_headers = {
'Accept-Encoding' => "gzip,deflate,sdch",
'Accept-Language' => "en-US,en;q=0.8",
}
agent
end
end
And actually fetching information:
response = mechanize_browser.get(url)
And then closing after the response:
def close_mechanize_browser
#mechanize_browser = nil
end
Thanks in advance!
Since you manually can't close each instance of Mechanize, you can try invoking Mechanize as a block. According to the docs:
After the block executes, the instance is cleaned up. This includes closing all open connections.
So, rather than abstracting Mechanize.new into a custom function, try running Mechanize via the start class method, which should automatically close all your connections upon completion of the request:
Mechanize.start do |m|
m.get("http://example.com")
end
I ran into this same issue. The Mechanize start example by #zeantsoi is the answer that I ended up following, but there is also a Mechanize.shutdown method if you want to do this manually without their block.
There is also an option that you can add a lambda on post_connect_hooks
Mechanize.new.post_connect_looks << lambda {|agent, url, response, response_body| agent.shutdown }
Im trying to create a table on BigQuery - I have a single dataset and need to use the api to add a table and import data (json.tar.gz) from cloud storage. I need to be able to use the ruby client to automate the whole process. I have two questions:
I have read the docs and tried to get it to upload (code below) and have not been successful and have absolutely no idea what Im doing wrong. Could somebody please enlighten me or point me in the right direction?
Once I make the request, how do I know when the job has actually finished? From the API, I presume Im meant to use a jobs.get request? Having not completed the first part I have been unable to get to look at this aspect.
This is my code below.
config= {
'configuration'=> {
'load'=> {
'sourceUris'=> ["gs://person-bucket/person_json.tar.gz"],
'schema'=> {
'fields'=> [
{ 'name'=>'person_id', 'type'=>'integer' },
{ 'name'=> 'person_name', 'type'=>'string' },
{ 'name'=> 'logged_in_at', 'type'=>'timestamp' },
]
},
'destinationTable'=> {
'projectId'=> "XXXXXXXXX",
'datasetId'=> "personDataset",
'tableId'=> "person"
},
'createDisposition' => 'CREATE_IF_NEEDED',
'maxBadRecords'=> 10,
}
},
'jobReference'=>{'projectId'=>XXXXXXXXX}
}
multipart_boundary="xxx"
body = "--#{multipart_boundary}\n"
body += "Content-Type: application/json; charset=UTF-8\n\n"
body += "#{config.to_json}\n"
body += "--#{multipart_boundary}\n"
body +="Content-Type: application/octet-stream\n\n"
body += "--#{multipart_boundary}--\n"
param_hash = {:api_method=> bigquery.jobs.insert }
param_hash[:parameters] = {'projectId' => 'XXXXXXXX'}
param_hash[:body] = body
param_hash[:headers] = {'Content-Type' => "multipart/related; boundary=#{multipart_boundary}"}
result = #client.execute(param_hash)
puts JSON.parse(result.response.header)
I get the following error:
{"error"=>{"errors"=>[{"domain"=>"global", "reason"=>"wrongUrlForUpload", "message"=>"Uploads must be sent to the upload URL. Re-send this request to https://www.googleapis.com/upload/bigquery/v2/projects/XXXXXXXX/jobs"}], "code"=>400, "message"=>"Uploads must be sent to the upload URL. Re-send this request to https://www.googleapis.com/upload/bigquery/v2/projects/XXXXXXXX/jobs"}}
From the request header, it appears to be going to the same URI the error says it should go to, and I am quite at a loss for how to proceed. Any help would be much appreciated.
Thank you and have a great day!
Since this is a "media upload" request, there is a slightly different protocol for making the request. The ruby doc here http://rubydoc.info/github/google/google-api-ruby-client/file/README.md#Media_Upload describes it in more detail. I'd use resumable upload rather than multipart because it is simpler.
Yes, as you suspected, the way to know when it is done is to do a jobs.get() to look up the status of the running job. The job id will be returned in the response from jobs.insert(). If you want more control, you can pass your own job id, so that in the event that the jobs.insert() call returns an error you can find out whether the job actually started.
Thank you for that. Answer resolved. Please see here :
How to import a json from a file on cloud storage to Bigquery
I think that the line of code in the docs for the resumable uploads section (http://rubydoc.info/github/google/google-api-ruby-client/file/README.md#Media_Upload) should read:
result = client.execute(:api_method => drive.files.insert,
Otherwise, this line will throw an error with 'result' undefined:
upload = result.resumable_upload
I have been struggling with this issue for a while now. I am trying to interface with the Blockchain API using rails. The API notifies my app of a specific event by means of a callback and I have to answer to this callback by responding “*ok*”. I am having a hard time creating a valid JSON response that consists only of “*ok*”. I can send an empty string format.json { render :json => {} } or something like format.json { render :json => {:ok => “*ok*”} } but can’t figure out how to send just the required “*ok*”.
http://blockchain.info/api/api_receive
Expected Response
In order to acknowledge successful processing of the callback the server should respond with the text "ok". If the server responds with anything else the callback will be resent again every new block (approximately every 10 minutes) up to 1000 times (1 week).
Thank you!
"*ok*"
This is invalid JSON. If you need to respond with plaintext, don't send a JSON response.
format.text { render text: “*ok*” }
or if the server isn't making a text/plaintext GET request, just respond without a respond_to block.
render text: "*ok*" and return
I'm using the following code to perform a request on the server from within a rake task:
app = ActionDispatch::Integration::Session.new(Rails.application)
app.host!('localhost:3000')
app.get(path)
This works well.
However, if I call app.get(path) again with the same path, the request is not repeated and the previous result is returned.
Is there a way I can force app.get to repeat the call?
Try to reset the session:
app.reset!
Here is how it works when reset,
def reset!
#https = false
#controller = #request = #response = nil
#_mock_session = nil
#request_count = 0
#url_options = nil
self.host = DEFAULT_HOST
self.remote_addr = "127.0.0.1"
self.accept = "text/xml,application/xml,application/xhtml+xml," +
"text/html;q=0.9,text/plain;q=0.8,image/png," +
"*/*;q=0.5"
unless defined? #named_routes_configured
# the helpers are made protected by default--we make them public for
# easier access during testing and troubleshooting.
#named_routes_configured = true
end
end
Otherwise it will just re-use the last response:
# this is a private method in Session, it will called every time you call `get/post, etc`
def process
......
#request_count += 1
#request = ActionDispatch::Request.new(session.last_request.env)
response = _mock_session.last_response
#response = ActionDispatch::TestResponse.new(response.status, response.headers, response.body)
#html_document = nil
....
end
Good luck!
I've worked out what's going on.
Basically the observation "the request is not repeated" is Rails' own caching in action. Which makes sense, app.get is treated as any other request, if caching is enabled, the cache is returned, and if it's not, it will repeat (as #henrikhodne claimed). This explains why a puts in the cached controller will not output the second time.
To verify, add a puts in 2 controller methods, but only set the expires_in in the second. The first one will repeat the output, the second will not.
The way to force the request to repeat is to bust the cache by modifying the URL e.g.
app.get("/") becomes app.get("/?r=123456") as you would if using HTTP. It all seems obvious in hindsight, basically app.get is treated exactly as a client request, and all the same rules apply.