We have a Rails 3.2.12 app that fetches data from a partner API by making POST requests. However, we're finding that our code seems to be a bottleneck, as it takes longer to process requests than expected. Can we do anything to speed up the code?
Should we use another XML parser? Is there a faster way to post SSL requests in Ruby?
Thanks!
def ced( ad )
# Set vars
u = 'e'
pw = 'p'
ad = ad.join ','
url = 'https://r.e.com/interface.asp?ResponseType=XML&Command=BC' + '&UID=' + u + '&PW=' + pw + '&DL=' + ad
results = []
# Invoke API command
uri = URI.parse url
http = Net::HTTP.new uri.host, uri.port
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
req = Net::HTTP::Get.new uri.request_uri
resp = http.request req
# Parse API response
resp_xml = Nokogiri.XML resp.body
resp_xml.remove_namespaces!
num_errors = resp_xml.at_xpath('//ErrCount').content
# Any errors?
if num_errors != '0'
error = true
# No errors, process domains
else
resp_xml.xpath('//CheckResult').each do |elem|
results << {
:domain => elem.xpath('D').text.downcase,
:status => case elem.xpath('RRPCode').text
when '210' then 'a'
when '211' then 't'
else
error = true
'error'
end
}
end
end
<truncated>
end
Related
Hoping for some help as this one has me baffled...
I created a user account and API credentials at FTX.com.
They have an interesting Auth setup which is detailed here: https://docs.ftx.com/?python#authentication
They only provide code examples for python, javascript and c#, but I need to implement the integration on a RoR app.
Here's a link which also provides an example for both GET and POST calls: https://blog.ftx.com/blog/api-authentication/
I'm using:
ruby '3.0.1'
gem 'rails', '~> 6.1.4', '>= 6.1.4.1'
also,
require 'uri'
require 'net/https'
require 'net/http'
require 'json'
I got the authentication working for GET calls as follows:
def get_market
get_market_url = 'https://ftx.com/api/markets/BTC-PERP/orderbook?depth=20'
api_get_call(get_market_url)
end
def api_get_call(url)
ts = (Time.now.to_f * 1000).to_i
signature_payload = "#{ts}GET/api/markets"
key = ENV['FTX_API_SECRET']
data = signature_payload
digest = OpenSSL::Digest.new('sha256')
signature = OpenSSL::HMAC.hexdigest(digest, key, data)
headers = {
'FTX-KEY': ENV['FTX_API_KEY'],
'FTX-SIGN': signature,
'FTX-TS': ts.to_s
}
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.read_timeout = 1200
http.use_ssl = true
rsp = http.get(uri, headers)
JSON.parse(rsp.body)
end
This works great and I get the correct response:
=>
{"success"=>true,
"result"=>
{"bids"=>
[[64326.0, 2.0309],
...
[64303.0, 3.1067]],
"asks"=>
[[64327.0, 4.647],
...
[64352.0, 0.01]]}}
However, I can't seem to authenticate correctly for POST calls (even though as far as I can tell I am following the instructions correctly). I use the following:
def create_subaccount
create_subaccount_url = 'https://ftx.com/api/subaccounts'
call_body =
{
"nickname": "sub2",
}.to_json
api_post_call(create_subaccount_url, call_body)
end
def api_post_call(url, body)
ts = (Time.now.to_f * 1000).to_i
signature_payload = "#{ts}POST/api/subaccounts#{body}"
key = ENV['FTX_API_SECRET']
data = signature_payload
digest = OpenSSL::Digest.new('sha256')
signature = OpenSSL::HMAC.hexdigest(digest, key, data)
headers = {
'FTX-KEY': ENV['FTX_API_KEY'],
'FTX-SIGN': signature,
'FTX-TS': ts.to_s
}
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.read_timeout = 1200
http.use_ssl = true
request = Net::HTTP::Post.new(uri, headers)
request.body = body
response = http.request(request)
JSON.parse(response.body)
end
Also tried passing headers via request[] directly:
def api_post_call(url, body)
ts = (Time.now.to_f * 1000).to_i
signature_payload = "#{ts}POST/api/subaccounts#{body}"
key = ENV['FTX_API_SECRET']
data = signature_payload
digest = OpenSSL::Digest.new('sha256')
signature = OpenSSL::HMAC.hexdigest(digest, key, data)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.read_timeout = 1200
http.use_ssl = true
request = Net::HTTP::Post.new(uri)
request['FTX-KEY'] = ENV['FTX_API_KEY']
request['FTX-SIGN'] = signature
request['FTX-TS'] = ts.to_s
request.body = body
response = http.request(request)
JSON.parse(response.body)
end
This is the error response:
=> {"success"=>false, "error"=>"Not logged in: Invalid signature"}
My feeling is the issue is somewhere in adding the body to signature_payload before generating the signature via HMAC here..?:
signature_payload = "#{ts}POST/api/subaccounts#{body}"
Thinking this because, if I leave out #{body} here, like so:
signature_payload = "#{ts}POST/api/subaccounts"
the response is:
=> {"success"=>false, "error"=>"Missing parameter nickname"}
I have tried several iterations of setting up the POST call method using various different net/https examples but have had no luck...
I have also contacted FTX support but have had no response.
Would truly appreciate if anyone has some insight on what I am doing wrong here?
try this headers
headers = {
'FTX-KEY': ENV['FTX_API_KEY'],
'FTX-SIGN': signature,
'FTX-TS': ts.to_s,
'Content-Type' => 'application/json',
'Accepts' => 'application/json',
}
Here's a working example of a class to retrieve FTX subaccounts. Modify for your own purposes. I use HTTParty.
class Balancer
require 'uri'
require "openssl"
include HTTParty
def get_ftx_subaccounts
method = 'GET'
path = '/subaccounts'
url = "#{ENV['FTX_BASE_URL']}#{path}"
return HTTParty.get(url, headers: headers(method, path, ''))
end
def headers(*args)
{
'FTX-KEY' => ENV['FTX_API_KEY'],
'FTX-SIGN' => signature(*args),
'FTX-TS' => ts.to_s,
'Content-Type' => 'application/json',
'Accepts' => 'application/json',
}
end
def signature(*args)
OpenSSL::HMAC.hexdigest(digest, ENV['FTX_API_SECRET'], signature_payload(*args))
end
def signature_payload(method, path, query)
payload = [ts, method.to_s.upcase, "/api", path].compact
if method==:post
payload << query.to_json
elsif method==:get
payload << ("?" + URI.encode_www_form(query))
end unless query.empty?
payload.join.encode("UTF-8")
end
def ts
#ts ||= (Time.now.to_f * 1000).to_i
end
def digest
#digest ||= OpenSSL::Digest.new('sha256')
end
end
I'm working with the Microsoft Emotion API for processing emotions in video in a Rails app. I was able to make the call to the API to submit an operation, but now I have to query another API to get the status of the operation and once it's done it will provide the emotions data.
My issue is that when I query the results API, the response is that my operation is not found. As in, it doesn't exist.
I first sent the below request through my controller, which worked great:
#static controller
uri = URI('https://api.projectoxford.ai/emotion/v1.0/recognizeinvideo')
uri.query = URI.encode_www_form({})
request = Net::HTTP::Post.new(uri.request_uri)
request['Ocp-Apim-Subscription-Key'] = ENV['MEA_SubscriptionKey1']
request['Content-Type'] = 'application/octet-stream'
request.body = File.read("./public/mark_zuck.mov")
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
# Get response headers
response.each_header do |key, value|
p "#{key} => #{value}"
end
# Get operation location and id of operation
operation_location = response["operation-location"]
oid = operation_location.split("/")[6]
The response of this first call is:
"operation-location => https://api.projectoxford.ai/emotion/v1.0/operations/e7ef2ee1-ce75-41e0-bb64-e33ce71b1668"
The protocol is for one to grab the end of the "operation-location" url, which is the operation id, and send it back to the results API url like below:
# parse operation ID from url and add it to results API url
url = 'https://api.projectoxford.ai/emotion/v1.0/operations/' + oid
uri = URI(url)
uri.query = URI.encode_www_form({})
request = Net::HTTP::Get.new(uri.request_uri)
request['Ocp-Apim-Subscription-Key'] = ENV['MEA_SubscriptionKey1']
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
# Get response headers
response.each_header do |key, value|
p "#{key} => #{value}"
end
The result I get is:
"{\"error\":{\"code\":\"Unspecified\",\"message\":\"Operation not found.\"}}"
I get the same result when I query the Microsoft online API console with the operation id of an operation created through my app.
Does anyone have any ideas or experience with this? I would greatly appreciate it.
You do not need parse the "oid" out of "operation-location" header, as it is already the URL you should GET the status.
The following code works for me. Use it to see if you still see the issue.
require 'net/http'
require 'uri'
uri = URI('https://api.projectoxford.ai/emotion/v1.0/recognizeinvideo')
uri.query = URI.encode_www_form({})
request = Net::HTTP::Post.new(uri.request_uri)
request['Ocp-Apim-Subscription-Key'] = '<your key>'
request['Content-Type'] = 'application/octet-stream'
videoFile = File.open("c:\\1mb.mp4", "rb")
request.body = videoFile.read
videoFile.close
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
puts response.message
puts response.read_body
# Get response headers
response.each_header do |key, value|
p "#{key} => #{value}"
end
# Get operation location url for subsequent calls
operation_location = response["operation-location"]
operation_url = operation_location
uri = URI(operation_url)
uri.query = URI.encode_www_form({})
loop do
request = Net::HTTP::Get.new(uri.request_uri)
request['Ocp-Apim-Subscription-Key'] = '<your key>'
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
puts response.read_body
response_msg = response.read_body
break if response_msg.include?("Succeeded") or response_msg.include?("Failed")
sleep 20
end
puts response.message
puts response.read_body
I am trying to use the bitbucket API. I have successfully got the flow working where I am able to retrieve the access token and access token secret. After that, I have not been able to get anything to work. I can't find any example on to get this to work with Ruby. The closest I think I've found is this link:
https://gist.github.com/erikeldridge/383159
However in this example, he doesn't add the user's token and token secret, so I've updated it, here's his utility code:
# A utility for signing an url using OAuth in a way that's convenient for debugging
# Note: the standard Ruby OAuth lib is here http://github.com/mojodna/oauth
# License: http://gist.github.com/375593
# Usage: see example.rb below
require 'uri'
require 'cgi'
require 'openssl'
require 'base64'
class OauthUtil
attr_accessor :consumer_key, :consumer_secret, :token, :token_secret, :req_method,
:sig_method, :oauth_version, :callback_url, :params, :req_url, :base_str
def initialize
#consumer_key = ''
#consumer_secret = ''
#token = ''
#token_secret = ''
#req_method = 'GET'
#sig_method = 'HMAC-SHA1'
#oauth_version = '1.0'
#callback_url = ''
end
# openssl::random_bytes returns non-word chars, which need to be removed. using alt method to get length
# ref http://snippets.dzone.com/posts/show/491
def nonce
Array.new( 5 ) { rand(256) }.pack('C*').unpack('H*').first
end
def percent_encode( string )
# ref http://snippets.dzone.com/posts/show/1260
return URI.escape( string, Regexp.new("[^# {URI::PATTERN::UNRESERVED}]") ).gsub('*', '%2A')
end
# #ref http://oauth.net/core/1.0/#rfc.section.9.2
def signature
key = percent_encode( #consumer_secret ) + '&' + percent_encode( #token_secret )
# ref: http://blog.nathanielbibler.com/post/63031273/openssl-hmac-vs-ruby-hmac-benchmarks
digest = OpenSSL::Digest::Digest.new( 'sha1' )
hmac = OpenSSL::HMAC.digest( digest, key, #base_str )
# ref http://groups.google.com/group/oauth-ruby/browse_thread/thread/9110ed8c8f3cae81
Base64.encode64( hmac ).chomp.gsub( /\n/, '' )
end
# sort (very important as it affects the signature), concat, and percent encode
# #ref http://oauth.net/core/1.0/#rfc.section.9.1.1
# #ref http://oauth.net/core/1.0/#9.2.1
# #ref http://oauth.net/core/1.0/#rfc.section.A.5.1
def query_string
pairs = []
#params.sort.each { | key, val |
pairs.push( "#{ percent_encode( key ) }=#{ percent_encode( val.to_s ) }" )
}
pairs.join '&'
end
# organize params & create signature
def sign( parsed_url )
#params = {
'oauth_consumer_key' => #consumer_key,
'oauth_nonce' => nonce,
'oauth_signature_method' => #sig_method,
'oauth_timestamp' => Time.now.to_i.to_s,
'oauth_version' => #oauth_version
}
# if url has query, merge key/values into params obj overwriting defaults
if parsed_url.query
#params.merge! CGI.parse( parsed_url.query )
end
# #ref http://oauth.net/core/1.0/#rfc.section.9.1.2
#req_url = parsed_url.scheme + '://' + parsed_url.host + parsed_url.path
# create base str. make it an object attr for ez debugging
# ref http://oauth.net/core/1.0/#anchor14
#base_str = [
#req_method,
percent_encode( req_url ),
# normalization is just x-www-form-urlencoded
percent_encode( query_string )
].join( '&' )
# add signature
#params[ 'oauth_signature' ] = signature
return self
end
end
and here's my modified code:
require 'oauth_util.rb'
require 'net/http'
o = OauthUtil.new
o.consumer_key = MY_CONSUMER_KEY
o.consumer_secret = MY_SECRET
o.token = ACCESS_TOKEN_RETURNED_FROM_BB
o.token_secret = ACCESS_TOKEN_SECRET_RETURNED_FROM_BB
url = 'https://bitbucket.org/api/1.0/user'
parsed_url = URI.parse( url )
Net::HTTP.start( parsed_url.host ) { | http |
req = Net::HTTP::Get.new "#{ parsed_url.path }?#{ o.sign(parsed_url).query_string }"
response = http.request(req)
print response.read_body
}
Sadly, all I get is
301 Moved Permanently
301 Moved Permanently
nginx/1.5.10
Anyone have any luck using the BB API in Ruby after getting an access token back? Thanks for any help,
Kevin
EDIT:
There is a bitbucket API wrapper gem that has authentication built-in.
ORIGINAL:
I'm wondering if the "301 moved permanently" error is caused by the code making an http request instead of https.
without ssl:
result = Net::HTTP.get(URI.parse('http://bitbucket.org/api/1.0/user'))
# this returns 301 Moved Permanently
But when I used ssl instead (without the oauth headers/params) I got 401 Unauthorized.
with ssl:
1.9.3-p194 :063 > uri = URI.parse('https://bitbucket.org/api/1.0/user')
=> #<URI::HTTPS:0x007f846c5822d8 URL:https://bitbucket.org/api/1.0/user>
1.9.3-p194 :064 > http = Net::HTTP.new(uri.host, uri.port)
=> #<Net::HTTP bitbucket.org:443 open=false>
1.9.3-p194 :065 > http.use_ssl = true
=> true
1.9.3-p194 :066 > request = Net::HTTP::Get.new(uri.request_uri)
=> #<Net::HTTP::Get GET>
1.9.3-p194 :067 > response = http.request(request)
=> #<Net::HTTPUnauthorized 401 UNAUTHORIZED readbody=true>
I manually requesting a request token from twitter and pass the callback to the default page for a device, but get an error
Started GET "/users/auth/twitter/callback?device=mobile&oauth_token=mVpOFb1ruczKw7LzbgQYX73nq81hiw5OEBSOpob5rJk&oauth_verifier=WzBwpFdf7rYDH4DDWNbIfYPkHrIUzam9Ld6vskQrzNA" for 127.0.0.1 at 2014-02-03 18:00:03 +0400
omniauth: (twitter) Authentication failure! invalid_credentials: OAuth :: Unauthorized, 401 Unauthorized
If I log in through Devise, all without errors. Instructions took here. Why is this happening?
class Api::TwitterController < ApplicationController
def get_auth_token
consumer_key = OAUTH_KEYS[Rails.env]['twitter']['client_id'] # Obtainable from your destination site's API admin panel
consumer_secret = OAUTH_KEYS[Rails.env]['twitter']['secret_key'] # As above
callback_url = user_omniauth_callback_url(:twitter, device: :mobile)
method = 'POST'
uri = 'https://api.twitter.com/oauth/request_token'
params = set_params(consumer_key)
params['oauth_callback'] = url_encode(callback_url)
params['oauth_signature'] = url_encode(sign(consumer_secret + '&', signature_base_string(method, uri, params)))
token_data = parse_string(request_data(header(params), uri, method))
auth_token, auth_token_secret = [token_data['oauth_token'], token_data['oauth_token_secret']] # save these values, they'll be used again later
redirect_to "https://api.twitter.com/oauth/authorize?oauth_token=#{auth_token}"
end
private
# where parse_string is simply
def parse_string(str)
ret = {}
str.split('&').each do |pair|
key_and_val = pair.split('=')
ret[key_and_val[0]] = key_and_val[1]
end
ret
end
def set_params(consumer_key)
params = {
'oauth_consumer_key' => consumer_key, # Your consumer key
'oauth_nonce' => generate_nonce, # A random string, see below for function
'oauth_signature_method' => 'HMAC-SHA1', # How you'll be signing (see later)
'oauth_timestamp' => Time.now.getutc.to_i.to_s, # Timestamp
'oauth_version' => '1.0' # oAuth version
}
end
def generate_nonce(size=7)
Base64.encode64(OpenSSL::Random.random_bytes(size)).gsub(/\W/, '')
end
def signature_base_string(method, uri, params)
# Join up the parameters into one long URL-safe string of key value pairs
encoded_params = params.sort.collect{ |k, v| url_encode("#{k}=#{v}") }.join('%26')
# Join the above with your method and URL-safe destination URL
method + '&' + url_encode(uri) + '&' + encoded_params
end
# I'm a PHP developer primarily, hence the name of this function!
def url_encode(string)
CGI::escape(string)
end
# where sign is:
def sign(key, base_string)
digest = OpenSSL::Digest::Digest.new('sha1')
hmac = OpenSSL::HMAC.digest(digest, key, base_string)
Base64.encode64(hmac).chomp.gsub(/\n/, '')
end
# where header is:
def header(params)
header = "OAuth "
params.each do |k, v|
header += "#{k}=\"#{v}\", "
end
header.slice(0..-3) # chop off last ", "
end
def request_data(header, base_uri, method, post_data=nil)
url = URI.parse(base_uri)
http = Net::HTTP.new(url.host, 443) # set to 80 if not using HTTPS
http.use_ssl = true # ignore if not using HTTPS
if method == 'POST'
# post_data here should be your encoded POST string, NOT an array
resp, data = http.post(url.path, post_data, { 'Authorization' => header })
else
resp, data = http.get(url.to_s, { 'Authorization' => header })
end
resp.body
end
end
Problem solved, it was necessary to add some data in the session
auth_token, auth_token_secret = [token_data['oauth_token'], token_data['oauth_token_secret']]
session['oauth'] ||= {}
session['oauth']['twitter'] ||= {}
session['oauth']['twitter']['request_token'] = auth_token
session['oauth']['twitter']['request_secret'] = auth_token_secret
session['oauth']['twitter']['callback_confirmed'] = true
I'm trying to do a simple GET with ruby to the Facebook fql.query method without success.
The url is basically structured like this:
https://api.facebook.com/method/fql.query?query=SELECT total_count FROM link_stat WHERE url = "http://twitter.com/"&format=json
I've read in a few posts here on StackOverflow about how to make those requests, but even tho I keep getting:
/usr/lib/ruby/1.8/net/http.rb:560:in `initialize': getaddrinfo: Name or service not known (SocketError)
On the first line of http_get function.
def http_get(domain,path,params)
return Net::HTTP.get(domain, "#{path}?".concat(params.collect { |k,v| "#{k}=#{CGI::escape(v.to_s)}" }.join('&'))) if not params.nil?
return Net::HTTP.get(domain, path)
end
def getFacebookStats(url)
params = {
:query => 'SELECT total_count FROM link_stat WHERE url = "' + url + '"',
:format => 'json'
}
http = http_get('https://api.facebook.com', '/method/fql.query', params)
puts http
end
The http call accepts a host, not a URL:
def http_get(domain,path,params)
path = unless params.blank
path + "?" + params.collect { |k,v| "#{k}=#{CGI::escape(v.to_s)}" }.join('&')
else
path
end
request = Net::HTTP.get(domain, path)
end
def get_facebook_stats(url)
params = {
:query => 'SELECT total_count FROM link_stat WHERE url = "' + url + '"',
:format => 'json'
}
http = http_get('api.facebook.com', '/method/fql.query', params)
puts http
end
Please do not use camel case on method names on Ruby.
If you want to make HTTPS calls, you will have to use a different call:
require 'net/http'
require 'net/https'
http = Net::HTTP.new('somehost.com', 443)
http.use_ssl = true
path = '/login.html'
resp, data = http.get(path, nil)