I'm building out a platform for displaying data in charts that pulls from Zendesk's API. I'm running into trouble in that only 100 records at a time can be pulled with one call. How do I pull multiple pages of records from this resource?
Here is the code I use to make the call:
require 'net/http'
require 'uri'
require 'json'
#imports User data from the zendesk api and populates the database with it.
uri = URI.parse("https://samplesupport.zendesk.com/api/v2/users.json")
request = Net::HTTP::Get.new(uri)
request.content_type = "application/json"
request.basic_auth("sampleguy#sample.com", "samplepass")
req_options = {
use_ssl: uri.scheme == "https",
}
#response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
puts #response.body
puts #response.message
puts #response.code
This works fine for calling down one 'page' of resources...any help with grabbing multiple pages using my script would be greatly appreciated. Thank you!
Based on ZenDesk's documentation they return a next_page attribute in their payload. So you should just check for its existence and then query again if it exists. Repeat as needed.
require 'json'
# setup to query for the first page
results = JSON.parse(#response.body)
users = results['users'] #to get the users
if results['next_page']
# Do another query to results['next_page'] URL and add to users list
Related
I'm using Ruby on Rails 5 and I need to execute the following command in my application:
curl -F 'client_id=126581840734567' -F 'client_secret=678ebe1b3b8081231aab27dff738313' -F 'grant_type=authorization_code' -F 'redirect_uri=https://uri.com/' -F 'code=AQBi4L2Ohy3Q_N3V48OygFm0zb3gEsL985x5TIyDTNDJaLs93BwXiT1tyGYWoCg1HlBDU7ZRjUfLL5HVlzw4G-7YkVEjp6Id2WuqOz0Ylt-k2ADwDC5upH3CGVtHgf2udQhLlfDnQz5NPsnmxjg4bW3PJpW5FaQs8fn1ztgYp-ssfAf6IRt2-sI45ZC8cqqr5K_12y0Nq_Joh0H-tTfVyNLKatIxHPCqRDb3tfqgmxim1Q' https://api.instagram.com/oauth/access_token
so that it returns something like:
{"access_token": "IGQVJYS0k8V6ZACRC10WjYxQWtyMVRZAN8VXamh0RVBZAYi34RkFlOUxXZnTJsbjlEfnFJNmprQThmQ4hTckpFUmJEaXZAnQlNYa25aWURnX3hpO12NV1VMWDNMWmdIT3FicnJfZAVowM3VldlVWZAEViN1ZAidHlyU2VDMUNuMm2V", "user_id": 17231445640157812}
Is there a way to make Rails execute those types of commands? I was trying the following:
uri = URI.parse('https://api.instagram.com/oauth/access_token')
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri)
request.set_form_data({
"client_id" => "126581840734567",
"client_secret" => "678ebe1b3b8081231aab27dff738313",
"grant_type" => "authorization_code",
"redirect_uri" => "http://nace.network/",
"code" => params[:code]
})
res = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
but I get the following error:
end of file reached
in this line:
res = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
You're using HTTPS, so you need to add this to your code:
Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
res = http.request(request)
end
But if you don't need persistent connections, you could also use this:
res = Net::HTTP.post_form(uri,
"client_id" => "126581840734567",
"client_secret" => "678ebe1b3b8081231aab27dff738313",
"grant_type" => "authorization_code",
"redirect_uri" => "http://nace.network/",
"code" => params[:code]
)
Also, you could consider using a library like Faraday, which is a lot easier to deal with.
Edit
This is from TinMan's comment below, sound points.
Using cURL from inside Ruby or Rails is extremely valuable. There is an incredible amount of functionality inside cURL that isn't implemented in Rails or Ruby; Even Ruby's HTTP clients have a hard time replicating it, so cURL is very acceptable depending on the needs of the application. And, depending on the application, because cURL is in compiled C, it could easily outrun pure Ruby clients.
Curl is a means of issuing HTTP (or HTTPs) requests from the command line.
You don't want to use CURL in Rails. You want to issue HTTP requests from within Rails. Using curl is okay, it's one way to issue HTTP requests from with Rails.
We can refine that down further to, you want to issue HTTP requests from Ruby. Narrowing/distilling down to the most basic version of the problem is always good to do.
We knew all this already probably - still worth writing down for us all to benefit from!
Use HTTP in Ruby
We want to use a HTTP Client. There are many but, for this I'm going to use Faraday (a gem) 'cause I like it.
You've made a good start with Ruby's built in NET:HTTP but I prefer Faraday's DSL. It results in more readable and extendable code.
So, here is a class! I barely tested this so, use as a starting point. Make sure you write some unit tests for it.
# This is a Plain Old Ruby Object (PORO)
# It will work in Rails but, isn't Rails specific.
require 'faraday' # This require is needed as it's a PORO.
class InstagramOAuth
attr_reader :code
# The code parameter will likely change frequently, so we provide it
# at run time.
def initialize(code)
#code = code
end
def get_token
connection.get('/oauth/access_token') do |request|
request.params[:code] = code
end
end
private
def connection
#connection ||= Faraday.new(
url: instagram_api_url,
params: params,
ssl: { :ca_path => https_certificate_location }
)
end
def instagram_api_url
#url ||= 'https://api.instagram.com'
end
# You need to find out where these are for your self.
def https_certificate_location
'/usr/lib/ssl/certs'
end
def params
# These params likely won't change to often so we set a write time
# in the class like this.
{
client_id: '126581840734567',
client_secret: '678ebe1b3b8081231aab27dff738313',
grant_type: 'authorization_code',
redirect_uri: 'https://uri.com/'
}
end
end
# How do we use it? Like so
# Your big old authorisation code from your question
code = 'AQBi4L2Ohy3Q_N3V48OygFm0zb3gEsL985x5TIyDTNDJaLs93BwXiT1tyGYWoCg1HlBDU'\
'7ZRjUfLL5HVlzw4G-7YkVEjp6Id2WuqOz0Ylt-k2ADwDC5upH3CGVtHgf2udQhLlfDnQz'\
'5NPsnmxjg4bW3PJpW5FaQs8fn1ztgYp-ssfAf6IRt2-sI45ZC8cqqr5K_12y0Nq_Joh0H'\
'-tTfVyNLKatIxHPCqRDb3tfqgmxim1Q'
# This will return a Faraday::Response object but, what is in it?
response = InstagramOAuth.new(code).get_token
# Now we've got a Hash
response_hash = response.to_hash
puts 'Request made'
puts "Request full URL: #{response_hash[:url]}"
puts "HTTP status code: #{response_hash[:status]}"
puts "HTTP response body: #{response_hash[:body]}"
When I ran the snippet above I got the following. The class works, you just need to tweak the request params until you get what you want. Hopefully the class demonstrates how to send HTTP requests in Ruby/Rails.
Request made
Request full URL: https://api.instagram.com/oauth/access_token?client_id=126581840734567&client_secret=678ebe1b3b8081231aab27dff738313&code=AQBi4L2Ohy3Q_N3V48OygFm0zb3gEsL985x5TIyDTNDJaLs93BwXiT1tyGYWoCg1HlBDU7ZRjUfLL5HVlzw4G-7YkVEjp6Id2WuqOz0Ylt-k2ADwDC5upH3CGVtHgf2udQhLlfDnQz5NPsnmxjg4bW3PJpW5FaQs8fn1ztgYp-ssfAf6IRt2-sI45ZC8cqqr5K_12y0Nq_Joh0H-tTfVyNLKatIxHPCqRDb3tfqgmxim1Q&grant_type=authorization_code&redirect_uri=https%3A%2F%2Furi.com%2F
HTTP status code: 405
HTTP response body:
Additional Reading
. https://lostisland.github.io/faraday/usage/
. https://github.com/lostisland/faraday/wiki/Setting-up-SSL-certificates
I am trying to cache the results of API calls. My application makes multiple calls with the same result and I would like it to use the cache to save time. When I use Rails.cache.fetch the block of code (with the API request) is executed every time even though the keys are the same. I have enabled caching in the development environment using rails dev:cache.
I tried to test in the rails console but my local rails console also won't store any keys in the cache. On heroku, the console works for caching but the application still sends every API request.
I have tried to use both memory and file-based caching locally.
Here is the method in application_record.rb
I am removing the parameters that are not consistent or important for my purpose and then using the parameters and path as the cache key.
class ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
def send_request(params,path)
key = params.except(:token, :start_date, :end_date)
key[:path] = path
puts key.to_s
Rails.cache.fetch(key, expires_in: 5.minutes) do
if (!$token || $token == '') then self.authenticate end
params[:token] = $token
url = URI("https://<api-domain>/"+path+"/?"+params.to_query)
puts '**** sending request *****'
begin
Net::HTTP.start(url.host, url.port, :use_ssl => url.scheme == 'https') do |http|
request = Net::HTTP::Get.new(url)
request["cache-control"] = 'no-cache'
response = http.request(request)
if response.code == "200"
return response
elsif response.code[0] == "4"
puts "recursive = "+response.code
puts response.read_body
$token = nil
send_request(params,path)
else
puts "request_report server error"
puts response.code
puts JSON.parse(response.read_body)
response
end
end
rescue
response = "SocketError: check that the server allows outbound https and that the bucksense API is live"
end
end
end
The log shows that requests are made every time. The first and second requests for both of these campaigns are exactly the same.
Entering new campaign: Show********************************
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1D", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1M", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
Campaign finished: ********************************
Entering new campaign: ********************************
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1D", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1M", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
Campaign finished: ********************************
I expect this to make API calls only when the same request has not been made within 5 minutes. Instead, it makes the API call every single time.
Thanks for the help, let me know if I'm making a stupid mistake, or if this is a poor way to achieve my desired results.
HTTP::Response objects are not serializable, so they can't be stored in the cache. (See this Rails comment.)
I'm getting a no implicit conversion of String into Integer error that has me stumped, and unable to import user records and seed my database with them.
So far I have no problem accessing the data, but receive an error referencing the '[]' on the line with User.find... on it
The code I'm using is as follows:
require 'net/http'
require 'uri'
require 'json'
require 'faker'
#this script imports APR user data from the zendesk api and populates
the database with it.
uri = URI.parse("https://blahsupport.zendesk.com/api/v2/users.json")
request = Net::HTTP::Get.new(uri)
request.content_type = "application/json"
request.basic_auth("blah#blah.com", "blahpass")
req_options = {
use_ssl: uri.scheme == "https",
}
#response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
puts #response.body
puts #response.message
puts #response.code
info = #response.body
info.force_encoding("utf-8")
File.write('blahusers1.json', info)
puts "File Created Successfully!"
file = File.read('blahusers1.json')
users = JSON.load(file)
users.each do |a|
User.find_or_create_by_zendesk_id(:zendesk_id => a['id'], :url => a['url'], :name => a['name'], :email => a['email'])
end
Any ideas on how I've gotten this error? Thank you for any help!
**Edit
Below is an example of the data being returned.
{"users":[{"id":333653859,"url":"https://blahblah.zendesk.com/api/v2/users/333653859.json","name":"Randy Blah","email":"randy#blah.com","created_at":"2014-08-06T14:31:24Z","updated_at":"2018-04-04T14:22:06Z","time_zone":"Pacific Time (US & Canada)","phone":null,"shared_phone_number":null,"photo":{"url":"https://aprtechsupport.zendesk.com/api/v2/attachments/68955389.json","id":68955389,"file_name":"Work.jpg","content_url":"https://aprtechsupport.zendesk.com/system/photos/6895/5389/Work.jpg","mapped_content_url":"https://blahblah.zendesk.com/system/photos/6895/5389/Work.jpg","content_type":"image/jpeg","size":2528,"width":80,"height":80,"inline":false,"thumbnails":[{"url":"https://blahblah.zendesk.com/api/v2/attachments/68955399.json","id":68955399,"file_name":"Work_thumb.jpg","content_url":"https://blahblah.zendesk.com/system/photos/6895/5389/Work_thumb.jpg","mapped_content_url":"https://blahblah.zendesk.com/system/photos/6895/5389/Work_thumb.jpg","content_type":"image/jpeg","size":2522,"width":32,"height":32,"inline":false}]},"locale_id":1,"locale":"en-US","organization_id":null,"role":"admin","verified":true,"external_id":null,"tags":[],"alias":"","active":true,"shared":false,"shared_agent":false,"last_login_at":"2018-04-04T14:21:44Z","two_factor_auth_enabled":null,"signature":"Thanks for contacting the helpdesk!\n-Randy","details":"","notes":"","role_type":null,"custom_role_id":null,"moderator":true,"ticket_restriction":null,"only_private_comments":false,"restricted_agent":false,"suspended":false,"chat_only":false,"default_group_id":21692179,"user_fields":{}}
The example data you posted has a root object users that contains the array of user objects. So when you loop users using users.each, a is actually an Array and not a user Hash like you expected.
When you try to access an element of an Array using a 'String' index, it gives you the exception – no implicit conversion of String into Integer
So, try changing
users = JSON.load(file)
to
users = JSON.load(file)['users']
to get it working like how you'd expect.
I'm trying to pull a facebook avatar via auth. Here's what i'm doing:
def image_uri
require 'net/http'
image = URI.parse(params[:image]) # https://graph.facebook.com/565515262/picture
fetch = Net::HTTP.get_response(image)
based = 'data:image/jpg;base64,' << Base64.encode64(fetch)
render :text => based
end
I'm getting the following error (new error — edited):
Connection reset by peer
I've tried googling about, I can't seem to get a solution, any ideas?
I'm basically looking for the exact functioning of PHP's file_get_contents()
Try escaping the URI before parsing:
URI.parse URI.escape(params[:image])
Make sure that params[:image] does contain the uri you want to parse... I would instead pass the userid and interpolate it into the uri.
URI.parse URI.escape("https://graph.facebook.com/#{params[:image]}/picture)"
Does it throw the same error when you use a static string "https://graph.facebook.com/565515262/picture"
What does it say when you do
render :text => params[:image]
If both of the above don't answer your question then please try specifying the use of HTTPS-
uri = URI('https://secure.example.com/some_path?query=string')
Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https').start do |http|
request = Net::HTTP::Get.new uri.request_uri
response = http.request request # Net::HTTPResponse object
end
Presuming you are on ruby < 1.9.3, you will also have to
require 'net/https'
If you are on ruby 1.9.3 you don't have to do anything.
Edit
If you are on the latest version, you can simply do:
open(params[:image]) # http://graph.facebook.com/#{#user.facebook_id}/picture
HI ,
i am new to ROR
i am writing Ruby code for calling API blogs
i have written ruby code for
Creating the blog by
require 'net/http'
require 'uri'
url = URI.parse('http://localhost:3000/api/blogs/create.xml')
req = Net::HTTP::Post.new(url.path)
req.basic_auth 'a', 'a'
req.set_form_data({'blogpost[title]'=>'TestingAPIBlogposttitle',
'blogpost[description]'=>'Testing api desc',
'blogpost[category_id]'=>'3121'}, ';')
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req)}
case res
when Net::HTTPSuccess, Net::HTTPRedirection
puts res.body
else
res.error!
end
which runs successfully by creating a new blog
And i have a search code
require 'net/http'
require 'uri'
require 'cgi'
## Change this part according to the api to be accessed and the params to be passed.
uri = URI.parse( "http://localhost:3000/api/blogs/show/blogtitle.xml" )
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.path)
request.basic_auth 'a', 'a'
response = http.request(request)
puts response.body
which returns the
<?xml version="1.0" encoding="UTF-8"?>
<blogpost>
<created-at type="datetime">2010-09-02T08:18:22Z</created-at>
<description><p>Blog desc</p></description>
<slug>blogtitle</slug>
<title>blogtitle</title>
<user>
<firstname>admin</firstname>
<lastname>k</lastname>
<login>admin</login>
</user>
</blogpost>
Now i am trying to Update a BLog
for this how to write the code
i tried by simply changing the POST.new by PUT.new
but it didnt works for me
its showing me the error even if i gave admin User credentials
It might be worth trying a POST request but also adding a _method = 'put' parameter to the request. Rails can simulate a PUT request in this way though I would expect it to respond correctly to an HTTP PUT too.