"undefined method []" when parsing reddit api - ruby-on-rails

Question

I'm trying to request the json pages for multiple subreddits and take the title and link from each page for a college project. here's the code in question:
require 'rufus-scheduler'
require 'json'
require 'httparty'
ENV['TZ'] = 'Europe/Dublin'
scheduler = Rufus::Scheduler::singleton
scheduler.every '12h00m', :first_at => Time.now + 10 do
array_of_subreddits = ["pics", "memes", "funny", "aww", "memes",
"birdswitharms"]
array_of_subreddits.each do |category|
sleep 10 #wait 10 seconds between each request
#response = JSON.parse(HTTParty.get("http://reddit.com/r/#{category}/.json?limit=25").body)
#response['data']['children'].each do |data|
#link = data['data']['url']
#title = data['data']['title']
#category = category
Pic.create([{:title => "#{#title}", :link => "#{#link}", :category => "#{#category}"}])
end
end
end
this works perfectly sometimes, it will run through each one and end as it should. more often than not however it will give me this message after after one or two passes:
NoMethodError (undefined method `[]' for nil:NilClass):
app/controllers/home_controller.rb:17:in `block in index'
app/controllers/home_controller.rb:9:in `each'
app/controllers/home_controller.rb:9:in `index'
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_source.erb (4.8ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_trace.html.erb (2.2ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/_request_and_response.html.erb (1.2ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/templates/rescues/diagnostics.html.erb within rescues/layout (66.2ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_markup.html.erb (0.4ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_inner_console_markup.html.erb within layouts/inlined_string (0.3ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/_prompt_box_markup.html.erb within layouts/inlined_string (0.3ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/style.css.erb within layouts/inlined_string (0.5ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/console.js.erb within layouts/javascript (51.6ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/main.js.erb within layouts/javascript (0.3ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/error_page.js.erb within layouts/javascript (0.5ms)
Rendered /Users/conorbreen/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/web-console-2.3.0/lib/web_console/templates/index.html.erb (124.8ms)

Answer 1

Creating client classes is a much better way to work with httparty:
class RedditClient
include HTTParty
format :json
base_uri "http://reddit.com/r/"
def self.get_category(category, *opts)
opts.reverse_merge(limit: 25)
get("/#{category}.json", opts)
end
end
This way HTTParty handles JSON parsing for us and does not attempt to an convert an empty response. Its also much easier to test separately.
However you should still check if the response was successful before trying to use it:
#response = RedditClient.get_category(category)
if #response.success?
attrs = #response['data']['children'].map do |child|
{
category: category,
link: child['data']['url'],
title: child['data']['title']
}
end
Pic.create!(attrs)
else
# log it or raise some sort of error
end
Note that you where passing an array containing a single hash to .create. You can instead pass an array of hashes and it will insert the records in a single SQL insert statement.

Answer 2

When you get errors like this, you should always dump the actual response so you can inspect it. The fact you got a error for a nil with code doing stuff like ['data']['children'] means Id guess you got a JSON response, but one that is missing one of the first items (e.g. ['data'] returned nil).
Dont just assume every request is successful, many things can make a HTTP fail. Its possible that you get a valid JSON response back, just not the one you are expecting, for example an error message which would have told you the problem.
Also even with a 10 second delay, you may be hitting a rate limit (never tested Reddit personally), but read the rules
Many default User-Agents (like "Python/urllib" or "Java") are drastically limited to encourage unique and descriptive user-agent strings.

Answer 3

This kind of errors are most common in ruby or rails. Can be handled in multiple ways. As #Stefan suggested you can use any of the bellow.
Most simply like this
response = HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
if response.success?
response_body = response.body
# continue
end
or
response = HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
case response.code
when 200
puts "Good!"
# Continue your parsing
when 404
puts "NOT FOUND!"
when 500...600
puts "ERROR #{response.code}"
end
or
begin
HTTParty.get('http://reddit.com/r/#{category}/.json?limit=25')
rescue HTTParty::Error
# HTTParty errors like Not found
rescue StandardError
# StandardError like Timeout
else
# continue
end