Heroku throws OpenURI::HTTPError (403 Forbidden): - ruby-on-rails

Question

I have a Rails 4 Application serving up JSON data to a BackboneJS frontend client. The backend scrapes some content from Craigslist and serves it to the frontend as JSON. Locally, in development, it works just as expected.
On Heroku, the application layout does get served correctly and assets seem to load just fine. It isn't until backboneJS queries for data to fill the appropriate view, and there the app fails due to the OpenURI error.
More specifically, OpenURI keeps returning the
OpenURI::HTTPError (403 Forbidden)
when executing the following line in the Rails controller:
open( "#{clist.url}" )
I have spent several hours trying various 'solved' solutions I have found on Stack Overflow and Github and just trying to look through the Heroku logs for other errors, but the error remains no matter which suggested 'solutions' I try out.
As of now, I have tried the following suggested solutions, along with several other silly ones:
Required open-uri in my application controller
Added the 'User-Agent' key to my open method call
Changed the clist.url from "https" to "http" to avoid redirects
Also, the app is straight forward and doesn't require authentication at the moment either.
Apart from Stack Overflow and github, I couldn't find any suggested solutions elsewhere. Any help with any suggestions on other debugging tips to try or suggested solutions would be extremely appreciated. I am fairly new to Heroku, so still getting acquainted with debugging issues like this on a remote production engine.
Here is the relevant code that fetches from Craigslist (don't judge me too harshly. All/Most of this method is set to be refactored and put into it's own Class/Model, which is where it belongs ):
def index
#listings = []
#Retrieve job listings from Craigslist (see method sync_list below ... )
#raw_listings = sync_clist
# filters applied at this point ...
# and they transform #raw_listings to ...
# the array #listings
render json: #listings
end
def sync_clist
##search_items = SearchItem.all
#search_items = SearchItem.all[0..1]
site = #sites[0]
# in the Craigslist HTML, the second element in the returned job listing # is the better one to use
href_idx = 1
##### LINE 123 is the next one:
#search_items.each_with_index do |search_item, idx|
puts "#{search_item.url.upcase}"
##### LINE 125 ... FAILURE_HERE? *****************
html = open( "#{search_item.url}", 'User-Agent' => "Ruby/#{RUBY_VERSION}")
page = Nokogiri::HTML( html.read, nil, 'utf-8' )
category_idx = idx % 4
isNearby_listing = false #also capture 'nearby' jobs on Craigslist
page.css( site[:joblist_css] )[0..-2].each_with_index do |listing, i|
# convert the relative url in the list to a full-url
locale_idx = idx/#clist_locales.length
listing_url = listing.css('a')[href_idx]['href']
# Craigslist only lists the relative path of job urls - relative to the
# current search location. The 'More Local' items, however, return the
# full url.
if !isNearby_listing
posting_url = site[:protocol] + site[:locales][locale_idx] + "." + site[:host] + listing_url
else
posting_url = listing_url
end
# Once the appropriate heading is reached, the 'More Local' listings
# items begin appearing
if listing.next_sibling.node_name == 'h4'
isNearby_listing = true
end
posting_date = listing.css('time')[0]['datetime']
job_listing = { :source => site[:sitename].upcase,
:title => listing.css('a')[href_idx].text,
:url => posting_url,
:listing_id => listing["data-pid"],
:location => #clist_locales[locale_idx],
:content => "",
:telecommute => "",
:contract => "",
:pt_ft => "",
:favorite => false,
:posted_date => posting_date,
:category => #clist_categories[category_idx],
:apply_state => "new"
}
#new_listings << job_listing
end
end
#new_listings
end
Here is the output of my Heroku logs in case it helps:
```
2015-04-27T03:47:14.611278+00:00 app[web.1]: => Rails 4.0.8 application starting in production on http://0.0.0.0:43688
2015-04-27T03:47:14.611280+00:00 app[web.1]: => Run `rails server -h` for more startup options
2015-04-27T03:47:14.611310+00:00 app[web.1]: Started GET "/" for 24.5.106.52 at 2015-04-27 03:47:14 +0000
2015-04-27T03:47:14.611272+00:00 app[web.1]: => Booting WEBrick
2015-04-27T03:47:14.611281+00:00 app[web.1]: => Ctrl-C to shutdown server
2015-04-27T03:47:14.664777+00:00 app[web.1]: Rendered app/root.html.erb within layouts/application (0.5ms)
2015-04-27T03:47:14.661735+00:00 app[web.1]: Processing by AppController#root as HTML
2015-04-27T03:47:14.664783+00:00 app[web.1]: Rendered app/root.html.erb within layouts/application (0.5ms)
2015-04-27T03:47:14.674604+00:00 app[web.1]: Completed 200 OK in 13ms (Views: 12.3ms | ActiveRecord: 0.0ms)
2015-04-27T03:47:14.674612+00:00 app[web.1]: Completed 200 OK in 13ms (Views: 12.3ms | ActiveRecord: 0.0ms)
2015-04-27T03:47:14.611302+00:00 app[web.1]: Started GET "/" for 24.5.106.52 at 2015-04-27 03:47:14 +0000
2015-04-27T03:47:14.661749+00:00 app[web.1]: Processing by AppController#root as HTML
2015-04-27T03:47:15.943141+00:00 heroku[router]: at=info method=GET path="/assets/application-45c34fbd86efe641e061caa3b34737d7.css" host=APPNAME.herokuapp.com request_id=dbf8473e-2880-42f6-8ef4-11e0da7141b4 fwd="24.5.106.52" dyno=web.1 connect=2ms service=73ms status=200 bytes=569692
2015-04-27T03:47:15.944069+00:00 heroku[router]: at=info method=GET path="/assets/application-537f60efd0378faaddaea08875f25055.js" host=APPNAME.herokuapp.com request_id=5c7cf6db-393f-434e-a494-d3664d31a20f fwd="24.5.106.52" dyno=web.1 connect=2ms service=70ms status=200 bytes=965489
2015-04-27T03:47:17.631614+00:00 app[web.1]: Started GET "/posts" for 24.5.106.52 at 2015-04-27 03:47:17 +0000
2015-04-27T03:47:17.631624+00:00 app[web.1]: Started GET "/posts" for 24.5.106.52 at 2015-04-27 03:47:17 +0000
2015-04-27T03:47:17.636842+00:00 app[web.1]: Processing by PostsController#index as JSON
2015-04-27T03:47:17.636831+00:00 app[web.1]: Processing by PostsController#index as JSON
2015-04-27T03:47:17.665568+00:00 app[web.1]: HTTP://SFBAY.CRAIGSLIST.ORG/SEARCH/SOF?QUERY=RAILS
2015-04-27T03:47:18.048174+00:00 heroku[router]: at=info method=GET path="/posts" host=APPNAME.herokuapp.com request_id=a2724fb9-6872-4435-8dfc-0df1b73fb761 fwd="24.5.106.52" dyno=web.1 connect=2ms service=417ms status=500 bytes=330
2015-04-27T03:47:18.042116+00:00 app[web.1]: Completed 500 Internal Server Error in 405ms
2015-04-27T03:47:18.043586+00:00 app[web.1]: OpenURI::HTTPError (403 Forbidden):
2015-04-27T03:47:18.042129+00:00 app[web.1]: Completed 500 Internal Server Error in 405ms
2015-04-27T03:47:18.043590+00:00 app[web.1]: app/controllers/posts_controller.rb:123:in `each'
2015-04-27T03:47:18.043588+00:00 app[web.1]: app/controllers/posts_controller.rb:125:in `block in sync_clist'
2015-04-27T03:47:18.043593+00:00 app[web.1]: app/controllers/posts_controller.rb:123:in `sync_clist'
2015-04-27T03:47:18.043584+00:00 app[web.1]:
2015-04-27T03:47:18.043591+00:00 app[web.1]: app/controllers/posts_controller.rb:123:in `each_with_index'
2015-04-27T03:47:18.043596+00:00 app[web.1]:
2015-04-27T03:47:18.043594+00:00 app[web.1]: app/controllers/posts_controller.rb:53:in `index'
2015-04-27T03:47:18.043597+00:00 app[web.1]:
2015-04-27T03:47:18.043602+00:00 app[web.1]:
2015-04-27T03:47:18.043604+00:00 app[web.1]: app/controllers/posts_controller.rb:125:in `block in sync_clist'
2015-04-27T03:47:18.043603+00:00 app[web.1]: OpenURI::HTTPError (403 Forbidden):
2015-04-27T03:47:18.043606+00:00 app[web.1]: app/controllers/posts_controller.rb:123:in `each'
2015-04-27T03:47:18.043607+00:00 app[web.1]: app/controllers/posts_controller.rb:123:in `each_with_index'
2015-04-27T03:47:18.043610+00:00 app[web.1]: app/controllers/posts_controller.rb:53:in `index'
2015-04-27T03:47:18.043609+00:00 app[web.1]: app/controllers/posts_controller.rb:123:in `sync_clist'
2015-04-27T03:47:18.043613+00:00 app[web.1]:
2015-04-27T03:47:18.043611+00:00 app[web.1]:
```

Answer 1

Taking #TarynEast's suggestion and running app code in the Heroku console, I switched to using the 'net/http' library instead of 'open-uri' to retrieve the Craigslist webpage. The following message was returned by Craigslist when using 'net/http':
"This IP has been automatically blocked.\nIf you have questions, please email: blocks-b1402369961264436#craigslist.org\n"
So, apparently, either all of Heroku IPs have been blocked, or, more likely, it's just my app specifically that has been blocked even though my app only pings craigslist at most 8 times per site load. Maybe that was enough to block it since Craigslist is a very popular target of scraping apps. Anyways, mystery solved as to why the 403 Error. At least app still works locally.
[Update:] According to a quick Google search, Craigslist blocks the entirety of AWS and Heroku IPs, amongst others. See SO question here: Craigslist blocking Heroku/AWS

Answer 2

I had a similar issue when I had a Rails app hosted on Heroku opening a file held on S3. I got the same error because the URL was wrong.