Ruby 2 Upgrade Breaks Nokogiri and/or open-uri Encoding? - ruby-on-rails

Question

I have a mystery to solve when upgrading our Rails3.2 Ruby 1.9 app to a Rails3.2 Ruby 2.1.2 one. Nokogiri seems to break, in that it changes its behavior using open-uri. No gem versions are changed, just the ruby version (this is all on OSX Mavericks, using brew, gcc4 etc).
Steps to reproduce:
$ ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-darwin13.1.0]
$ rails console
Connecting to database specified by database.yml
Loading development environment (Rails 3.2.18)
> feed = Nokogiri::XML(open(URI.encode("http://anyblog.wordpress.org/feed/")))
=> #(Document:0x3fcb82f08448 {
name = "document",
children = [
..
> feed.xpath("//item").count
=> 10
So all good! Next, after a rvm change to Ruby 2.1.2 and a bundle install..
$ ruby -v
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin13.0]
$ rails console
Connecting to database specified by database.yml
Loading development environment (Rails 3.2.18)
> feed = Nokogiri::XML(open(URI.encode("http://anyblog.wordpress.org/feed/")))
=>
> feed.inspect
=> "#<Nokogiri::XML::Document:0x86a1f21c name=\"document\">"
> feed.xpath("//item").count
=> 0
So it looks like the 'open' encoding has changed, in that a gzip http stream isn't being fed correctly to nokogiri? I checked with a nokogiri -v and it is using the packaged xml libs rather than system ones. Is this a open-uri Ruby 2.1.2 issue?
Another theory is that one of the gems has monkey patched open-uri to fix something in 1.9 and that is breaking 2.1? Help or ideas please!
EDIT: Here's more info not using Nokogiri, i.e. thinking this is more a open-uri issue on Ruby 2.1.2:
> open(url) {|f|
* f.each_line {|line| p line}
* p f.content_type
* p f.charset
* p f.content_encoding
* }
"\u001F\x8B\b\u0000\u0000\u0000\u0000\u0000\u0000\u0003\xED\x9D\xDBr\eW\xB2\xA6\xAF\xED\xA7\xA8\xCD\u001E\xB7/$\u0010..
(snip)
3\xF3\xA79\xA7\xFAɗ\xFF\u000F\xEAo\x9C\u0014k\xE8\u0000\u0000"
"text/xml"
"utf-8"
["gzip"]
=> ["gzip"]
..the 1.9 version was readable, i.e. gzip was applied already.
If I go into a clean ruby irb it works ok, so it must be something in my rails gems that is changing the behavior of open-uri open to not deflate/gzip. I have a lot of gems referenced.. :(

Answer 1

Ok, here's an answer, and maybe the answer. Ruby 2 changed how it uses headers in HTTP requests and zipping/deflating, but at some point they changed their minds back and put it to be how 1.9 worked. In the interim some Rails gem maintainers monkey patched HTTP:Net to make their gems work on both 1.9 and 2.0. Those monkey patches still linger in older versions of gems and cause issues like I saw upgrading from 1.9 to 2.1
A summary of the issue and solution here:
http://avi.io/blog/2013/12/17/do-not-upgrade-your-rails-project-to-ruby-2-before-you-read-this/
We use the gem right_aws, and the details of that issue with ruby versions is here:
https://github.com/sferik/twitter/issues/473
The solution was to undo the monkey patch using this as a gem reference in our Gemfile:
gem 'right_http_connection', git: 'git://github.com/rightscale/right_http_connection.git', ref: '3359524d81'
Background reading and more info:
https://github.com/rightscale/right_aws/issues/167